Entropy-based pre-filtering using neural networks for streaming applications

Abstract

To provide a neural network-based pre-filter for content streaming applications.SOLUTION: In various examples, a DNN-based pre-filter for content streaming applications is used to dynamically adapt scene entropy in response to changing network or system conditions of an end-user device. For example, where network and / or system performance issues or degradation are identified, the DNN may be implemented as a frame pre-filter to reduce the complexity or entropy of the frame prior to streaming, thereby allowing the frame to be streamed at a reduced bit rate without requiring a change in resolution. The DNN-based pre-filter may be tuned to maintain image detail along an object, or the like such that scene navigation may be easier and more natural to the user.SELECTED DRAWING: Figure 1A

Description

[Technical Field]

[0001] This invention relates to entropy-based pre-filtering using a neural network for streaming applications. [Background technology]

[0002] Conventional video game streaming systems employ dynamic resolution changes (e.g., from 1080p to 720p) to cope with changing network conditions. For example, these systems can reduce frame resolution as network bandwidth decreases and increase frame resolution as network bandwidth increases. To do this, these conventional systems can introduce intra-coded frames ("I-frames")—or other intra-frame types—in the video stream to enable resolution transitions. However, I-frames are only subject to spatial compression, not temporal compression, and therefore require a higher bit rate to transmit. As a result, since dynamic resolution changes can be triggered in response to already strained network conditions, these conventional systems may transmit high-bit-rate I-frames at times when channel capacity is reduced. Therefore, end-user devices may receive incomplete I-frames (e.g., due to packet loss) and / or complete or incomplete I-frames with higher latency. Consequently, the image quality of the stream frames may be reduced, and the frames may be received with higher latency, resulting in noticeable lag—which impacts the user experience, especially in high-performance applications such as cloud gaming, virtual reality, augmented reality, and / or mixed reality applications. For example, streams received from these conventional systems may display a degraded video stream with reduced image detail due to packet loss, without explaining the preservation of important scene information. Furthermore, these conventional systems may require the client decoder to be reinitialized to support new resolutions, which consumes additional resources on the end-user side and can contribute to degradation of the displayed video quality—for example, degradation due to "hangs," "stutters," "hitches," or "jitter," where the video may be displayed at a reduced frame rate or with interrupted frame pacing. [Overview of the Initiative] [Means for solving the problem]

[0003] Embodiments of this disclosure relate to a neural network-based prefilter for content streaming applications. A system and method are disclosed for dynamically adapting scene entropy (e.g., complexity) in response to changes in network or system conditions to manage the streaming bit rate. For example, in contrast to conventional systems, network and / or system conditions of an end-user device may be monitored to determine whether frame entropy should be adapted to reduce the bit rate of the streaming video. If network and / or system conditions are optimal, reduction may not be necessary, and full complexity (highest entropy) frames may be streamed to the end-user device. However, if network and / or system performance issues or degradations are identified, a deep neural network (DNN) may be implemented as a frame prefilter to reduce the complexity or entropy of frames before streaming—thereby allowing frames to be streamed at a reduced bit rate without the need to change the resolution. As a result, frame resolution changes are not required, and inserting I-frames or other intra-frame types into a low-capacity network channel may not be necessary, thereby also avoiding decoder reinitialization on end-user devices. To compensate for the loss of detail from pre-filters, DNN-based pre-filters can be adjusted to preserve image detail along objects, boundaries, and / or surface edges (which may be visually / perceptually important to retain) so that scene navigation—for example, by a user joining an instance of the application—can be easier and more natural for the user.

[0004] In addition, in the embodiment, different DNNs may be trained for different entropy values ​​to adapt to different network and / or system conditions. For example, a DNN may be selected that pre-filters fewer frames when network conditions are improved than when network conditions are conceded (e.g., resulting in pre-filtered frames with higher entropy). In this way, rather than adjusting the frame resolution—a drawback of conventional systems—the system can adjust the amount of filtering using an intelligently selected DNN that adjusts the bit rate to suit the current network and / or system conditions of the end-user device.

[0005] The system and method for neural network-based pre-filtering for content streaming applications will be described in detail below with reference to the attached diagrams. [Brief explanation of the drawing]

[0006] [Figure 1A] This data flowchart illustrates an exemplary process for training a deep neural network (DNN) to pre-filter frames, according to some embodiments of the present disclosure. [Figure 1B] This data flowchart illustrates an exemplary process for training a deep neural network (DNN) to pre-filter frames, according to some embodiments of the present disclosure. [Figure 1C] This diagram shows exemplary DNN architectures for use in frame pre-filtering, according to some embodiments of the present disclosure. [Figure 2A] This data flow diagram illustrates an exemplary process for generating edge maps according to some embodiments of the present disclosure. [Figure 2B] This data flow diagram illustrates an exemplary process for temporally filtering edge maps according to some embodiments of the present disclosure. [Figure 2C] This is an illustrative visualization of a frame, according to some embodiments of the present disclosure. [Figure 2D] This is an exemplary visualization of a depth diagram generated using the frame of Figure 2C, according to some embodiments of the present disclosure. [Figure 2E] This is an exemplary visualization of a surface normal map generated using the frame of Figure 2C, according to some embodiments of the present disclosure. [Figure 2F] Figure 2C shows an exemplary visualization of an edge map generated using the frame according to some embodiments of the present disclosure. [Figure 3] This flowchart illustrates a method for training a DNN to pre-filter frames, according to some embodiments of the present disclosure. [Figure 4] This data flow diagram illustrates an exemplary process for pre-filtering frames using a DNN, according to some embodiments of the present disclosure. [Figure 5A] This is an exemplary visualization of pre-filtered frames at different entropy levels, according to some embodiments of the present disclosure. [Figure 5B] This is an exemplary visualization of pre-filtered frames at different entropy levels, according to some embodiments of the present disclosure. [Figure 5C] This is an exemplary visualization of pre-filtered frames at different entropy levels, according to some embodiments of the present disclosure. [Figure 5D] This is an exemplary visualization of pre-filtered frames at different entropy levels, according to some embodiments of the present disclosure. [Figure 6] This flowchart illustrates a method for pre-filtering frames using a DNN according to some embodiments of the present disclosure. [Figure 7] This is a block diagram of an exemplary content streaming system suitable for use in implementing some embodiments of the present disclosure. [Figure 8]This is a block diagram of an exemplary computing device suitable for use in implementing some embodiments of the present disclosure. [Figure 9] This is an exemplary data center block diagram suitable for use in implementing some embodiments of the present disclosure. [Modes for carrying out the invention]

[0007] Systems and methods relating to neural network-based pre-filtering for content streaming applications are disclosed. For example, the systems and methods of the disclosure may be implemented for any application in which frame pre-filtering processing may be implemented to reduce the bit rate or complexity of frames—for example, in simulation applications, virtual reality (VR), augmented reality (AR), and / or mixed reality (MR) applications, content editing applications, social media applications, remote desktop applications, content streaming applications, game streaming applications, video conferencing applications, and / or similar. As such, the pre-filters described herein may be useful for adjusting the frame entropy level of images or videos generated from any source and within any number of different application types.

[0008] The systems and methods of this disclosure can pre-filter frames using at least one deep neural network (DNN) (for example, before compression and transmission to an end-user device). For example, an entropy control parameter (e.g., λ) for filtering may be determined depending on the current system and / or network conditions of the end-user device. In some embodiments, the entropy control parameter may be inversely proportional to the desired entropy level, complexity, or detail of the scene at a given time. For example, if the system and / or network performance is poor, a lower entropy level may be desirable, and therefore the entropy control parameter may be high so that a DNN-based pre-filter is selected that filters—or blurs—the frame more thoroughly in an attempt to reduce the bit rate of the stream. For example, in embodiments, any number of different DNNs may be trained as pre-filters corresponding to their respective entropy control parameters. Network and / or system conditions may be determined, compared with a lookup table, or inserted into an algorithm, and the entropy control parameter may be calculated. After calculation, the entropy control parameters may be used to select a corresponding DNN-based prefilter, and the frames may be filtered to produce prefiltered frames. When network and / or system performance changes between streams, the DNN-based prefilter may be modified to account for changes in the entropy control parameters. In some embodiments, in addition to or instead of having separate DNNs with different entropy control parameter values, a single DNN may be trained to perform different levels of filtering based on the currently determined entropy control parameter values. In such embodiments, the values ​​of the entropy control parameters may be provided to the DNN as separate inputs, and these values ​​may be used by the DNN to perform the relevant levels of filtering.

[0009] During training, one or more DNNs may be trained through an unsupervised learning process that includes one or more loss functions. For example, in each iteration, the DNN may receive as input the current frame and edge map—for example, a saliency map, binary map, or image, and / or similar—encoded with values ​​indicating the locations of surfaces, objects, and / or boundaries depicted in the frame. In some examples, depth information and / or surface normal information may be used to compute the edge map for each frame. For example, depth information and / or surface normal information may be held by the currently running application, and this information may be used to generate the edge map or saliency map. The depth information or representation and / or surface normal information or representation may then be used to generate an edge map showing the edges depicted by the frame. For example, depth representations and / or regular expressions may be scaled and / or normalized, filtered (e.g., using morphological closing, including expansion and erosion), and / or subjected to edge detection operations (e.g., by using edge detection algorithms, including the Sobel operator). The resulting edge map may include a salience map, a binary map, or an image, and / or another edge map representation. In such examples, edge pixels may be encoded with a first value (e.g., 1), and non-edge pixels may be encoded with a second value (e.g., 0) to indicate the edge location within the frame. An edge map corresponding to a frame, and the frame itself, may be applied as input to a DNN.

[0010] A DNN can compute a filtered image as its output, and the filtered image can be compared to a frame and / or edge map using one or more loss functions—in an unsupervised manner, as in the embodiment. For example, an edge loss function can be used to train a DNN-based prefilter to preserve edge detail between a frame of a scene and the corresponding prefiltered frame of the scene, so that pixel values ​​along and / or near identified edges in the frame are preserved—or strictly preserved—so that navigation through the scene is sharper. In the embodiment, an edge loss function can compare pixels in the frame and the prefiltered frame at pixel locations determined to correspond to edges from the edge map. That is, for a given pixel from the edge map indicated to correspond to an edge, the pixel value at that location in the frame and the pixel value at that location in the prefiltered frame can be compared in such a way that the difference is penalized—for example, the greater the difference in pixel values, the higher the loss. As another example, an entropy loss function can be used to reduce or satisfy the frame entropy corresponding to the current desired entropy control parameter value (for example, the higher the value of the entropy control parameter, the lower the entropy of the pre-filtered frame, and vice versa). For example, the entropy loss function can measure the pixel gradient in a portion of the pre-filtered frame—excluding the portion corresponding to edges as identified using an edge map—in order to reduce the gradient between neighboring pixels. In such an example, the higher the value of the entropy control parameter, the greater the penalty imposed using the entropy loss function for higher gradients. When a high value of the entropy control parameter is used as such (for example, indicating a lower frame entropy), the gradient between neighboring or surrounding pixels can be reduced so that the difference in pixel values ​​between neighboring or surrounding pixels is minimized.Similarly, for lower entropy control parameter values (e.g., indicating higher frame entropy), the gradients between neighboring or surrounding pixels can be reduced less so that pixel differences are allowed to be larger (but not as large as in a full entropy frame). When multiple loss functions are used, the loss functions can be weighted during training. For example, the entropy control parameter value can be used to weight the loss function so that for higher entropy control parameter values, the consistency between the pixel values of neighboring or surrounding pixels is enforced more strongly, and vice versa.

[0011] The DNN used can include any DNN type - e.g., a convolutional neural network (CNN) - and can include any architecture type - e.g., an autoencoder architecture. After being trained, the selected DNN-based prefilter can be used to generate a prefiltered frame using the input frame and the corresponding edge map. The calculated prefiltered frame can then be encoded and compressed at a lower bitrate than the input frame - when less than full entropy is desired, and the encoded frame can be transmitted to an end-user device. As a result, the bitrate can be lowered for the stream and latency can be introduced without frame resolution changes - e.g., the frame resolution remains consistent throughout so that only the level of detail within the frame is changed.

[0012] Referring to FIGS. 1A - 1B, FIGS. 1A - 1B are exemplary data flow diagrams of processes 100A and 100B of frame - prefilter processing according to some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are merely illustrative examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, groups of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted altogether. Further, many of the elements described herein are functional entities that may be implemented as individual or distributed components or in combination with other components and in any suitable combination and location. The various functions described herein as being performed by entities may be implemented by hardware, firmware, and / or software. For example, the various functions may be implemented by processor - executable instructions stored in memory. Processes 100A and 100B may be implemented using features, functionality, and / or components similar to those of the exemplary content - streaming system 700 of FIG. 7, the exemplary computing device 800 of FIG. 8, and / or the exemplary data center 900 of FIG. 9.

[0013] Process 100A in Figure 1A includes an application engine 102, an edge map generator 104, an edge map 106, frame data 108, a DNN 110, pre-filtered frames 112, and a training engine 114. In some embodiments, the application engine 102 may also be a streaming application configured to provide image frames, I-frames, code, files, and other data necessary to display an application visualization. In non-limiting examples, the application engine 102 may run a simulation application, a virtual reality (VR), an augmented reality (AR), and / or mixed reality (MR) application, a content editing application, a social media application, a remote desktop application, a content streaming application, a game streaming application, a video conferencing application, and / or the like.

[0014] During operation, process 100A may be implemented to train DNN 110 via an unsupervised learning process that includes one or more loss functions. Initially, at each interval, application engine 102 may provide edge map generator 104 with a frame—for example, an input frame 120 in Figure 1B that may depict a frame such as a video, game, or application. Edge map generator 104 can compute an edge map 106 using the depth information and / or surface normal information of the frame. In some embodiments, such as when depth information and / or surface normal information is not available for the application, edge map generator 104 may generate an edge map 106 using frame data 108—for example, by using edge detection in frame data 108. For example, a clear contrast between the pixel values ​​of neighboring pixels represented by the frame data may indicate an edge. Edge map 106 may include a saliency map, a binary map or image, a grayscale map or image, and / or another edge map representation. As shown in the illustrative edge map 122 of Figure 1B, edge pixels (e.g., shown as white pixels) may be encoded with a first value (e.g., 1), and non-edge pixels (e.g., shown as black pixels) may be encoded with a second value (e.g., 0) to indicate the location of the edge in the frame.

[0015] Depth information and / or surface normal information may be maintained by the application engine 102, and this information may be used to generate the edge map 106. A brief look at Figures 2C-2F shows frames 220C / D / E / F of the scene, each depicting a different exemplary visualization of frame information. In some embodiments, frame 220C may depict a full-detail frame, and the edge map generator 104 may process frame 220C to determine, extract, and / or identify depth information to generate frame 220D. Frame 200D may depict depth information that can be encoded into frame 220C by the application engine 102 and / or another application or process. The depth information in frame 220D may then be used to generate the edge map 220F. For example, the edge map generator 104 may use differences in depth information (e.g., differences in depth values ​​of neighboring pixels exceeding a threshold) to identify edges in frame 220C and generate edge map 220F. In some embodiments, frame 220C may be processed by the edge map generator 104 to determine, extract, and / or identify surface normal information (e.g., lines, rays, and / or vectors perpendicular to a given surface) from frame 220C to generate frame 220E in which surface normal information can be depicted - for example, surface normal values ​​at each pixel may be encoded in the pixels. The surface normal information may then be used to generate edge map 220F. For example, since it can be assumed that the edges of an object may be formed by the intersection of two or more planes or half-planes, the edge map generator 104 may use surface normal vector changes exceeding a threshold (e.g., angles between neighboring vectors) to identify edges in frame 220C and generate edge map 220F.

[0016] In some embodiments, one or more of frames 220D and 220E—for example, corresponding to a depth map and a surface normal map, respectively—may be used to generate an edge map of frame 220F. In other embodiments, frame 220C—without first determining depth and / or surface normal information—may be used to generate an edge map of frame 220F. For example, changes in pixel values ​​exceeding a threshold may indicate edge locations, and this information may be used to determine edges for the edge map. However, using pixel values ​​alone without a depth map and / or surface normal map may lead to less accurate results than using a depth map and / or surface normal map.

[0017] In further embodiments for generating edge maps, such as those shown in process 200 in Figure 2A, depth information and / or normal information may be scaled and / or normalized, filtered (e.g., using morphological closing, including expansion and erosion), and / or subjected to edge detection operations (e.g., by using edge detection algorithms, including the Sobel operator). In addition, or alternatively, a temporally filtered edge map may be generated based on process 200B in Figure 2B. For example, in process 200B, frame motion vectors (e.g., corresponding to the motion of a virtual camera in a virtual environment from time t-1 to time t) may be used to warp or compensate for the motion and to generate a warped edge map at t-1. In such an example, the edge map corresponding to time t-1 may be transformed into the coordinate system of time t so that the warped edge map at time t-1 and the edge map at time t can be mixed together. The warped edge map can then be mixed with the current edge map to generate a time-stable edge map in the embodiment, according to the following equation (1): α * EMAP(t)+(1-α) * Warped EMAP(t-1)(1) Therefore, α is a weight that can be empirically determined to produce the most accurate and time-stable results (for example, between 0.0 and 1.0, or 0.5 in non-limiting embodiments).

[0018] Returning to Figure 1A, in some embodiments, the edge map 106 corresponding to the frame provided by the application engine, and the frame data 108 of the frame provided by the application engine, can be applied as input to the DNN 110. The DNN 110 can include any DNN type—for example, a convolutional neural network (CNN)—and in embodiments, it can include any architecture type—for example, an autoencoder, an encoder / decoder, and / or other architectures. Although described as DNN110, DNN110 may include, but is not limited to, any type of machine learning model, such as linear regression, logistic regression, decision trees, support vector machines (SVMs), naive Bayes, k-nearest neighbors (Knn), K-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoders, convolutions, cycles, perceptrons, long / short-term memory / LSTMs, Hopfield, Boltzmann, deep belief, deconvolution, adversarial generation, liquid state machines, etc.), machine learning models using computer vision algorithms, and / or other types of machine learning models. After being trained by the training engine 114, DNN110 can be used to generate pre-filtered frames 112 using frame data 108 and edge maps 106.

[0019] Referring briefly to Figure 1C, Figure 1C is an exemplary DNN architecture 110A for use in pre-filtering an input frame 120 according to some embodiments of the present disclosure. The DNN may include any number of layers 130. One or more of the layers 130 may include an input layer. The input layer may hold values ​​related to the input frame 120 (e.g., before or after post-processing). One or more of the layers 130 may include a convolutional layer. The convolutional layer may compute the output of neurons connected to local regions in the input layer, each neuron computing the dot product of its weights and the small regions to which they are connected in the input volume. One or more of the layers 130 may include a ReLU (rectified linear unit) layer. The ReLU layer may apply, for example, a zero-thresholding element-wise activation function, such as max(0,x). The resulting volume of the ReLU layer may be the same as the volume of the input to the ReLU layer. One or more of the layers 130 may include a pooling layer. A pooled layer may perform a down-sampling operation along a spatial dimension (e.g., height and width) that may result in a volume smaller than the input to the pooled layer. One or more of the layers 130 may include one or more fully connected layers. Each neuron in a fully connected layer may be connected to each of the neurons in the previous volume. A fully connected layer may compute class scores, and the resulting volume may be 1x1x the number of classes. In some examples, a CNN may include fully connected layers such that the outputs of one or more of the layers 130 of the CNN may be provided as inputs to the fully connected layers of the CNN. In some examples, one or more convolutional streams may be implemented by a DNN, and some or all of the convolutional streams may include their respective fully connected layers. In some non-restrictive embodiments, a DNN may include a series of convolutional and max-pooled layers to facilitate image feature extraction, followed by multiscale augmented convolutional and upsampling layers to facilitate global context feature extraction.

[0020] The input layer, convolutional layer, pooling layer, ReLU layer, and fully connected layer are discussed herein in relation to DNN110, but this is not intended to be limiting. For example, additional or alternative layers may be used in DNN110, e.g., normalization layers, SoftMax layers, and / or other layer types. In embodiments where DNN110 includes a CNN, different order and number of layers of the CNN may be used depending on the embodiment. In other words, the order and number of layers 130 of DNN110 are not limited to any one architecture.

[0021] In addition, some of the layers 130 may include parameters (e.g., weights and / or biases), for example, convolutional layers and fully connected layers, while others may not, for example, ReLU layers and pooling layers. In some examples, the parameters may be learned by the DNN 110 during training. Furthermore, some of the layers 130 may include additional hyperparameters (e.g., learning rate, stride, epoch, etc.), for example, convolutional layers, fully connected layers, and pooling layers, while others may not, for example, ReLU layers. The parameters and hyperparameters should not be limited and may vary depending on the embodiment.

[0022] In some embodiments, to train the DNN110, the training engine 114 may use one or more loss functions, for example, the one used in the unsupervised training block 124 in Figure 1B. The training engine 114 may also receive and / or access various training data sets from the training data store 126 for use in training the DNN110. With respect to loss functions, for example, the edge loss function of the training engine 114 may be used to train the DNN110 to maintain edge detail between the frame output by the application engine 102 and the pre-filtered frame 112, such that pixel values ​​along and / or close to identified edges in the frame edge map 106 are maintained—or at least close—so that navigation through the scene depicted by the application corresponding to the application engine 102 is sharper. For example, when viewing the input frame 120 in comparison to the pre-filtered frame 112, the input frame 120 contains a considerable level of detail that is not visible in the pre-filtered frame 112. Specifically, the ground 130A of the input frame 120 contains details that are not visible in the ground 130B of the pre-filtered frame 112. In the embodiment, the edge loss function can compare the pixel values ​​of the pre-filtered frame 112 at pixel locations determined to correspond to edges from the frame data 108 and the edge map 106. That is, for a given pixel from the edge map 106 indicated as corresponding to an edge, the pixel value at that location in the frame data 108 and the pixel value at that location in the pre-filtered frame 112 can be compared in such a way that the difference is penalized.

[0023] In some practical examples, the edge loss function can be calculated according to the following equation (2): L エッジ =ΣΣ||EMASK(i,j) *(F(i,j)-PF(i,j))|| n (2) Therefore, the pixel positions (i,j) in the edge mask (e.g., 122) are used to compare the corresponding pixel positions (i,j) in the original frame (F) (e.g., frame 120) with the corresponding pixel positions (i,j) in the pre-filtered frame (PF) (128). This process can be repeated for each pixel determined from the edge map to correspond to an edge pixel.

[0024] As another example, the entropy loss function of the training engine 114 may be used to reduce or satisfy the frame entropy corresponding to the current desired entropy control parameter value (e.g., a higher value of the entropy control parameter, a lower entropy of the pre-filtered frame, and vice versa). For example, the entropy loss function may measure the pixel gradient in a portion of the pre-filtered frame 112—excluding the portion corresponding to edges as identified using the edge map 106—to reduce the gradient between neighboring pixels. In such an example, the higher the entropy control parameter value, the greater the penalty imposed using the entropy loss function for higher gradients. When a high value of the entropy control parameter is used as such (e.g., indicating a lower frame entropy), the gradient between neighboring or surrounding pixels may be reduced so that the difference in pixel values ​​between neighboring or surrounding pixels is minimized. Similarly, for lower entropy control parameter values ​​(e.g., exhibiting higher frame entropy), the gradient between neighboring or surrounding pixels can be reduced less, allowing for larger pixel differences (but not as large as in a perfect entropy frame).

[0025] In some embodiments, the entropy loss function can be calculated according to the following equation (3): L エントロピ =ΣΣ||(PF(i,j)-PF(i+m,j+n))||(3) Here, (i, j) corresponds to the (x, y) pixel position that is not an edge pixel, m is the pixel distance in the x direction, and n is the pixel distance in the y direction. For example, n can include values of +1 and -1, m can include values of +1 and -1, and one or more of these value combinations are used to identify a pixel and compare it with pixel (i, j).

[0026] When multiple loss functions are used, the loss functions can be weighted during training. For example, for higher entropy control parameter values, the consistency between the pixel values of neighboring or surrounding pixels is enforced more strongly, and vice versa, the entropy control parameter value can be used to weight the loss functions. For example, the total loss function can be calculated according to the following equation (4): Total loss = L エッジ + λ * L エントロピ (4) Here, λ corresponds to the entropy control parameter. As such, in instances with lower entropy, the higher the λ value, whereby the entropy loss function is weighted more heavily compared to the edge loss function. In contrast, in instances with higher entropy, the lower the λ value, whereby the entropy loss function is weighted less compared to the edge loss function.

[0027] Referring here to Figure 3, each block of the method 300 described herein includes a computing process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by processor execution instructions stored in memory. Method 300 may also be performed as computer-usable instructions stored in a computer storage medium. Method 300 may be provided, to name a few, as a standalone application, a service or a hosted service (standalone or in combination with another hosted service), or as a plug-in to another product. In addition, Method 300 is described with respect to the system in Figure 1 as an example. However, this method may be performed in addition to or otherwise by any one system or any combination of systems, including but not limited to those described herein.

[0028] Figure 3 is a flowchart illustrating method 300 for training a DNN to pre-filter a frame, according to some embodiments of the present disclosure. Method 300 includes, in block B302, generating a saliency map showing the edges depicted in the frame, using at least one of the surface normals or depth values ​​corresponding to the frame. For example, depth information and / or surface normal information may be held by the currently running application, and this information may be used to generate the edge map or saliency map. In some embodiments, in addition to or instead of using surface normals and / or depth values, pixel values ​​from the frame data may be used.

[0029] Method 300 includes, in block B304, calculating a pre-filtered frame using a DNN and at least in part on data representing a salience map and the frame. For example, DNN 110 may be used to generate a pre-filtered frame using frame data and an edge map.

[0030] Method 300 includes calculating a first loss value in block B306 using a first loss function, at least in part on comparing a first pixel value of the pre-filtered frame with a second pixel value of the frame at a pixel location corresponding to an edge, as determined from the saliency map. For example, the edge loss function can compare the pixel values ​​of the frame with those of the pre-filtered frame at pixel locations determined to correspond to edges from the edge map. That is, for a given pixel from the edge map indicated to correspond to an edge, the pixel value at that location in the frame and the pixel value at that location in the pre-filtered frame can be compared in such a way that the difference is penalized—for example, the greater the difference in pixel values, the higher the loss.

[0031] Method 300 includes, in block B308, calculating a second loss value using a second loss function, at least in part, based on comparing the pixel values ​​of adjacent pixels in the pre-filtered frame. For example, the entropy loss function can measure the pixel gradient in a portion of the pre-filtered frame—excluding the portion corresponding to edges as identified using an edge map—to reduce the gradient between neighboring pixels. For example, for a DNN110 corresponding to a higher entropy control parameter (and therefore a lower frame entropy), the difference between neighboring pixels may be penalized more. As another example, for a DNN110 corresponding to a lower entropy control parameter (and therefore a higher frame entropy), the difference between neighboring pixels may be penalized less—but still penalized in order to reduce the frame entropy from that of the original frame.

[0032] Method 300 includes updating one or more parameters of the DNN in block B310 based at least in part on the first and second loss values. For example, the parameters of DNN 110 (e.g., weights and / or biases) may be updated using the training engine 114 until an acceptable level of accuracy is achieved. In some embodiments, the first and second loss values ​​may be used together—and weighted—to generate the final loss value, as described herein, for example.

[0033] Figure 4 shows a data flowchart illustrating an exemplary process 400 for pre-filtering frames using a DNN according to some embodiments of the present disclosure. Figure 4 includes a video game streaming system 422, an application engine 102, an edge map generator 104, an edge map 106, frame data 108, a selected DNN 418, a pre-filtered frame 112, an encoder 402, an encoded frame 404, an end-user device 420, a channel condition monitor 414, channel condition data, a DNN selector 412, and a DNN 430.

[0034] The end-user device 420 may include smartphones, laptop computers, tablet computers, desktop computers, wearable devices, game consoles, virtual reality (VR) or augmented reality (AR) systems (e.g., headsets, computers, game consoles, remote controls, controllers, and / or other components), content streaming devices (e.g., NVIDIA SHIELD), smart home devices which may include intelligent personal assistants, and / or other types of devices capable of supporting application streaming.

[0035] The end-user device 420 may include a decoder 406, a display 408, and an application 410. Only a few components and / or features of the end-user device 420 are shown in Figure 4, but this is not intended to be limiting. For example, the end-user device 420 may include additional or alternative components, such as those described later with respect to the computing device 800 in Figure 8. The application 410 may be any application in which frame pre-filtering processing may be implemented to reduce the bit rate or complexity of frames—for example, in simulation applications, virtual reality (VR), augmented reality (AR), and / or mixed reality (MR) applications, content editing applications, social media applications, remote desktop applications, content streaming applications, game streaming applications, video conferencing applications, and / or similar.

[0036] Display 408 may include any type of display capable of displaying Application 410 (e.g., light-emitting diode displays (LEDs), organic LED displays (OLEDs), liquid crystal displays (LCDs), active-matrix OLED displays (AMOLEDs), quantum dot displays (QDDs), plasma displays, LED / LCD displays, and / or other types of displays). In some examples, Display 408 may include multiple displays (e.g., a dual-monitor display for a computer game, such as a first display for configuring the game and a virtual reality display for playing the game). In some examples, Display 408 is a touchscreen display, such as a touchscreen on a smartphone, tablet computer, laptop computer, or similar device.

[0037] During operation, the end-user device 420 may transmit channel conditions (e.g., bandwidth, channel capacity, bit rate, signal-to-noise ratio (SINR), spectral efficiency, and / or additional state information) to the channel condition monitor of the video game streaming system 422. The channel condition monitor 414 may process and / or format the transmitted channel conditions to generate channel condition data 416 that can be provided to the DNN selector 412. The DNN selector 412 may compare the channel condition data 416 with channel condition data 416 inserted into a lookup table or algorithm, and entropy control parameters may be calculated by the DNN selector 412. After calculation, the DNN selector 412 may access the DNN 430 and use the entropy control parameters to select a DNN corresponding to the control parameters associated with the channel condition data 416. Note that the DNN 430 and the selected DNN 418 may correspond to trained or deployed instances of DNN 110 in Figures 1A-1C.

[0038] DNN430 can store several DNNs, each corresponding to a different level of filtering—for example, to different entropy control parameters. Depending on the level of filtering, the pre-filtered frames may contain some visual detail when displayed on the display 408 of the end-user device 420. Briefly referring to Figures 5A-5D, for example, each of frames 500A / B / C / D corresponds to different entropy control parameters (e.g., 0.1, 0.5, 1.0, and 3.0, respectively) corresponding to high entropy, medium entropy, low entropy, and very low entropy, respectively. Each of frames 500A / B / C / D contains archways 502A / B / C / D. As illustrated, archway 502A contains a significant level of detail with clearly defined and visible bricks in archway 502A. Archway 502B contains less detail compared to 502A. Archway 502B includes some texture to indicate brickwork, but the brickwork is not clearly defined visually. Archway 502C includes less detail compared to 502B. Archway 502C includes some detail, but the brickwork is invisible. Finally, archway 502D includes even less detail compared to archway 502C. Archway 502D does not include texture, and the brickwork is invisible. However, the edges of archway 502D are preserved, allowing the user to navigate the game corresponding to frames 500A / B / C / D.

[0039] Returning to Figure 4, the DNN selector 412 can use entropy control parameters to select a DNN corresponding to the control parameters—for example, the selected DNN 418. The video game streaming system 422 can then provide the selected DNN 418—as described in relation to Figures 1A-1C—with frame data 108 and edge maps 106 to generate a pre-filtered frame 112. The pre-filtered frame 112 can then be encoded and compressed by the encoder 402 to generate an encoded frame 404. The encoded frame 404 may contain a reduced bit rate compared to the input frame—if the selected DNN 418 is trained to reduce entropy—and the encoded frame 404 can then be sent to an application 410 on an end-user device 420. The application 410 can use the decoder 406 to decode the encoded frame 404 and to generate the pre-filtered frame 112 for display via the display 408.

[0040] In some embodiments, in addition to having different DNN430 for different entropy control parameters, or instead—for example, to generate pre-filtered frames 112 with varying entropy levels—a single DNN430 may be trained to use the entropy control parameters as input, and using the entropy control parameters in addition to the frame data 108 and edge map 106, the DNN430 may compute pre-filtered frames 112 according to a desired entropy value.

[0041] Referring here to Figure 6, each block of Method 600 as described herein includes a computation process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory. Method 600 may also be performed as computer-usable instructions stored on a computer storage medium. Method 600 may be provided, to name a few, as a standalone application, a service or a hosted service (standalone or in combination with another hosted service), or as a plug-in to another product. In addition, Method 600 is described, for example, with respect to the system in Figure 4. However, this method may be performed in addition to or otherwise by any one system or any combination of systems, including but not limited to those described herein.

[0042] Figure 6 is a flowchart illustrating method 600 for pre-filtering frames using a DNN, according to some embodiments of the present disclosure. Method 600 includes, in block B602, selecting a DNN from a plurality of deep neural networks (DNNs) based at least in part on at least one of the network or system conditions corresponding to the end-user device. For example, if the system and / or network performance is poor, a lower entropy may be desirable, and therefore the entropy control parameter may be higher so that a DNN-based pre-filter is selected that attempts to filter—or blur—frames more thoroughly by trying to reduce the bit rate of the stream.

[0043] Method 600 includes generating a saliency map in block B604 that shows the edges depicted in a frame, using at least one of the surface normals or depth values ​​corresponding to the frame. For example, depth information and / or surface normal information may be held by the currently running application, and this information may be used to generate an edge map or saliency map using the edge map generator 104. In the embodiment, in addition to or instead of using depth information and / or surface normal information, frame data may be used to compare neighboring pixel values ​​and to determine edges from the sharp contrast in neighboring pixel values.

[0044] Method 600 includes, in block B606, computing pre-filtered frames using the selected DNN and at least in part on the data representing the salience map and frames. For example, after being trained by the training engine, the selected DNN 418 may be used to generate pre-filtered frames 112 using the frame data 108 and the edge map 106.

[0045] Method 600 includes, in block B608, transmitting data representing a pre-filtered frame to an end-user device. For example, the calculated pre-filtered frame 112 can then be encoded and compressed at a lower bit rate than the input frame—if an entropy that is never perfect is desired—and the encoded frame 404 can then be transmitted to the end-user device 420. As a result, the entropy of the frames can be adjusted so that the pre-filtered frame can be transmitted at the same frame resolution, rather than having to switch image resolutions due to channel conditions. In this way, the difference in pixel values ​​from frame to frame can be smaller as the entropy decreases, so the amount of data required to transmit the pre-filtered frame at its frame resolution can be reduced compared to the original frame data 108, thereby resulting in reduced latency.

[0046] Exemplary Content Streaming System Referring now to Figure 7, which is an exemplary system diagram of a content streaming system 700 according to some embodiments of the present disclosure. Figure 7 includes an application server 702 (which may include components, features, and / or functionality similar to the exemplary computing device 800 in Figure 8), a client device 704 (which may include components, features, and / or functionality similar to the exemplary computing device 800 in Figure 8), and a network 706 (which may be similar to the network described herein). In some embodiments of the present disclosure, the system 700 may be implemented. Application sessions may correspond to game streaming applications (e.g., NVIDIA GeFORCE NOW), remote desktop applications, simulation applications (e.g., autonomous or semi-autonomous vehicle simulations), computer-aided design (CAD) applications, virtual reality (VR) and / or augmented reality (AR) streaming applications, deep learning applications, and / or other application types.

[0047] In system 700, for an application session, the client device 704 can simply receive input data in response to input to the input device, send the input data to the application server 702, receive encoded display data from the application server 702, and display the display data on the display 724. As such, more computer-intensive calculations and processing are offloaded to the application server 702 (for example, rendering the graphics output of the application session—particularly rays or path tracing—is performed by the GPU of the game server 702). In other words, the application session is streamed from the application server 702 to the client device 704, thereby reducing the requirements of the client device 704 for graphics processing and rendering.

[0048] For example, with respect to the instantiation of an application session, client device 704 may display frames of the application session on display 724 based on receiving display data from application server 702. Client device 704 can receive input to one of the input devices and generate input data in response. Client device 704 can send input data to application server 702 via communication interface 720 and via network 706 (e.g., the Internet), and application server 702 can receive input data via communication interface 718. The CPU can receive the input data, process the input data, and send data to the GPU to cause the GPU to generate rendering of the application session. For example, the input data may represent the movement of the user's character in a game session of a game application, firing a weapon, reloading, passing a ball, rotating a vehicle, etc. The rendering component 712 can render an application session (for example, representing the results of input data), and the rendering capture component 714 can capture the rendering of the application session as display data (for example, as image data capturing the rendered frames of the application session). The rendering of the application session may include ray or path-trace illumination and / or shadow effects computed using one or more parallel processing units of the application server 702—for example, a GPU—which may further use one or more dedicated hardware accelerators or processing cores to perform ray or path-trace techniques. In some embodiments, one or more virtual machines (VMs)—for example, one or more virtual components, such as vGPUs, vCPUs, etc.—may be used by the application server 702 to support the application session.The encoder 716 can then encode the display data to produce encoded display data, which can be transmitted to the client device 704 via the network 706 through the communication interface 718. The client device 704 can receive the encoded display data via the communication interface 720, and the decoder 722 can decode the encoded display data to produce display data. The client device 704 can then display the display data through the display 724.

[0049] Exemplary computing devices Figure 8 is a block diagram of an exemplary computing device 800 suitable for use in implementations of some embodiments of the present disclosure. The computing device 800 may include an interconnection system 802 that directly or indirectly connects the following devices: memory 804, one or more central processing units (CPUs) 806, one or more graphics processing units (GPUs) 808, a communication interface 810, input / output (I / O) ports 812, input / output components 814, a power supply device 816, one or more presentation components 818 (e.g., displays), and one or more logical units 820. In at least one embodiment, the computing device 800 may include one or more virtual machines (VMs), and / or any of its components may include virtual components (e.g., virtual hardware components). As a non-restrictive example, one or more of the GPUs 808 may include one or more vGPUs, one or more of the CPUs 806 may include one or more vCPUs, and / or one or more of the logical units 820 may include one or more virtual logical units. As such, computing device 800 may include individual components (e.g., a complete GPU dedicated to computing device 800), virtual components (e.g., a portion of a GPU dedicated to computing device 800), or a combination thereof.

[0050] The various blocks in Figure 8 are shown as being connected via an interconnection system 802 having lines, but this is not intended to be limiting and is merely for clarity. For example, in some embodiments, a presentation component 818, e.g., a display device, could be considered an I / O component 814 (e.g., if the display is a touchscreen). In another example, the CPU 806 and / or GPU 808 could include memory (e.g., memory 804 could represent a storage device in addition to the memory of the GPU 808, CPU 806, and / or other components). In other words, the computing devices in Figure 8 are merely illustrative. Categories such as “workstation,” “server,” “laptop,” “desktop,” “tablet,” “client device,” “mobile device,” “handheld device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types are all intended to fall within the scope of the computing devices in Figure 8 and are therefore not distinguished.

[0051] The interconnection system 802 may represent one or more links or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnection system 802 may include one or more bus or link types, such as an industry standard architecture (ISA) bus, an extended industry standard architecture (EISA) bus, a video electronics standards association (VESA) bus, a peripheral component interconnect (PCI) bus, a peripheral component interconnect express (PCIe) bus, and / or other types of buses or links. In some embodiments, direct connections exist between components. For example, the CPU 806 may be directly connected to the memory 804. Furthermore, the CPU 806 may be directly connected to the GPU 808. When there are direct or point-to-point connections between components, the interconnection system 802 may include PCIe links to implement the connections. In these examples, the PCI bus does not need to be included in the computing device 800.

[0052] Memory 804 may include any of various computer-readable media. Computer-readable media may be any available media that can be accessed by computing device 800. Computer-readable media may include both volatile and non-volatile media, as well as removable and non-removable media. In practice, and not limited to, computer-readable media may include computer storage media and communication media.

[0053] Computer storage media may include both volatile and non-volatile media and / or removable and non-removable media, implemented in any method or technique for storing information, such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 804 may store computer-readable instructions (e.g., programs and / or program elements, such as an operating system). Computer storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD) or other optical disk storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other media that may be used to store desired information and may be accessed by computing device 800. In this specification, computer storage media does not include signals themselves.

[0054] Computer storage media include any information distribution medium that can implement computer-readable instructions, data structures, program modules, and / or other data types in a modulated data signal, such as a carrier wave or other transport mechanism. The term “modulated data signal” can mean a signal having one or more of its characteristics set or modified in the signal by a scheme for encoding information, etc. By example, and not limited to, computer storage media may include wired media, such as wired networks or direct wired connections, and wireless media, such as acoustic, RF, infrared, and other wireless media. Any combination of the foregoing should also be included in the scope of computer-readable media.

[0055] The CPU 806 may be configured to control one or more components of the computing device 800 and execute at least some computer-readable instructions for performing one or more of the methods and / or processes described herein. The CPU 806 may each include one or more cores (e.g., 1, 2, 4, 8, 28, 72, etc.) having the ability to process a large number of software threads simultaneously. The CPU 806 may include any type of processor, and may include different types of processors depending on the type of computing device 800 to which it is implemented (e.g., a processor with fewer cores for a mobile device and a processor with more cores for a server). For example, depending on the type of computing device 800, the processor may be an Advanced RISC Machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). The computing device 800 may include one or more CPUs 806 in addition to one or more microprocessors or supplemental coprocessors, such as numerical coprocessors.

[0056] In addition to, or instead of, the CPU 806, the GPU 808 may be configured to control one or more components of the computing device 800 to execute at least some computer-readable instructions for performing one or more of the methods and / or processes described herein. One or more of the GPU 808 may be an integrated GPU (for example, having one or more of the CPU 806), and / or one or more of the GPU 808 may be a separate GPU. In embodiments, one or more of the GPU 808 may be a coprocessor of one or more CPUs of the CPU 806. The GPU 808 may be used by the computing device 800 to render graphics (for example, 3D graphics) or to perform general-purpose computing. For example, the GPU 808 may be used for general-purpose computing on a GPU (GPGPU). It may be used for a GPU. The GPU808 may include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. The GPU808 may generate pixel data for an output image in response to rendering commands (for example, rendering commands from the CPU806 received via the host interface). The GPU808 may include graphics memory, such as display memory, for storing pixel data or any other suitable data, such as GPGPU data. Display memory may be included as part of memory 804. A GPU808 may include two or more GPUs operating in parallel (for example, via a link). The link can connect the GPUs directly (for example, using NVLINK) or via a switch (for example, using NVSwitch). When coupled together, each GPU808 can generate pixel data or GPGPU data for different parts of an output or for different outputs (for example, the first GPU for the first image and the second GPU for the second image). Each GPU may contain its own memory or share memory with other GPUs.

[0057] In addition to or instead of the CPU 806 and / or GPU 808, the logical unit 820 may be configured to control one or more components of the computing device 800 and execute at least some computer-readable instructions for executing one or more of the methods and / or processes described herein. In embodiments, the CPU 806, GPU 808, and / or logical unit 820 may execute any combination of methods, processes, and / or parts thereof separately or together. One or more of the logical units 820 may be a part of one or more of the CPU 806 and / or GPU 808 and / or integrated therein, and / or one or more of the logical units 820 may be separate components or otherwise external to the CPU 806 and / or GPU 808. In embodiments, one or more of the logical units 820 may be a coprocessor of one or more of the CPU 806 and / or GPU 808.

[0058] Examples of logic units 820 include one or more processing cores and / or components thereof, such as Data Processing Units (DPUs), Tensor Cores (TCs), Tensor Processing Units (TPUs), Pixel Visual Cores (PVCs), Vision Processing Units (VPUs), Graphics Processing Clusters (GPCs), Texture Processing Clusters (TPCs), Streaming Multiprocessors (SMs), Tree Traversal Units (TTUs), Artificial Intelligence Accelerators (AIAs), Deep Learning Accelerators (DLAs), Arithmetic-Logic Units (ALUs), Application-Specific Integrated Circuits (ASICs), and Floating Point Units (FPUs). Includes Units, input / output (I / O) elements, PCI or PCIe elements, and / or similar components.

[0059] The communication interface 810 may include one or more receivers, transmitters, and / or transceivers that enable computing device 800 to communicate with other computing devices via an electronic communication network, including wired and / or wireless communication. The communication interface 810 may include components and functionalities to enable communication over any of several different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth®, Bluetooth® LE, ZigBee®, etc.), wired networks (e.g., communicating via Ethernet® or InfiniBand), low-power wide-area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, the logic unit 820 and / or the communication interface 810 may include one or more data processing units (DPUs) to transmit data received via the network and / or via the interconnection system 802 directly to one or more GPUs 808 (e.g., their memory).

[0060] I / O port 812 may allow computing device 800 to be logically coupled to other devices, including I / O components 814, presentation components 818, and / or other components, some of which can be incorporated into (e.g., integrated into) computing device 800. Exemplary I / O components 814 include microphones, mice, keyboards, joysticks, gamepads, game controllers, satellite dishes, scanners, printers, wireless devices, etc. I / O components 814 may provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by the user. In some cases, the input may be sent to an appropriate network element for further processing. The NUI may implement any combination of speech recognition, stylus recognition, facial recognition, biometric recognition, gesture recognition both on and near the screen, air gestures, head and target tracking, and touch recognition related to the display of computing device 800 (as described in more detail below). The computing device 800 may include depth cameras for gesture detection and recognition, such as stereo camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof. In addition, the computing device 800 may include accelerometers or gyroscopes (for example, as part of an inertia measurement unit (IMU)) to enable motion detection. In some examples, the output of the accelerometer or gyroscope may be used by the computing device 800 to render immersive augmented reality or virtual reality.

[0061] The power supply device 816 may include a hardwired power supply device, a battery power supply device, or a combination thereof. The power supply device 816 can supply power to the computing device 800 to enable the components of the computing device 800 to operate.

[0062] The presentation component 818 may include a display (e.g., a monitor, touchscreen, television screen, heads-up display (HUD), other display types, or a combination thereof), a speaker, and / or other presentation components. The presentation component 818 can receive data from other components (e.g., a GPU 808, a CPU 806, a DPU, etc.) and output data (e.g., images, videos, sounds, etc.).

[0063] Exemplary data center Figure 9 shows an exemplary data center 900 that may be used in at least one embodiment of the present disclosure. The data center 900 may include a data center infrastructure layer 910, a framework layer 920, a software layer 930, and / or an application layer 940.

[0064] As shown in Figure 9, the data center infrastructure layer 910 may include a resource orchestrator 912, grouped computing resources 914, and node computing resources ("node CRs") 916(1) to 916(N), where "N" represents any complete, positive integer. In at least one embodiment, the node CRs 916(1) to 916(N) may include, but are not limited to, any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field-programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output (NW I / O) devices, network switches, virtual machines (VMs), power modules, and / or cooling modules. In some embodiments, one or more nodes CR916(1) to 916(N) may correspond to a server having one or more of the aforementioned computing resources. In addition, in some embodiments, nodes CR916(1) to 9161(N) may include one or more virtual components, such as vGPUs, vCPUs, and / or similar, and / or, one or more nodes CR916(1) to 916(N) may correspond to a virtual machine (VM).

[0065] In at least one embodiment, the grouped computing resources 914 may include a separate group of nodes CR916 housed in one or more racks (not shown), or a number of racks (also not shown) housed in data centers in various geographical locations. A separate group of nodes CR916 within the grouped computing resources 914 may include grouped compute, network, memory, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, several nodes CR916, including CPUs, GPUs, DPUs, and / or other processors, may be grouped in one or more racks to provide computing resources to support one or more workloads. One or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.

[0066] The resource orchestrator 912 can configure or otherwise control one or more nodes CR916(1) to 916(N) and / or grouped computing resources 914. In at least one embodiment, the resource orchestrator 912 may include a software design infrastructure (SDI) management entity for the data center 900. The resource orchestrator 912 may include hardware, software, or any combination thereof.

[0067] In at least one embodiment, as shown in Figure 9, the framework layer 920 may include a job scheduler 932, a configuration manager 934, a resource manager 936, and / or a distributed file system 938. The framework layer 920 may include a framework to support software 932 of the software layer 930 and / or one or more applications 942 of the application layer 940. The software 932 or application 942 may include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure, respectively. The framework layer 920 may be, but is not limited to, a type of free and open-source software web application framework, such as Apache Spark® ("Spark"), which can use the distributed file system 938 for large-scale data processing (e.g., "big data"). In at least one embodiment, the job scheduler 932 may include a Spark driver to facilitate scheduling of workloads supported by various layers of the data center 900. The configuration manager 934 may have the ability to configure different layers, for example, a software layer 930 and a framework layer 920 including Spark and a distributed file system 938 to support large-scale data processing. The resource manager 936 may have the ability to manage clustered or grouped computing resources mapped or allocated for support of the distributed file system 938 and the job scheduler 932. In at least one embodiment, the clustered or grouped computing resources may include computing resources 914 grouped in the data center infrastructure layer 910. The resource manager 936 can coordinate with the resource orchestrator 912 to manage these mapped or allocated computing resources.

[0068] In at least one embodiment, the software 932 included in the software layer 930 may include software used by at least a portion of nodes CR916(1) to 916(N), grouped computing resources 914, and / or the distributed file system 938 of the framework layer 920. One or more types of software may include, but are not limited to, internet web page search software, email virus scanning software, database software, and streaming video content software.

[0069] In at least one embodiment, the application 942 included in the application layer 940 may include one or more types of applications used by at least a portion of the nodes CR916(1) to 916(N), the grouped computing resources 914, and / or the distributed file system 938 of the framework layer 920. One or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments.

[0070] In at least one embodiment, any of the configuration manager 934, resource manager 936, and resource orchestrator 912 may implement any number and type of self-rewriting actions based on any amount and type of data obtained in any technically possible manner. Self-rewriting actions may free the data center operator of data center 900 from making potentially poor configuration decisions and potentially avoiding underutilized and / or underperforming parts of the data center.

[0071] Data Center 900 may include tools, services, software, or other resources for training one or more machine learning models or for predicting or inferring information using one or more machine learning models according to one or more embodiments described herein. For example, a machine learning model may be trained by calculating weight parameters by a neural network architecture using the software and / or computing resources described herein with respect to Data Center 900. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks may be used for inferring or predicting information using the resources described herein with respect to Data Center 900 by using weight parameters calculated via one or more training techniques, not limited to those described herein.

[0072] In at least one embodiment, the data center 900 may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, and / or other hardware (or corresponding virtual computing resources) for performing training and / or inference using the aforementioned resources. Furthermore, one or more of the aforementioned software and / or hardware resources may be configured as services that enable users to train or perform inference of information, such as image recognition, speech recognition, or other artificial intelligence services.

[0073] Exemplary network environment A network environment suitable for use in implementing the embodiments of this disclosure may include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. Each client device, server, and / or other device type (e.g., each device) may be implemented as one or more instances of the computing device 800 in Figure 8—for example, each device may include similar components, features, and / or functionalities of the computing device 800. In addition, if backend devices (e.g., servers, NAS, etc.) are implemented, they may be included as part of the data center 900, examples of which are further detailed herein with respect to Figure 9.

[0074] Components of a network environment may communicate with one another via the network, whether wired, wireless, or both. A network may include multiple networks, or a network of networks. For example, a network may include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks, such as the Internet and / or the Public Switched Telephone Network (PSTN), and / or one or more private networks. If a network includes a wireless telecommunications network, even components such as base stations, communication towers, or access points (like other components) may provide wireless connectivity.

[0075] Compatible network environments may include one or more peer-to-peer network environments—where servers may not be included in the network environment—and one or more client-server network environments—where one or more servers may be included in the network environment. In a peer-to-peer network environment, the functionality described herein with respect to the server may be implemented in any number of client devices.

[0076] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, or a combination thereof. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more of the servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework to support the software in the software layer and / or one or more applications in the application layer. The software or application may each include web-based service software or applications. In the embodiment, one or more of the client devices may use the web-based service software or applications (for example, by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a type of free and open-source software web application framework that may use a distributed file system for large-scale data processing (e.g., "big data").

[0077] A cloud-based network environment may provide cloud computing and / or cloud storage that implements any combination of the computing and / or data storage functions (or one or more of them) described herein. Any of these various functions may be distributed across multiple locations from a central or core server (e.g., one or more data centers that may be distributed state, territory, country, or world). If the connection to the user (e.g., client device) is relatively close to the edge server, the core server may delegate at least a portion of its functionality to the edge server. The cloud-based network environment may be private (e.g., limited to a single organization), public (e.g., available to multiple organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0078] A client device may include at least some of the components, features, and functionalities of the exemplary computing device 800 described herein with respect to Figure 8. As an example, and not limited to, a client device may be implemented as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, video camera, surveillance device or system, vehicle, ship, airship, virtual machine, drone, robot, handheld communication device, hospital device, gaming device or system, entertainment system, vehicle computer system, embedded system controller, remote control, instrument, consumer electronic device, workstation, edge device, any combination of these depicted devices, or any other suitable device.

[0079] This disclosure may be described in general terms with computer code or machine-usable instructions, including computer-executable instructions such as program modules, which are executed by a computer or other machine, such as a personal data assistant or other handheld device. Generally, a program module, including routines, programs, objects, components, and data structures, refers to code that performs a specific task or implements a specific abstract data type. This disclosure may be implemented in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, and more specialized computing devices. This disclosure may also be implemented in a distributed computing environment where tasks are performed by remote processing devices linked over a communication network.

[0080] In this specification, any “and / or” statement relating to two or more elements should be interpreted as meaning only one element or a combination of elements. For example, “element A, element B, and / or element C” may include element A only, element B only, element C only, element A and element B, element A and element C, element B and element C, or elements A, B, and C. In addition, “at least one of element A or element B” may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Furthermore, “at least one of element A and element B” may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0081] The subject matter of this disclosure is described herein with the specifics described herein in order to satisfy legal requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, it is intended that the subject matter of the claims may be carried out in other ways, including by the inventors, in conjunction with other current or future technologies, different steps or combinations of steps similar to those described herein. Furthermore, although the terms “step” and / or “block” may be used herein to imply different elements of the way in which they are used, these terms should not be construed as implying any particular order among the various steps described herein unless the order of the individual steps is expressly stated, and unless it is so.

Claims

1. Using at least one of the surface normal information corresponding to the frame or the depth information corresponding to the frame, a salience map is generated showing the edges depicted within the frame. Using a deep neural network (DNN) and based at least partially on the salience map and the data representing the frame, calculate a pre-filtered frame, and The data representing the pre-filtered frame is sent to the end-user device. Processing circuit for A processor equipped with the following features.

2. The processor according to claim 1, wherein the saliency map includes one of a binary image or a grayscale image and the pixel values ​​of the saliency map, and the magnitude of the pixel value of a pixel corresponding to an edge can be distinguished from the magnitude of the pixel value of a pixel that does not correspond to an edge.

3. The processor according to claim 1, further comprising a processing circuit for encoding the pre-filtered frame to generate an encoded frame, wherein the data representing the pre-filtered frame is data representing the encoded frame.

4. Monitor one or more network or system conditions related to the end-user device. Based at least partially on one or more of the aforementioned network or system conditions, the value of the entropy control parameter is determined, and Based at least partially on the aforementioned values, select the DNN from a plurality of DNNs corresponding to each of the values ​​of the entropy control parameters. The processor according to claim 1, further comprising a processing circuit for the purpose of

5. The depth information corresponds to a depth map, the surface normal information corresponds to a surface normal map, and the generation of the salience map is At least one of scaling or normalization of at least one of the depth map or the surface normal map for generating one or more first intermediate maps, Performing morphological closing with respect to the one or more first intermediate maps in order to generate one or more second intermediate maps, and To generate the aforementioned salience map, an edge detection algorithm is performed with respect to one or more second intermediate maps. The processor according to claim 1, including the following:

6. The processor according to claim 5, wherein the edge detection algorithm includes a Sobel operator.

7. The processor according to claim 1, wherein the DNN is trained using a first loss function corresponding to preserving edge detail and a second loss function corresponding to the amount of filtering.

8. The processor according to claim 1, wherein the DNN includes an autoencoder architecture.

9. The processor according to claim 1, wherein the calculation of the pre-filtered frame is further based at least in part on the application of an entropy control parameter value to the DNN, and further, the amount of filtering of the pre-filtered frame is at least in part on the entropy control parameter value.

10. The aforementioned processor, A system for performing simulation operations. A system for performing deep learning operations. Systems implemented using edge devices, A system that incorporates one or more virtual machines (VMs). A system implemented at least partially in a data center, or A system implemented using cloud computing resources, at least partially. The processor according to claim 1, which is included in at least one of the following.

11. One or more processing units, Selecting a DNN from multiple deep neural networks (DNNs) based at least partially on at least one of the network or system conditions corresponding to the end-user device. To generate a salience map showing the edges depicted within the frame, using at least one of the surface normals or depth values ​​corresponding to the frame. Using the selected DNN and at least partially based on the salience map and the data representing the frame, calculate the pre-filtered frame, and To transmit data representing the pre-filtered frame to the end-user device. When an operation including the above is performed by one or more processing units, one or more memory units that store the instructions to be executed by the one or more processing units and A system that includes these features.

12. The above operation further, Determining the value of the entropy control parameter, at least in part, based on analyzing the aforementioned network or system conditions. Includes, The system according to claim 11, wherein the selection of the DNN from the plurality of DNNs is at least partially based on the DNN including the respective values ​​corresponding to the entropy control parameters.

13. The system according to claim 11, wherein the saliency map includes a binary image comprising a first pixel having a first value and a second pixel having a second value, and the first pixel corresponds to the edge.

14. The system according to claim 11, wherein the operation further includes encoding the prefiltered frame to generate an encoded frame, the data representing the prefiltered frame being data representing the encoded frame.

15. The system according to claim 11, wherein each of the plurality of DNNs includes the respective values ​​of the entropy control parameter.

16. The depth value is encoded in the depth map, the surface normal is encoded in the surface normal map, and the generation of the salience map is At least one of scaling or normalization of at least one of the depth map or the surface normal map for generating one or more first intermediate maps, Performing morphological closing with respect to the one or more first intermediate maps in order to generate one or more second intermediate maps, and To generate the aforementioned salience map, an edge detection algorithm is performed with respect to one or more second intermediate maps. The system according to claim 11, including the following:

17. The system according to claim 16, wherein the edge detection algorithm includes a Sobel operator.

18. The aforementioned system A system for performing simulation operations. A system for performing deep learning operations. Systems implemented using edge devices, A system that incorporates one or more virtual machines (VMs). A system implemented at least partially in a data center, or A system implemented using cloud computing resources, at least partially. The system according to claim 11, which is included in at least one of the following.

19. A method performed by a processor, Using at least one of the surface normals or depth values ​​corresponding to the frame, a salience map is generated showing the edges depicted within the frame. Calculating pre-filtered frames using a deep neural network (DNN) and based at least partially on the salience map and the data representing the frames, A first loss value is calculated using a first loss function, at least in part, based on comparing a first pixel value of the pre-filtered frame with a second pixel value of the frame at a pixel position corresponding to an edge, as determined from the salience map. A second loss value is calculated using a second loss function, at least in part, based on comparing the pixel values ​​of adjacent pixels within the pre-filtered frame. Updating one or more parameters of the DNN based at least partially on the first loss value and the second loss value. Methods that include...

20. The method according to claim 19, further comprising calculating a total loss value using a weighted combination of the first loss value and the second loss value, wherein updating the one or more parameters is at least partially based on the total loss value.

21. The method according to claim 19, wherein the calculation of the second loss value is at least partially based on the associated entropy control parameter value of the DNN, and one or more other DNNs include entropy control parameter values ​​different from the associated entropy control parameter value.

22. The method according to claim 19, wherein the DNN is trained using unsupervised learning.

23. The method according to claim 19, wherein for at least one training frame used to train the DNN, a corresponding saliency map is generated using an edge detection algorithm with respect to the at least one training frame.

Images