Context aware error concealment for improved reasoning accuracy

Abstract

The invention relates to context-aware error concealment for improved reasoning accuracy. In various examples, systems and methods related to context-aware error concealment are disclosed to improve reasoning accuracy. The system may identify a frame of the video stream and determine that the frame contains corrupted or lost data. The system may generate a corrected frame by applying an error concealment function selected based on a location of corrupted or lost data in the frame and a region of interest in the frame.

Description

Background Technology

[0001] Video streaming involves encoding video data and transmitting it over a network to a remote client device, where it is then decoded. A drawback of video streaming is the potential for data loss during transmission, which can be caused by factors such as network latency and packet loss. Packet loss can lead to severe artifacts or latency, negatively impacting the overall streaming experience. Corrupted or missing data in the video stream can also affect the accuracy of AI's predictive / inference operations on the stream. Summary of the Invention

[0002] Embodiments of this disclosure relate to context-aware video stream error hiding techniques for improving the accuracy of artificial intelligence (AI) operations. The systems and methods described herein improve upon conventional error hiding techniques by selectively applying error hiding based on the location of detected video data corruption / loss. Unlike conventional methods, which apply computationally expensive error hiding functions to every part of every frame, thus degrading AI performance, the techniques described herein can selectively apply error hiding to the frame portions most relevant to downstream AI operations. In some implementations or scenarios, error hiding may not be applied even if corruption or data loss is detected. This significantly reduces the computational resources required to apply error hiding to video stream data, thereby improving the performance of downstream AI operations.

[0003] At least one aspect involves one or more processors. The one or more processors may include one or more circuits. The one or more circuits can identify frames of a video stream. The one or more circuits can determine that the frame contains corrupted or missing data. The one or more circuits can generate a corrected frame by applying an error-hiding function selected at least based on the location of the corrupted or missing data in the frame and / or the region of interest in the frame.

[0004] In some implementations, one or more circuits may select an error concealment function based at least on the location of corrupted or missing data within the region of interest in a frame. In some implementations, one or more circuits may generate a corrected frame by applying the error concealment function to at least a portion of the region of interest. In some implementations, one or more circuits may apply the error concealment function by using the frame as input to a machine learning model.

[0005] In some implementations, one or more circuits may receive an encoded bitstream of the video stream. In some implementations, one or more circuits may generate frames by decoding the encoded bitstream, where decoding the frames indicates the location of corrupted or lost data. In some implementations, the error concealment function is the first of a plurality of error concealment functions. In some implementations, one or more circuits may determine that an object has been detected in a predetermined number of previous frames in the video stream.

[0006] In some implementations, one or more circuits may select a first error concealment function from a plurality of error concealment functions based at least on the detection of objects in a predetermined number of previous frames. The first error concealment function can utilize more computational resources compared to a second error concealment function among the plurality of error concealment functions. In some implementations, one or more circuits may generate corrected frames by applying the first error concealment function to at least a portion of the frames in which the object is estimated to exist. In some implementations, one or more circuits may determine that corrupted or lost data is located outside the region of interest. In some implementations, one or more circuits may select a first error concealment function from a plurality of error concealment functions based at least on the fact that corrupted or lost data is located outside the region of interest; the first error concealment function uses fewer computational resources compared to a second error concealment function among the plurality of error concealment functions.

[0007] At least one aspect relates to a system. The system may include one or more processors. The system can receive a request to process a video stream comprising multiple frames. The system can apply an error-hiding function to at least one of the multiple frames, the error-hiding function being selected based at least on the location of corrupted or missing data in the at least one frame and a region of interest in the at least one frame. The system can use the at least one frame as input to the request to execute a machine learning model.

[0008] In some implementations, the system may use a decoding process to determine that the at least one frame contains corrupted or missing data. In some implementations, a machine learning model generates an indication of objects in the at least one frame. In some implementations, the system may select a second error concealment function for at least one second frame of a plurality of frames, based at least on the indication of objects in the at least one frame.

[0009] In some implementations, a second error concealment function is further selected for at least one second frame based on the expected position of the object in at least one second frame. In some implementations, the location of corruption or data loss is either a macroblock location or a slice location in at least one frame. In some implementations, the system can apply an error correction function by executing a second machine learning model that takes at least one frame as input, and this second machine learning model is used to generate replacement information for corrupted or missing data in at least one frame.

[0010] At least one aspect relates to a method. The method may include: using one or more processors to identify frames of a video stream. The method may include: using one or more processors to determine that the frame contains corrupted or missing data. The method may include: using one or more processors to generate a corrected frame by applying an error-hiding function selected at least based on the location of the corrupted or missing data in the frame and a region of interest in the frame.

[0011] In some implementations, the method includes using one or more processors to select an error concealment function based at least on the location of the corrupted or lost data within the region of interest in the frame. In some implementations, the method includes using one or more processors to generate a corrected frame by applying the error concealment function to at least a portion of the region of interest.

[0012] The processors, systems, and / or methods described herein can be implemented by or included in at least one of the following: a control system for autonomous or semi-autonomous machines; a perception system for autonomous or semi-autonomous machines; a system for performing analog operations; a system for performing digital twin operations; a system for performing optical transmission simulation; a system for performing collaborative content creation of 3D assets; a system for performing deep learning operations; a system for performing generative AI operations using a large language model (LLM); a system for performing generative AI operations using a visual language model (VLM); a system implemented using an edge device; a system implemented using a robot; a system for performing conversational AI operations; a system for generating synthetic data; a system containing one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources. Attached Figure Description

[0013] The system and method for context-aware error hiding to improve inference accuracy will be described in detail below with reference to the accompanying drawings, wherein:

[0014] Figure 1 This is a block diagram of an example system for implementing context-aware error hiding to improve reasoning accuracy, according to some embodiments of the present disclosure.

[0015] Figure 2 An example data flow diagram according to some embodiments of the present disclosure is shown, illustrating how video data captured from a security device can be processed according to the context-aware error-hiding techniques described herein;

[0016] Figure 3 These are example diagrams based on some embodiments of the present disclosure, illustrating how corrupted or missing encoded data affects video decoding;

[0017] Figure 4 These are example diagrams based on some embodiments of the present disclosure, illustrating how corrupted or lost data can occur in video frame portions outside and inside the region of interest;

[0018] Figure 5 These are example diagrams illustrating large segments of missing or corrupted data in a display frame according to some embodiments of this disclosure;

[0019] Figure 6 This is a flowchart illustrating an example of a method for implementing context-aware error hiding to improve inference accuracy according to some embodiments of this disclosure;

[0020] Figure 7 This is a block diagram of an exemplary content streaming media system suitable for implementing some embodiments of the present disclosure;

[0021] Figure 8 This is a block diagram of an exemplary computing device suitable for implementing some embodiments of the present disclosure; and

[0022] Figure 9 This is a block diagram of an exemplary data center suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0023] This disclosure relates to systems and methods for implementing context-aware error hiding in video streams, which can be applied to improve the accuracy of machine learning operations. The error hiding techniques described herein can be performed by systems transmitting video data or data derived from video data, including security systems, artificial intelligence pipelines, or general-purpose video streaming platforms.

[0024] These techniques are particularly useful in unreliable or noisy networks, where data packets may be lost or transmitted data may be corrupted. Video streaming can be performed by transmitting data packets using streaming protocols such as Real-time Streaming Protocol (RTSP). Such streaming protocols transmit video data encoded as a “bitstream” to improve throughput and accuracy. If parts of the bitstream are corrupted or lost, the corresponding video frame information is also lost during decoding at the video stream receiver. In artificial intelligence systems that perform inference on the decoded video data, missing or corrupted frames can affect the accuracy of detection or tracking.

[0025] Traditional error-hiding methods attempt to mitigate the impact of corruption or data loss, but this leads to degraded inference performance because high-quality, computationally intensive error-hiding techniques are applied to every frame in the video stream where corrupted or lost data occurs. To address these shortcomings, the system and method described in this paper dynamically select from different error-hiding methods based on the location of the lost or corrupted frame data. In some implementations, error-hiding may not be applied even if corruption or data loss is detected.

[0026] Therefore, the systems and methods described herein can use regions of interest (ROIs) in a frame that correspond to the most relevant portion of the frame for inference detection, classification, segmentation, or other machine learning tasks. These ROIs can be predetermined or dynamically determined and can correspond to the portion of the video stream most likely to depict an object affected by artificial intelligence operations. For example, a ROI could be a portion near the center of a video frame. To improve overall performance, the systems and methods described herein can select the optimal error-hiding technique based on whether data corruption or loss in the video frame occurs in or near any ROI within the video frame.

[0027] In one example, if the corruption or data loss occurs outside the region of interest in a video frame, a computationally less computationally burdensome error-hiding technique can be chosen. In some implementations, error hiding may not be performed if the data loss or corruption occurs outside the region of interest. In another example, if the corruption or data loss occurs within the region of interest in a video frame, a high-quality, computationally intensive error-hiding technique can be applied to improve the accuracy of machine learning inference. An example of computationally intensive error hiding is using a generative AI model that takes a corrupted frame as input and generates an uncorrupted video frame as output.

[0028] The error-hiding technique chosen for a video frame may depend on whether an object has been detected in one or more regions of interest (ROIs) in the video stream recently. For example, if no object has been detected in a ROI within a predetermined number of recent consecutive frames, a computationally less demanding error-hiding technique can be chosen. In another example, if an object has been detected in one or more ROIs within a predetermined number of recent consecutive frames, a high-quality, computationally intensive error-hiding technique can be applied.

[0029] When aggregated data is lost or corrupted, error concealment techniques can also be selectively applied to different portions of a video frame (e.g., macroblocks). For example, if data packets corresponding to slices of video data are lost or corrupted, error concealment might be applied only to portions of the video frame corresponding to one or more regions of interest (e.g., macroblocks), rather than the entire video frame. A similar approach can be used to selectively apply error concealment to portions of the regions of interest where objects are estimated to appear, at least based on the detection of objects in previous video frames.

[0030] refer to Figure 1 , Figure 1 This is an example computing environment according to some embodiments of the present disclosure, which includes a system for implementing context-aware error hiding to improve inference accuracy. It should be understood that such and other arrangements described herein are merely illustrative examples. Other arrangements and elements (e.g., machines, interfaces, functions, commands, function groups, etc.) may be used in addition to the arrangements and elements shown, or other arrangements and elements may be used instead of the arrangements and elements shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components, or in combination with other components, and can be implemented in any suitable combination and location. The various functions performed by the entities described herein can be performed by hardware, firmware, and / or software. For example, various functions can be implemented by a processor executing instructions stored in memory.

[0031] System 100 is shown to include a data processing system 102 that receives encoded video data 112, which may be an encoded bitstream of a video stream with encoded frames 113. Data processing system 102 may implement various techniques described herein to hide errors in the encoded video data 112, thereby improving inference accuracy. Data processing system 102 may receive the encoded video data 112, for example, from one or more computer networks. In some implementations, data processing system 102 may access the encoded video data from a data repository or storage system. The storage system may be an external server, a distributed storage / computing environment (e.g., a cloud storage system), or any other type of storage device or system communicating with data processing system 102. In some implementations, the storage system may be part of data processing system 102 or may be located internally to data processing system 102. In such implementations, data processing system 102 may access the encoded video data 112 from internal memory.

[0032] Data processing system 102 may access, retrieve, or otherwise receive encoded video data 112 in response to a request to process it. This request may be provided by a device located outside and communicating with data processing system 102 (e.g., a client device communicating over a network). In some implementations, the request may be provided in response to input at data processing system 102, such as by an operator of data processing system 102. The request may specify the encoded video data 112 to be processed or specify the location from which the data processing system will retrieve the encoded video data 112.

[0033] Encoded video data 112 is shown as comprising one or more encoded frames 113. Encoded video data 112 can be an encoded bitstream of video data generated by one or more video capture devices (e.g., cameras, security cameras, integrated webcams, or smartphone cameras) or applications that generate video frames (e.g., remote gaming applications, remote desktop applications, etc.). Encoded video data 112 can be generated from any suitable source, including video playback processes, game playback processes (e.g., video output from a remotely executed video game), and other video data sources. Encoded video data 112 may contain information related to video stream frames, such as the resolution, frame rate, or other attributes of the video stream. Encoded video data 112 can be encoded according to any suitable codec standard, including but not limited to H.264 (AVC), H.265 (HEVC), H.266 (VVC), AV1, VP8, VP9, ​​and other codec standards, or any other video codec that supports segmenting video frames into different geometric regions (e.g., slices or macroblocks). The frame data in the encoded video data 112 can be encoded into one or more encoded frames 113.

[0034] Encoded video frame 113 is shown to be stored as part of encoded video data 112. The decoder can be or includes software, hardware, or a combination of both. Each frame in the encoded video data 112 can be an independent encoded data unit and can be compressed using various techniques to reduce redundancy between consecutive frames. Encoded frame 113 and / or encoded video data 112 can contain metadata describing the structure and attributes of each frame, such as frame type (I-frame, P-frame, B-frame), size, timestamp, and dependencies on other frames. In some implementations, the data of encoded frame 113 can be stored sequentially or interleaved as part of encoded video data 112.

[0035] Data processing system 102 can execute decoder 104 to decode encoded video data 112. To do this, decoder 104 can parse encoded video data 112 to extract any relevant video metadata, such as frame size, frame rate, and audio sampling rate. Decoder 104 can identify the codec based at least on the metadata and use the identified codec to decode encoded video data 112 to generate frames of video stream and / or audio data. In some implementations, such video metadata can be accessed separately from encoded video data 112 (e.g., stored separately or provided separately in one or more network packets, etc.). Decoding encoded video data 112 can include decompressing or performing the inverse operation of any encoding operation used to generate encoded video data 112.

[0036] Decoded data generated from encoded video data 112 is stored as decoded frames 110. Decoded frames 110 may include the original video frame data (e.g., a set of pixels at the video stream resolution). Decoded frames 110 may be stored in the frame buffer or other memory area of ​​the data processing system 102 for processing by the machine learning model 118 and / or the error hider 108. As shown, in this example, decoded frames 110 are provided to the error hider 108. In some implementations, each decoded frame 110 may be processed sequentially by the error hider 108. In some implementations, decoded frames 110 may be processed in batches by the error hider 108 (e.g., once a predetermined number of decoded frames 110 have been generated).

[0037] Decoder 104 is shown to include error detector 106. Error detector 106 may include software, hardware, or a combination of hardware and software, and can be used to detect errors in encoded frames 113 when decoding encoded video data 112. Error detector 106 can detect errors caused by data corruption or loss in encoded video data 112. Since encoded video data 112 may contain streaming video data from one or more video capture systems, some portions of encoded video data 112 may be corrupted or lost during transmission. In implementations where encoded video data 112 is stored in a repository or storage system, corruption may occur in the repository / storage system or during transmission to the repository / storage system.

[0038] Corruption in the encoded video data 112 may include the loss of one or more portions of the encoded frames 113, resulting in the inability to correctly reconstruct the information in the frames. Error detector 106 can use various techniques, such as parity checking, checksums, or error detection methods implementing Cyclic Redundancy Check (CRC), to detect these portions. In some implementations, the encoded video data 112 may be or contain one or more portions of an encoded bitstream, and error detector 106 can detect instances of missing portions of the encoded frame 113 by identifying missing portions of the encoded bitstream corresponding to the encoded frame 113. For example, while decoder 104 is decoding the encoded video data 112, error detector 106 can scan the bitstream for inconsistencies or anomalies indicating corruption. Doing so may include accessing the data corresponding to the encoded frame 113 to detect deviations from the expected pattern, including but not limited to frame size differences, unexpected interruptions in the encoded bitstream, or incorrect checksum values.

[0039] Any encoded frame 113 detected by error detector 106 as containing missing or corrupted data can be marked for error hiding. These markers can be stored as part of the metadata contained in the decoded frame 110. In some implementations, the decoded frame 110 marked as containing corrupted or missing data can be stored along with an indication of corruption / data loss. In some implementations, error detector 106 can determine the location of corruption or missing data within the decoded frame 110 (e.g., pixel coordinates, region coordinates, identifiers of one or more macroblocks, slices, or other geometric portions, etc.). The location of any corrupted or missing regions in the decoded frame 110 can be provided to error hider 108 as metadata of the decoded frame 110.

[0040] After generating decoded video data for video frames, decoder 104 can provide any decoded frame 110 containing corrupted or missing data to error hider 108. Error hider 108 can include software, hardware, or a combination of both, and can be executed to hide errors detected in decoded frame 110 using a suitable error hiding function. Error hider 108 can implement any suitable type of error hiding function to hide regions of corrupted or missing data in decoded frame 108, including but not limited to spatial hiding (e.g., interpolation), temporal hiding (e.g., replacing corrupted / missing pixels with corresponding pixels from previous frames), corrupted or missing data prediction based on motion vector estimation, generative machine learning techniques, or combinations thereof.

[0041] Error hider 108 can dynamically select one of a number of error hiding functions for decoded frame 110 containing corrupted or missing data, based at least on the location of the corrupted or missing data in the decoded frame 110 and the region of interest (ROI) in the decoded frame 110. The ROI can be a predetermined or dynamically identified pixel region in the decoded frame 110. The size, location, or other attributes of the ROI can be identified in the metadata of the encoded video data 112, extracted by decoder 104, and provided to error hider 108 for processing. In some implementations, the size, location, or other attributes of the decoded frame 110 can be provided frame-by-frame in the metadata of the encoded video data 112, where each encoded frame 113 contains metadata identifying the corresponding ROI.

[0042] In some implementations, the region of interest (ROI) can be determined at least based on the source of the encoded video data 112. In one example, the data processing system 102 may store a data structure that maps different sources of encoded video data 112 (e.g., different capture devices, video data storage locations, client devices, etc.) to corresponding metadata associated with the ROI. After receiving or identifying the encoded video data 112, the error hider 108 can access this data structure using the source of the encoded video data 112 to identify the size, location, or attributes of the ROI of that video source. In some implementations, the ROI may be discontinuous, and the decoded frame 110 may contain multiple consecutive ROIs. The ROI may differ for different sequences of decoded frames 110. In some implementations, the size or location of the ROI of the decoded frame 110 can be modified based at least on the output of one or more machine learning models 118, as described further in detail herein.

[0043] The type of error-hiding function used to hide lost or corrupted data in decoded frame 110 can be selected based on conditions including, but not limited to: the location and / or size of the lost or corrupted data in decoded frame 110; whether the lost or corrupted data portion in decoded frame 110 is located within the region of interest in decoded frame 110 (or, in some implementations, prior to the region of interest); and / or whether one or more objects were detected within the region of interest in one or more previous frames. Figure 3 , Figure 4 and Figure 5 Various examples of different conditions for selecting different error hiding functions are described.

[0044] In combination Figure 1 Reference in the context of the component described in Figure 3 , Figure 3 An example schematic diagram 300 according to some embodiments of the present disclosure is shown, illustrating how corrupted or missing encoded data affects video decoding. As shown, frame 302 includes a region of interest 304 and a region of missing or corrupted data 306. Although the region of interest 304 is marked with a horizontal line pattern, it should be understood that the region of interest 304 is not necessarily visually represented in frame 302. Furthermore, although the region of interest 304 is shown as a rectangle, it should be understood that any number of regions of interest 304 can be associated with frame 302, and the regions of interest 304 of frame 302 can have any size, shape, or position.

[0045] As shown in the figure, in this example, frame 302 contains a missing or corrupted data region 306, which is located below and outside the region of interest 304. In this example, the error concealer 108 can choose to compute an error concealment function with a relatively low impact, which may result in a decrease in the quality of the region corresponding to the missing or corrupted data after the error concealment function is applied. In this example, since the missing or corrupted data region 306 is located outside the region of interest, the quality of this part in frame 302 is not necessarily related to downstream inference operations (e.g., object detection, segmentation, etc.). Therefore, a computationally less demanding error concealment function can be used at the expense of the output quality of the corrected region to dynamically improve the computational performance of the decoding process.

[0046] Non-limiting examples of alternative low-complexity error-hiding techniques include low-complexity interpolation techniques (e.g., spatial hiding) or temporal hiding techniques. Temporal hiding techniques may include replacing lost or corrupted data regions 306 with corresponding pixels from previously decoded frames 110 stored in error hider 108. In some implementations, error hider 108 may not apply an error-hiding function, instead allowing decoded frames 110 to be processed according to the techniques described herein without modification. Further details regarding the selection criteria for other error-hiding functions will be discussed in conjunction with... Figure 4 and Figure 5 Describe it.

[0047] review Figure 1 Error concealer 108 can apply a selected error concealment function to decoded frames 110 to generate a sequence of corrected frames 114. Corrected frames 114 can contain modified data from decoded frames 110 generated by the selected error concealment function. Corrected frames 114 can be generated in the same order as decoded frames 110. Any generated corrected frames 114 can be interleaved with decoded frames 110 that do not have any corrupted or missing data, ensuring that the frame sequence generated by decoder 104 maintains the correct temporal order of the video stream.

[0048] As described herein, frames of encoded video data 112 may correspond to security cameras, video capture devices, or other video generation software, and may depict various objects or features of interest. Regions of interest in decoded frame 110 may correspond to regions in decoded frame 110 where objects or features of interest may appear. Corrected frames 114 (potentially including decoded frames 110 without error hiding) are provided to downstream processing tasks to detect the presence of one or more objects of interest. In this example, corrected frames 114 are provided as input to one or more machine learning models 118.

[0049] One or more machine learning models 118 may be or include any type of machine learning model that has been trained / updated to process frames of a video stream (e.g., decoded frame 110, corrected frame 114, etc.). For example, one or more machine learning models 118 may include one or more neural networks (e.g., deep neural networks (DNNs), convolutional neural networks (DNNs), recurrent neural networks (RNNs), fully connected networks, and combinations thereof) that have been trained / updated to perform machine learning tasks such as image classification, segmentation, or object detection. In some implementations, one or more machine learning models 118 may be or include other types of machine learning models, such as linear or logistic regression models, decision tree models, or support vector machine models, that process various data from decoded frame 110 and / or corrected frame 114.

[0050] In some implementations, the data processing system 102 can process one or more corrected frames 114 and / or decoded frames 110 (if error correction is not required) while generating one or more corrected frames 114 and / or decoded frames 110 (if error correction is not required) to detect one or more objects or features of interest located within one or more regions of interest in the corresponding frames. In some implementations, pixels corresponding to the regions of interest can be extracted from the frame and then fed as input to machine learning models(s) 118. In some implementations, the entire frame can be fed as input to machine learning models(s) 118. In an example where machine learning models(s) 118 contain neural networks, the data processing system 102 can execute machine learning models(s) 118 by feeding the input data to be processed to one or more input layers or input data structures of machine learning models(s) 118 and performing processing operations at each layer until one or more model outputs 120 are generated.

[0051] As described herein, machine learning model 118 may include one or more object detection models for processing regions of interest (ROIs) of at least the corrected frame 114 and / or the decoded frame 110 (if error correction is not required). In such implementations, model output 120 may include an indication of whether one or more objects of interest (ROIs) are detected in the ROI of the frame. Such an indication may include, but is not limited to, flags indicating the presence of one or more ROIs / features, bounding box data of one or more ROIs / features, and / or classifications of one or more ROIs / features detected in the frame. Model output 120 may be stored in association with the frame on which model output 120 is based. In some implementations, model output 120 and / or any corresponding frame may be provided to other downstream processing systems or processes. Model output 120 may be provided to one or more external computing systems or stored in one or more data repository / storage systems.

[0052] In some implementations, one or more model outputs 120 can be provided as input to the error concealer 108. For example, model outputs 120 indicating the detection of an object / feature of interest in a frame can be used to select an error concealment function for subsequent frames in the decoded frame sequence 110. In some implementations, the data processing system 102 can initialize or otherwise store / maintain one or more counters for tracking the number of consecutive corrected frames 114 depicting the object / feature of interest within a corresponding region of interest. These counters can be provided to or implemented by the error concealer 108 to dynamically select different error concealment functions, thereby improving the accuracy of downstream inference operations performed using machine learning models. Figure 4 and Figure 5 Various examples of different methods for selecting an error hiding function for decoded frame 110 are described.

[0053] In combination Figure 1 Reference in the context of the component described in Figure 4 , Figure 4 Example schematic diagram 400 is shown, illustrating how data corruption or loss can occur in multiple portions outside and inside a region of interest (ROI) in a video frame according to some embodiments of this disclosure. As shown, frame 402 includes a ROI 404 and lost or corrupted data regions 406A and 406B (sometimes collectively referred to as "lost or corrupted data region 406"). In some implementations, frame 402 may contain one or more objects / features of interest (408) (shown here as human silhouettes).

[0054] In some implementations, the error hider 108 can select an error hiding function based on whether the corrupted or missing data 406 is located before or inside the region of interest 404. In this scenario, different error hiding functions can be selected based on whether the object of interest 408 was detected within the region of interest in one or more previous frames. For example, if the corrupted or missing data 406 is located before or inside the region of interest 404, and the object of interest / feature 408 was not detected in one or more previous frames, a less computationally intensive error hiding function can be selected. Since the object of interest 408 was not detected in previous frames, it is not necessarily expected to appear in the current frame 402. Therefore, a less accurate, less computationally intensive error hiding function can be applied to frame 402 to resolve major artifacts and improve the overall visualization. Using an error hiding function that does not require significant computational power can improve the performance of the entire computational system. Furthermore, since the absence of an object of interest / feature 408 necessarily occurs within the region of interest 404 (based on its absence in a predetermined number of previous consecutive frames), the inference accuracy is unlikely to decrease even if the overall quality of frame 402 is affected by the corrupted or missing data 406.

[0055] This approach can be used even if the corrupted or missing data 406 appears before the region of interest 404 (e.g., in video data generated before decoding). In one example, the encoded macroblock data of frame 402 can be decoded line by line from left to right, starting from the top left corner of frame 402 and ending at the bottom right corner. In this example, a portion of the corrupted or missing data 406A is located in a line before the macroblocks constituting the region of interest 404. Since the decoding process may be sequential, information decoded from the corrupted or missing portion of data 406A may affect the encoded macroblocks decoded as part of the region of interest 404 in subsequent frames. In this scenario, hiding the error in the portion of the corrupted or missing data 406A can improve the quality of subsequent frames, thereby improving the overall inference accuracy of the entire frame sequence. In some implementations, the error hiding function can be selected based on the proximity of any corrupted or missing data 406 to the region of interest 404. In another example, if a portion of corrupted or missing data 406 appears in the same row as any macroblock in region of interest 404, then for the purpose of selecting the error hiding function, corrupted or missing data 406 can be considered to be located in region of interest 404.

[0056] In another example, if corrupted or missing data 406 is located before or within the region of interest 404, and the object of interest / feature 408 has been detected in a predetermined number (e.g., four) of previous consecutive frames, a highly computationally intensive error-hiding function can be selected to improve the quality of frame 402. Since the object of interest / feature 408 was detected in the predetermined number of previous frames, the same object of interest / feature 408 is also likely to appear in the region of interest 404 of the current frame 402. To ensure that the machine learning model 118 detects or otherwise accurately processes the object of interest / feature 408, a more accurate, computationally intensive error-hiding function can be applied to frame 402 to resolve major artifacts and improve overall visualization. Non-limiting examples of error hiding include: predicting pixel values ​​for missing / corrupted macroblocks using pixel data and / or motion vectors from previous frames; performing artificial intelligence models (e.g., machine learning models, deep learning models, generative artificial intelligence models, etc.) on certain portions of frame 402 to replace missing / corrupted data; and / or hybrid error-hiding techniques (e.g., a combination of temporal and spatial error hiding), etc. Selectively applying error-hiding functions that use relatively more computational resources for certain frames, while using methods with lower accuracy and lower resource intensity when the object of interest / feature 408 is unlikely to be detected, can improve the system's performance when processing multiple consecutive frames of a video stream.

[0057] In combination Figure 1 Reference in the context of the component described in Figure 5 , Figure 5 An example schematic diagram 500 according to certain embodiments of the present disclosure is shown, illustrating a large segment of lost or corrupted data in a frame. As shown, frame 502 includes the top of a lost / corrupted region of interest 504A, the bottom of an unlost / corrupted region of interest 504B, and lost or corrupted data 506 in a larger area. The top 504A and bottom 504B of the region of interest may be collectively referred to as "region of interest 504". In some implementations, frame 502 may contain one or more objects / features of interest 508 (shown here as the outline of a person).

[0058] In some implementations, error hider 108 can select an error hiding function based on the amount of missing / corrupted data in frame 502. In this example, at least one data slice represents a missing / corrupted portion of frame 502, including the top of region of interest 504A. When the amount of missing / corrupted data in frame 502 is relatively large, error hider 108 can select an error hiding function that applies only to certain portions of the frame (e.g., the corrupted portion of the region of interest) to improve the computational efficiency of the system. In one example, if an object of interest / feature 508 is detected in a predetermined number (e.g., four) of previous consecutive frames, a high-quality and computationally intensive error hiding function can be selected to improve the quality of frame 502. If error hider 108 determines that the amount of missing / corrupted data in frame 502 exceeds a threshold, error hider 108 can selectively apply a high-quality and computationally intensive error hiding function only to the missing / corrupted macroblocks corresponding to the region of interest. In this example, the error hiding function can be applied to the top of region of interest 504A.

[0059] In another example, if an object of interest / feature 508 is detected in a predetermined number (e.g., four) of the previous consecutive frames, a high-quality, computationally intensive error concealment function can be selected to improve the quality of frame 502. If the error concealer 108 determines that the amount of missing / corrupted data in frame 502 exceeds a threshold, the error concealer 108 can selectively apply the high-quality, computationally intensive error concealment function only to macroblocks predicted to contain object of interest 508 in the current frame 502. The error concealer 108 can use the location of the detected object of interest 508 in the previous frames (e.g., as indicated in model output 120) to estimate the location of object of interest 508 in the current frame 502. In some implementations, the estimated location of object of interest 508 in the current frame 502 can be determined based on motion vectors generated from the previous consecutive frames, which, when applied to macroblocks depicting object of interest 508 in the previous consecutive frames, can indicate the estimated location of object of interest 508 in the current frame 502. In some implementations, the error concealment function may be applied only to macroblocks that are estimated to depict the object of interest and are located within the region of interest associated with frame 502. Selectively applying high-quality error concealment functions to certain portions of a frame (e.g., the region of interest, the estimated location of the object of interest, etc.) while using lower-accuracy and lower-resource-intensive methods in other cases can improve the overall performance of the system when processing many consecutive frames of a video stream.

[0060] In some implementations, if an object of interest 508 is detected in a predetermined number of frames, and the amount of missing / corrupted data in frame 502 exceeds a threshold, the error hider 108 may choose to regenerate a portion of frame 502 using an error hiding function implemented by an artificial intelligence model (e.g., a machine learning model, such as NVIDIA's Deep Learning Super Sampling (DLSS), generative AI models, etc.). Such a machine learning model can be trained / updated to receive frames with missing / corrupted portions as input and generate output frames that include predictions of alternative pixels that will appear in the missing / corrupted portions. In some implementations, the machine learning model(s) may receive information from previous frames to estimate the pixel values ​​of the missing / corrupted portions in the current frame 502. To perform processing using previous consecutive frames, the error hider 108 may store / maintain one or more data structures that, in addition to storing the corresponding metadata and / or model output 120 associated with these frames, also store a sliding window of previously decoded / corrected frames.

[0061] refer to Figure 2 An example data flow diagram 200 according to some embodiments of the present disclosure is shown, illustrating how video data captured from security devices is processed in an environment containing one or more security cameras. Data flow diagram 200 shows a set of streaming video sources 202, including in this example security cameras 203A-203N (sometimes collectively referred to as "security camera 203"). Streaming video sources 202 can include any type of application or device capable of generating or otherwise providing video data. Video data generated or captured by streaming video sources 202 can be provided to a decoder / error hider process 204, which can implement... Figure 1 Any functionality of the decoder 104 and error hider 108. In some implementations, a corresponding decoder / error hider process 204 may be executed for each streaming video source 202 (e.g., each security camera 203). In some implementations, the decoder / error hider process 204 may receive and process multiple video streams in parallel. The decoder / error hider process 204 may be implemented in one or more computing systems (e.g., data processing system 102, etc.).

[0062] As described herein, the output of decoder / error hider process 204 may include a corrected sequence of frames of decoded video data, which may be provided to batch processing process 206. Batch processing process 206 may be used to aggregate decoded / corrected frame data into one or more data structures for processing by a distributed machine learning system (e.g., data processing system 102). Aggregating video data may include storing multiple frames in data structures compatible with different computing hardware, including graphics processing units (GPUs) or other distributed computing components / devices. The output of the batch processing process may be provided for processing using machine learning operation 208.

[0063] Machine learning operation 208 may include any machine learning model described herein (e.g., machine learning model 118). Machine learning operation 208 may include, but is not limited to, object / feature detection, segmentation, or classification. Machine learning operation 208 may be performed sequentially for each frame, or, in some implementations, in parallel for multiple frames (e.g., received from the output of batch processing 206). Machine learning operation 208 may generate outputs (e.g., model output 120) that may be provided to decoder / error hiding process 204. For example, an indication of whether a previous frame depicted an object / feature of interest (e.g., location, etc.) may be provided to decoder / error hiding process 204, such that the previous frame may influence the selection of error hiding functions for subsequent frames, as described herein. The output of machine learning operation 208 may be provided to one or more downstream operations 210, which may include storage in one or more data repositories / storage systems, encoding operations, video streaming operations, or other processing techniques.

[0064] Figure 6 This is a flowchart illustrating a method 600 for implementing context-aware error hiding to improve inference accuracy according to some embodiments of the present disclosure. Various operations of method 600 may be implemented by the same or different devices or entities at different points in time. For example, one or more first devices may implement operations related to decoding and correcting video data, and one or more second devices may implement operations related to machine learning operations (e.g., implementing machine learning model 118, etc.).

[0065] Each block of the method 600 described herein includes a computational process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions can be implemented by a processor executing instructions stored in memory. Method 600 can also be embodied as computer-usable instructions stored on a computer storage medium. Method 600 can be provided by a standalone application, service, or managed service (standalone or in combination with other managed services), or as a plug-in to other products. Furthermore, Method 600 is... Figure 1 and Figure 2 The method 600 is described using the system described herein as an example. However, this method 600 may be additionally or alternatively performed by any system or any combination of systems, including but not limited to the system described herein.

[0066] Method 600 includes at block B602: identifying frames (e.g., decoded frame 110, etc.) of a video stream (e.g., encoded video data stream 112). Frames of the video stream can be generated by decoding at least a portion of the encoded bitstream. In some implementations, frames can be received from a decoder system (e.g., a computing system implementing decoder 104, etc.). The frame may contain raw pixel data, the resolution and color depth of which are specified via metadata of the video stream. The frame may be part of a frame sequence of the video stream. Each frame of the video stream can be processed according to the operations described herein. In some implementations, the frame may be an encoded frame (e.g., encoded frame 113) that will be decoded according to the techniques described herein.

[0067] Method 600 at block B604 includes: determining whether the frame contains corrupted or missing data. Determining whether the frame contains missing / corrupted data can be performed during frame decoding. For this purpose, a combination of... Figure 1 This refers to any operation described in the decoder 104 and / or error detector 106. For example, various techniques such as parity checking, checksums, or more advanced error detection can be used to detect whether information in the frame is missing or corrupted. In some implementations, instances of missing or corrupted portions in a frame can be identified by detecting missing / corrupted portions of the encoded bitstream of the video stream. For example, the bitstream can be scanned / analyzed to look for inconsistencies or anomalies indicating data corruption or missingness.

[0068] Method 600, at block B606, includes generating a corrected frame by applying an error concealment function selected based at least on the location of the corrupted or missing data in the frame and the region of interest (ROI) within the frame. In one example, if the lost or corrupted data occurs outside the ROI in the frame, a low-quality error concealment function that does not require significant computational resources can be selected. In some implementations, if the lost or corrupted data occurs outside the ROI in the frame, the error concealment function may not be implemented. In another example, if the lost or corrupted data occurs within the ROI (or in a region of the frame that might affect the decoding of the ROI), a higher-quality, computationally more expensive error concealment function can be selected.

[0069] In some implementations, the error concealment function can be selected based on whether an object of interest (OP) is detected in a predetermined number (e.g., two, three, four, etc.) of previous consecutive frames in the video stream. As described herein, one or more machine learning models (e.g., machine learning model 118) can be used to detect the presence and location of OOP / features in the frames. In some implementations, if no OOP is detected in a predetermined number of previous consecutive frames, and missing / corrupted data is detected in the region of interest of the frame, a low-quality error concealment function that does not require significant computational resources can be selected. If an OOP is detected in a predetermined number of previous consecutive frames, and missing / corrupted data is detected in the region of interest of the frame, a higher-quality error concealment function that is more computationally expensive can be selected.

[0070] The amount of missing / corrupted data in a frame influences the choice of which error concealment function and / or how it is applied. In one example, if an object of interest is detected in a predetermined number of previous consecutive frames, and a large amount of missing / corrupted data (e.g., an entire slice, etc.) is detected in that frame, a very high-quality, but computationally more expensive, error concealment function can be chosen. In some implementations, if an object of interest is detected in a predetermined number of previous consecutive frames, and a large amount of missing / corrupted data (e.g., an entire slice, etc.) is detected in that frame, a very high-quality error concealment function can be applied to the macroblock in the frame where the object of interest / feature is estimated to appear. The estimated location of the object of interest / feature can be determined based on its position in one or more previous frames.

[0071] In some implementations, if the amount of missing / corrupted data in a frame exceeds a threshold, a generative machine learning model can be used to reconstruct the missing / corrupted portions of the frame, as described herein. Once an error concealment function is selected, it can be applied to the frame to generate one or more corrected frames (e.g., corrected frame 114). The corrected frames can be processed using one or more downstream processing operations, which may include processing using a machine learning model (e.g., machine learning model 118).

[0072] Example content streaming system

[0073] See now Figure 7 , Figure 7 This is an example system diagram of a content streaming system 700 according to some embodiments of the present disclosure. Figure 7 Includes one or more application servers 702 (which may include...) Figure 8 Example computing device 800 (similar components, features, and / or functions), one or more client devices 704 (which may include similar components, features, and / or functions to the example computing device 800), and one or more client devices 704. Figure 8(Similar components, features, and / or functions to example computing device 800) and one or more networks 706 (which may be similar to one or more networks described herein). In some embodiments of this disclosure, system 700 may be implemented to hide errors in a video stream by selectively applying an error hiding function selected based on the location and severity of corrupt / missing data. Application sessions may correspond to game streaming applications (e.g., NVIDIA GeForce Now), remote desktop applications, simulation applications (e.g., autonomous or semi-autonomous vehicle simulations), computer-aided design (CAD) applications, virtual reality (VR) and / or augmented reality (AR) streaming applications, deep learning applications, and / or other application types. For example, system 700 may be implemented to receive inputs indicating one or more features to be used in the output generated by a neural network model, provide inputs to the model to cause the model to generate outputs, and use the outputs for various operations, including display or simulation operations.

[0074] In system 700, for an application session, one or more client devices 704 may simply receive input data in response to input from one or more input devices 726, transfer the input data to one or more application servers 702, receive encoded display data from one or more application servers 702, and display the display data on a display 724. Therefore, computationally intensive computation and processing are offloaded to one or more application servers 702 (e.g., rendering of the graphics output for the application session—specifically ray or path tracing—is performed by one or more GPUs of one or more application servers 702). In other words, the application session is streamed from one or more application servers 702 to one or more client devices 704, thereby reducing the graphics processing and rendering requirements of one or more client devices 704.

[0075] For example, regarding the instantiation of an application session, client device 704 can display frames of the application session on display 724 based at least on receiving display data from one or more application servers 702. Client device 704 can receive input from one or more input devices 726 and generate input data in response. Client device 704 can send the input data to application server 702 via communication interface 720 and via network 706 (e.g., the Internet), and application server 702 can receive the input data via communication interface 718. CPU 708 can receive the input data, process the input data, and transmit data to GPU 710, which causes GPU 710 to generate a rendering of the application session. For example, the input data can represent a user's character's movement, firing weapons, reloading, passing the ball, steering a vehicle, etc., in a game session of a game application. Rendering component 712 can render the application session (e.g., the result representing the input data), and rendering capture component 714 can capture the rendering of the application session as display data (e.g., image data as captured rendering frames of the application session). Rendering of an application session may include lighting and / or shadow effects computed using one or more parallel processing units (such as GPUs) of application server 702, which may further utilize one or more dedicated hardware accelerators or processing cores to perform ray or path tracing techniques. In some embodiments, one or more virtual machines (VMs)—e.g., including one or more virtual components such as vGPUs, vCPUs, etc.—may be used by application server 702 to support the application session. Encoder 716 may then encode display data to produce encoded display data, which may be transmitted to client device 704 via communication interface 718 through network 706. Client device 704 may receive the encoded display data via communication interface 720, and decoder 722 may decode the encoded display data to produce display data. Client device 704 may then display the display data via display 724.

[0076] Example computing device

[0077] Figure 8This is a block diagram of an example computing device 800 suitable for implementing some embodiments of the present disclosure. The computing device 800 may include an interconnect system 802 directly or indirectly coupled to: a memory 804, one or more central processing units (CPUs) 806, one or more graphics processing units (GPUs) 808, a communication interface 810, input / output (I / O) ports 812, input / output components 814, a power supply 816, one or more presentation components 818 (e.g., one or more displays), and one or more logic units 820. In at least one embodiment, one or more computing devices 800 may include one or more virtual machines (VMs), and / or any component thereof may include virtual components (e.g., virtual hardware components). For a non-limiting example, one or more GPUs 808 may include one or more vGPUs, one or more CPUs 806 may include one or more vCPUs, and / or one or more logic units 820 may include one or more virtual logic units. Accordingly, one or more computing devices 800 may include discrete components (e.g., a full GPU dedicated to computing device 800), virtual components (e.g., a portion of the GPU dedicated to computing device 800), or a combination thereof.

[0078] although Figure 8 The various boxes are shown as being connected to lines via interconnect system 802, but this is not intended to be limiting and is merely for clarity. For example, in some embodiments, presentation component 818 (such as a display device) may be considered I / O component 814 (e.g., if the display is a touchscreen). As another example, CPU 806 and / or GPU 808 may include memory (e.g., memory 804 may represent a storage device in addition to the memory of GPU 808, CPU 806, and / or other components). In other words, Figure 8 The computing devices described are merely illustrative. No distinction is made between categories such as "workstation," "server," "laptop computer," "desktop computer," "tablet computer," "client device," "mobile device," "handheld device," "game console," "electronic control unit (ECU)," "virtual reality system," and / or other device or system types, as all are conceived in… Figure 8 Within the scope of computing devices.

[0079] Interconnect system 802 may represent one or more links or buses, such as address buses, data buses, control buses, or combinations thereof. Interconnect system 802 may be arranged in various topologies, including but not limited to bus, star, ring, mesh, tree, or hybrid topologies. Interconnect system 802 may include one or more bus or link types, such as Industry Standard Architecture (ISA) bus, Extended Industry Standard Architecture (EISA) bus, Video Electronics Standards Association (VESA) bus, Peripheral Component Interconnect (PCI) bus, Fast Peripheral Component Interconnect (PCIe) bus, and / or another type of bus or link. In some embodiments, there is a direct connection between components. For example, CPU 806 may be directly connected to memory 804. Further, CPU 806 may be directly connected to GPU 808. In cases where there is a direct connection or point-to-point connection between components, interconnect system 802 may include a PCIe link to perform the connection. In these examples, a PCI bus is not required in computing device 800.

[0080] The memory 804 may include any of a variety of computer-readable media. The computer-readable media may be any available medium accessible by the computing device 800. The computer-readable media may include volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, the computer-readable media may include computer storage media and communication media.

[0081] Computer storage media may include volatile and non-volatile media and / or removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 804 may store computer-readable instructions (e.g., representing programs and / or program elements, such as an operating system). Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, Digital Universal Disc (DVD) or other optical disc storage, magnetic tape cassettes, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by computing device 800. As used herein, computer storage media does not include the signal itself.

[0082] Computer storage media can embody computer-readable instructions, data structures, program modules, and / or other data types in modulated data signals (such as carrier waves or other transmission mechanisms) and include any information transmission medium. The term "modulated data signal" can refer to a signal whose characteristics are set or altered in a manner that encodes information in the signal. By way of example and not limitation, computer storage media can include wired media (such as wired networks or direct wired connections) and wireless media (such as acoustic, RF, infrared, and other wireless media). Any combination of the foregoing should also be included within the scope of computer-readable media.

[0083] CPU 806 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 800 to perform one or more of the methods and / or processes described herein. Each CPU 806 may include one or more cores (e.g., 1, 2, 4, 8, 28, 72, etc.) capable of processing multiple software threads simultaneously. CPU 806 may include any type of processor and may include different types of processors depending on the type of computing device 800 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device 800, the processor may be an advanced RISC machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or supplemental coprocessors such as math coprocessors, computing device 800 may also include one or more CPUs 806.

[0084] In addition to or replacing CPU 806, one or more GPUs 808 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 800 to perform one or more of the methods and / or processes described herein. One or more GPUs 808 may be integrated GPUs (e.g., having one or more CPUs 806) and / or one or more GPUs 808 may be discrete GPUs. In embodiments, one or more GPUs 808 may be a coprocessor of one or more CPUs 806. GPUs 808 may be used by computing device 800 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, GPUs 808 may be used for general-purpose computing on a GPU (GPGPU). GPUs 808 may include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. GPUs 808 may generate pixel data for outputting an image in response to rendering commands (e.g., rendering commands received from CPU 806 via a host interface). GPU 808 may include graphics memory, such as display memory, for storing pixel data or any other suitable data, such as GPGPU data. Display memory may be included as part of memory 804. GPU 808 may include two or more GPUs operating in parallel (e.g., via a link). The link may be directly connected to the GPUs (e.g., using NVLINK) or may connect the GPUs via a switch (e.g., using NVSwitch). When combined, each GPU 808 may generate pixel data or GPGPU data for different portions of the output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU 808 may include its own memory or may share memory with other GPUs.

[0085] In addition to or replacing CPU 806 and / or GPU 808, one or more logic units 820 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 800 to perform one or more of the methods and / or processes described herein. In embodiments, one or more CPUs 806, one or more GPUs 808, and / or one or more logic units 820 may perform any combination of methods, processes, and / or portions thereof, discretely or jointly. One or more logic units 820 may be part of and / or integrated into one or more CPUs 806 and / or GPUs 808, and / or one or more logic units 820 may be discrete components or otherwise external to CPUs 806 and / or GPUs 808. In embodiments, one or more logic units 820 may be coprocessors of one or more CPUs 806 and / or one or more GPUs 808.

[0086] Examples of logic unit 820 include one or more processing cores and / or components thereof, such as a data processing unit (DPU), tensor core (TC), tensor processing unit (TPU), pixel vision core (PVC), vision processing unit (VPU), image processing unit (IPU), graphics processing cluster (GPC), texture processing cluster (TPC), streaming multiprocessor (SM), tree traversal unit (TTU), artificial intelligence accelerator (AIA), deep learning accelerator (DLA), arithmetic logic unit (ALU), application-specific integrated circuit (ASIC), floating-point unit (FPU), input / output (I / O) element, peripheral component interconnect (PCI) or peripheral component interconnect fast (PCIe) element, etc.

[0087] The communication interface 810 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 800 to communicate with other computing devices via electronic communication networks (including wired and / or wireless communications). The communication interface 810 may include components and functions that enable communication over any of a plurality of different networks (e.g., wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication over Ethernet or wireless bandwidth), low-power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet). In one or more embodiments, the logic unit 820 and / or the communication interface 810 may include one or more data processing units (DPUs) to directly transmit data received via a network and / or via interconnect system 802 to one or more GPUs 808 (e.g., memory of one or more GPUs 808). In some embodiments, multiple computing devices 800 or components thereof (which may be similar or different from each other in various respects) may be communicatively coupled to send and receive data for performing various operations described herein, such as to facilitate the reduction of latency.

[0088] I / O port 812 enables computing device 800 to be logically coupled to other devices including I / O component 814, presentation component 818, and / or other components, some of which may be built into (e.g., integrated into) computing device 800. Illustrative I / O component 814 includes microphones, mice, keyboards, joysticks, gamepads, game controllers, disc-type satellite antennas, scanners, printers, wireless devices, etc. I / O component 814 provides a natural user interface (NUI) that processes air gestures, voice, or other physiological input generated by the user. In some instances, the input may be passed to appropriate network elements for further processing, such as modification and image registration. NUI can implement any combination of voice recognition, stylus recognition, facial recognition, biometric recognition, on-screen and near-screen gesture recognition, air gestures, head and eye tracking, and touch recognition associated with the display of computing device 800 (as described in more detail below). The computing device 800 may include a depth camera, such as a stereo camera system, an infrared camera system, an RGB camera system, touchscreen technology, and combinations thereof, for attitude detection and recognition. Additionally, the computing device 800 may include an accelerometer or gyroscope (e.g., as part of an inertial measurement unit (IMU)) that enables motion detection. In some examples, the computing device 800 may use the output of the accelerometer or gyroscope to render immersive augmented reality or virtual reality.

[0089] Power supply 816 may include a hardwired power supply, a battery power supply, or a combination thereof. Power supply 816 may provide power to computing device 800 so that the components of computing device 800 can operate.

[0090] One or more presentation components 818 may include displays (e.g., monitors, touchscreens, television screens, head-up displays (HUDs), other display types, or combinations thereof), speakers, and / or other presentation components. Presentation component 818 may receive data from other components (e.g., GPU 808, CPU 806, DPU, etc.) and output data (e.g., as images, videos, sounds, etc.).

[0091] Example Data Center

[0092] Figure 9 An example data center 900 that can be used in at least one embodiment of this disclosure is shown, such as implementing system 100, in conjunction with... Figure 2 The described operation, or in one or more examples of data center 900. Data center 900 may include data center infrastructure layer 910, framework layer 920, software layer 930, and / or application layer 940.

[0093] like Figure 9 As shown, the data center infrastructure layer 910 may include a resource coordinator 912, grouped computing resources 914, and node computing resources (“nodes CR”) 916(1)-916(N), where “N” represents any integer, a positive integer. In at least one embodiment, nodes CR 916(1)-916(N) may include, but are not limited to, any number of central processing units (“CPUs”) or other processors (including DPUs, accelerators, field-programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output (“NW”) devices, and network network interfaces (“NW”). I / O devices, network switches, virtual machines ("VMs"), power modules and / or cooling modules, etc. In some embodiments, one or more nodes CR916(1)-916(N) may correspond to a server having one or more of the aforementioned computing resources. Furthermore, in some embodiments, nodes CR916(1)-916(N) may include one or more virtual components, such as vGPU, vCPU, etc., and / or one or more nodes CR916(1)-916(N) may correspond to virtual machines (VMs).

[0094] In at least one embodiment, the grouped computing resources 914 may include separate groups of nodes CR916 housed within one or more racks (not shown) or within a plurality of racks in data centers (also not shown) located in different geographical locations. The separate groups of nodes CR916 within the grouped computing resources 914 may include the group's computing, networking, memory, or storage resources, which may be configured or allocated to support one or more workloads. In at least one embodiment, a plurality of nodes CR916, including CPUs, GPUs, DPUs, and / or other processors, may be grouped within one or more racks to provide computing resources to support one or more workloads. One or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.

[0095] Resource coordinator 912 may be configured or otherwise control one or more nodes CR916(1)-916(N) and / or grouped computing resources 914. In at least one embodiment, resource coordinator 912 may include a Software Design Infrastructure (“SDI”) management entity for data center 900. Resource coordinator 912 may include hardware, software, or some combination thereof.

[0096] In at least one embodiment, such as Figure 9 As shown, framework layer 920 may include a job scheduler 928, a configuration manager 934, a resource manager 936, and / or a distributed file system 938. Framework layer 920 may include a framework of software 932 supporting software layer 930 and / or one or more applications 942 of application layer 940. Software 932 or application 942 may respectively include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. Framework layer 920 may be, but is not limited to, a type of free and open-source software web application framework that can utilize the distributed file system 938 for large-scale data processing (e.g., "big data"), such as Apache Spark. TM(Hereinafter referred to as "Spark"). In at least one embodiment, job scheduler 928 may include Spark drivers to facilitate the scheduling of workloads supported by various layers of data center 900. Configuration manager 934 may be able to configure different layers, such as software layer 930 and framework layer 920 including Spark and distributed file system 938 for supporting large-scale data processing. Resource manager 936 may be able to manage cluster or group computing resources mapped to or allocated for supporting distributed file system 938 and job scheduler 928. In at least one embodiment, cluster or group computing resources may include group computing resources 914 at data center infrastructure layer 910. Resource manager 936 may coordinate with resource coordinator 912 to manage these mapped or allocated computing resources.

[0097] In at least one embodiment, the software 932 included in the software layer 930 may include software used by at least a plurality of portions of nodes CR916(1)-916(N), grouped computing resources 914, and / or the distributed file system 938 of the framework layer 920. One or more types of software may include, but are not limited to, internet web search software, email virus scanning software, database software, and streaming video content software.

[0098] In at least one embodiment, the application 942 included in the application layer 940 may include one or more types of applications used by at least a plurality of portions of nodes CR916(1)-916(N), grouped computing resources 914, and / or the distributed file system 938 of the framework layer 920. One or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing and machine learning applications (including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.) and / or other machine learning applications used in combination with one or more embodiments).

[0099] In at least one embodiment, any one of the configuration manager 934, resource manager 936, and resource coordinator 912 can implement any number and type of self-modification actions based at least on any amount and type of data acquired in any technically feasible manner. Self-modification actions can protect the data center operator of data center 900 from making potentially poor configuration decisions and may prevent underutilization and / or poor performance of parts of the data center.

[0100] According to one or more embodiments described herein, data center 900 may include tools, services, software, or other resources for updating / training one or more machine learning models or using one or more machine learning models to predict or infer information. For example, one or more machine learning models may be updated / trained by calculating weight parameters according to a neural network architecture using the software and / or computing resources described above with respect to data center 900. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks may be used to infer or predict information using the resources described above with respect to data center 900 by using weight parameters calculated through one or more training techniques (such as, but not limited to, those described herein).

[0101] In at least one embodiment, the data center 900 may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, and / or other hardware (or corresponding virtual computing resources) to perform training and / or inference using the aforementioned resources. Furthermore, one or more of the aforementioned software and / or hardware resources may be configured to allow users to update / train or perform information inference services, such as image recognition, speech recognition, or other artificial intelligence services.

[0102] Example network environment

[0103] A network environment suitable for implementing embodiments of this disclosure may include one or more client devices, servers, network attached storage (NAS), other back-end devices, and / or other device types. Client devices, servers, and / or other device types (e.g., each device) may be... Figure 8 The implementation is carried out on one or more instances of computing device 800, for example, each device may include similar components, features and / or functions of computing device 800. Furthermore, in the case of implementing backend devices (e.g., servers, NAS, etc.), the backend devices may be included as part of data center 900, an example of which is relative to the data center 900 described herein. Figure 9 To describe in more detail.

[0104] Components of a network environment can communicate with each other via one or more networks, which may be wired, wireless, or both. A network can include multiple networks or a network of networks. For example, a network may include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the Public Switched Telephone Network (PSTN)), and / or one or more private networks. Where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) can provide wireless connectivity.

[0105] A compatible network environment may include one or more peer-to-peer network environments—in which case the network environment may not include a server—and one or more client-server network environments—in which case the network environment may include one or more servers. In a peer-to-peer network environment, the functionality described herein with respect to one or more servers can be implemented on any number of client devices.

[0106] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, etc. The cloud-based network environment may include a framework layer, job scheduler, resource manager, and distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework for software supporting the software layer and / or one or more applications supporting the application layer. The software or application may respectively include web-based service software or applications. In embodiments, one or more client devices may use web-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, free and open-source software web application frameworks, such as those used for large-scale data processing (e.g., "big data") using distributed file systems.

[0107] A cloud-based network environment can provide cloud computing and / or cloud storage for any combination of the computing and / or data storage functions (or one or more portions thereof) described herein. Any of these different functions may be distributed across multiple locations from a central or core server (e.g., across one or more data centers distributed across states, regions, countries, globally, etc.). If the connection to the user (e.g., client device) is relatively close to the edge server, the core server may assign at least a portion of the functionality to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0108] One or more client devices may be included in this discussion. Figure 8At least some of the components, features, and functions of one or more example computing devices 800 described. By way of example and not limitation, a client device may be embodied as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, camera, surveillance equipment or system, vehicle, ship, spacecraft, virtual machine, drone, robot, handheld communication device, hospital equipment, gaming device or system, entertainment system, vehicle computer system, embedded system controller, remote control, electrical appliance, consumer electronics device, workstation, edge device, any combination of these described devices, or any other suitable device.

[0109] This disclosure can be described in the general context of computer code or machine-usable instructions (including computer-executable instructions, such as program modules) that are executed by a computer or other machine (such as a personal data assistant or other handheld device). Typically, a program module, including routines, programs, objects, components, data structures, etc., refers to code that performs a specific task or implements a specific abstract data type. This disclosure can be implemented in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, and more specialized computing devices. This disclosure can also be implemented in a distributed computing environment where tasks are performed by remote processing devices linked via a communication network.

[0110] As used herein, the phrase “and / or” relating to two or more elements should be interpreted as meaning only one element, or a combination of elements. For example, “element A, element B, and / or element C” can include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or element A, B, and C. Furthermore, “at least one of element A or element B” can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Additionally, “at least one of element A and element B” can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0111] The subject matter of this disclosure is specifically described herein to satisfy legal requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors have anticipated that the claimed subject matter may also be embodied in other ways in combination with other current or future techniques to include combinations of different steps or steps similar to those described in this document. Furthermore, although the terms “step” and / or “box” may be used herein to refer to different elements of the method employed, such terms should not be construed as implying any particular order among or between the various steps disclosed herein, unless and only if the order of individual steps is explicitly described.

Claims

1. One or more processors, comprising: One or more circuits are used for: Identify frames in a video stream; The frame is determined to contain corrupted or missing data; as well as A corrected frame is generated by applying an error-hiding function selected based at least on the location of the corrupted or missing data in the frame and the region of interest in the frame.

2. The processors according to claim 1, wherein, The one or more circuits are used to select the error hiding function based at least on the location of the corrupted or lost data being within the region of interest in the frame.

3. The processors according to claim 1, wherein, The one or more circuits are used to generate the corrected frame by applying the error concealment function to at least a portion of the region of interest.

4. The processors according to claim 1, wherein, The one or more circuits are used to apply the error concealment function by providing the frame as input to a machine learning model.

5. The processors according to claim 1, wherein, The one or more circuits are used for: Receive the encoded bitstream of the video stream; and The frame is generated by decoding the encoded bitstream, wherein the decoding of the frame indicates the location of the corrupted or lost data.

6. The processors according to claim 1, wherein, The error hiding function is the first error hiding function among a plurality of error hiding functions.

7. The processors according to claim 6, wherein, The one or more circuits are used for: Determine that an object was detected in a predetermined number of previous frames in the video stream; as well as Based at least on the detection of the object in the predetermined number of previous frames, the first error hiding function is selected from the plurality of error hiding functions, the first error hiding function using more computational resources than the second error hiding function among the plurality of error hiding functions.

8. The processors according to claim 7, wherein, The one or more circuits are used for: The corrected frame is generated by applying the first error-hiding function to at least a portion of the frames in which the object is estimated to appear.

9. The processors according to claim 6, wherein, The one or more circuits are used for: It is determined that the damaged or lost data is located outside the region of interest; as well as Based at least on the premise that the damaged or lost data is located outside the region of interest, the first error hiding function is selected from the plurality of error hiding functions, and the first error hiding function uses less computational resources compared to the second error hiding function among the plurality of error hiding functions.

10. The processors according to claim 1, wherein, The one or more processors are included in at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system used to perform simulation operations; Systems used to perform digital twin operations; A system for performing optical transmission simulation; A system for performing collaborative content creation for 3D assets; A system used to perform deep learning operations; Systems implemented using edge devices; Systems implemented using robots; Systems used to perform conversational AI operations; Systems that use large language model LLMs to perform generative AI operations; A system that uses a visual language model (VLM) to perform generative AI operations; A system for generating synthetic data; A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

11. A system comprising: One or more processors are used for: Receive a request to process a video stream containing multiple frames; An error-hiding function is applied to at least one of the plurality of frames, the error-hiding function being selected based at least on the location of corrupted or missing data in the at least one frame and the region of interest in the at least one frame; as well as The machine learning model is executed using the at least one frame as input to the request.

12. The system according to claim 11, wherein, The one or more processors are used for: The decoding process is used to determine that the at least one frame contains corrupted or missing data.

13. The system according to claim 11, wherein, The machine learning model generates indications of objects in the at least one frame, and wherein the one or more processors are used for: A second error hiding function is selected for at least one second frame of the plurality of frames, based at least on the indication of the object in the at least one frame.

14. The system according to claim 13, wherein, The second error-hiding function for the at least one second frame is further selected based on the expected position of the object in the at least one second frame.

15. The system according to claim 11, wherein, The location where the data is damaged or lost is either the macroblock location or the slice location of the at least one frame.

16. The system according to claim 11, wherein, The one or more processors are used for: The error correction function is applied by executing a second machine learning model that takes the at least one frame as input, the second machine learning model being used to generate replacement information for the damaged or missing data of the at least one frame.

17. One or more processors according to claim 11, wherein, The one or more processors are included in at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system used to perform simulation operations; Systems used to perform digital twin operations; A system for performing optical transmission simulation; A system for performing collaborative content creation for 3D assets; A system used to perform deep learning operations; Systems implemented using edge devices; Systems implemented using robots; Systems used to perform conversational AI operations; Systems that use large language model LLMs to perform generative AI operations; A system that uses a visual language model (VLM) to perform generative AI operations; A system for generating synthetic data; A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

18. A method comprising: Use one or more processors to identify frames in a video stream; The one or more processors are used to determine that the frame contains corrupted or missing data; as well as Using one or more processors, a corrected frame is generated by applying an error-hiding function selected at least based on the location of the corrupted or missing data in the frame and the region of interest in the frame.

19. The method of claim 18, further comprising: The error concealment function is selected using one or more processors, at least based on the fact that the location of the corrupted or lost data is within the region of interest in the frame.

20. The method of claim 18, further comprising: The corrected frame is generated using one or more processors by applying the error concealment function to at least a portion of the region of interest.

Images