Multi-image interpolation for real-time video processing using deep neural networks

Abstract

Approaches to improving frame rate and visual smoothness in real-time video streams through multi-frame interpolation are revealed. A neural classification network analyzes two consecutive images and outputs confidence scores indicating the reliability of motion data for each pixel. These scores determine whether the motion of a pixel is accurately described by the motion vectors or should be treated as static. The classification results are reused to generate intermediate frames by warping the original images based on their motion properties. Blended weights are calculated by combining the confidence scores of the warped motion vector with static values, and a second neural network refines the alignment and blending of the candidate images.This second network predicts intermediate streams and generates new blend weights that are used to warp and blend the candidate images, ultimately producing a final interpolated image that improves the visual smoothness and consistency in the video stream.

Description

BACKGROUND

[0001] In the field of real-time image and video processing, particularly in high-performance applications such as video games, virtual reality, and live streaming, image interpolation plays a crucial role in enhancing the visual experience by increasing the perceived frame rate. This process involves generating intermediate frames between existing images to achieve smoother motion and improved visual fluidity. However, conventional image interpolation methods can present challenges due to their computational intensity and the potential for introducing latency. This is especially problematic in scenarios that demand both high frame rates and low latency, such as competitive gaming or immersive virtual environments.As display resolutions and refresh rates continue to advance, the need for more efficient image interpolation methods to deliver high-quality images without compromising performance or increasing latency has become increasingly critical, especially in dynamic real-time environments.

[0002] US Patent 2019 / 0138889A1 describes a method in which several neural networks first determine the optical flow data—that is, the movement between two images—and predict visibility maps for each point in time. Based on the optical flow data, two consecutive distorted images are then generated, which are finally fused into an intermediate image. WO Patent 2018 / 089131A1 discloses a model for video image synthesis that integrates, among other things, a convolutional neural network with a voxel layer. In US Patent 2022 / 0303495A1, a forward flow from the first input image to the second input image and a backward flow from the second input image to the first input image are generated. Based on this flow data, an occlusion map and a consistency map are created, which, through subsequent blending, produce an interpolated image.In US 2024 / 0 098 216 A1, one or more neural networks are used to process two or more intermediate images between a first and a second image into an intermediate image using frame blending. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Various embodiments according to the present disclosure are described with reference to the drawings, in which: Fig. 1 An exemplary system for improving the frame rate and visual smoothness of real-time video streams by multi-image interpolation according to at least one embodiment is illustrated; Fig. 2A and Fig. 2B illustrate how intermediate frames are generated in a time axis or in the course of real-time video stream processing according to at least one embodiment; Fig. 3A illustrates a detailed method for generating candidate images according to at least one embodiment; Fig. 3B illustrates another exemplary embodiment in which the generation of intermediate images includes additional motion properties derived from additional motion candidates, according to at least one embodiment; Fig. 4. A detailed flowchart illustrates an exemplary method for generating an intermediate image from two input images according to at least one embodiment; Fig. 5 is a flowchart illustrating the generation of an intermediate image between two successive images from a video stream according to at least one embodiment; Fig. 6 illustrates an exemplary system that includes an intermediate image generation system according to at least one embodiment; Fig. 7A illustrates an inference and / or training logic according to at least one embodiment; Fig. 7B illustrates an inference and / or training logic according to at least one embodiment; Fig. 8 illustrates an exemplary data center system according to at least one embodiment; Fig. 9 illustrates a computer system according to at least one embodiment; Fig. 10 illustrates a computer system according to at least one embodiment; Fig. 11 illustrates at least parts of a graphics processor according to one or more embodiments; Fig. 12 at least parts of a graphics processor according to one or more embodiments illustrated; Fig. 13 an exemplary data flow diagram for an advanced computing pipeline according to at least one embodiment; Fig. 14 a system diagram for an exemplary system for training, adapting, instantiating and deploying machine learning models in an advanced computing pipeline according to at least one embodiment; and Fig. 15A and Fig. 15B illustrates a data flow diagram for a method for training a machine learning model and a client-server architecture for improving annotation tools with pre-trained annotation models according to at least one embodiment. DETAILED DESCRIPTION

[0004] The following description details various embodiments. For illustrative purposes, specific configurations and details are presented to provide a thorough understanding of these embodiments. However, it is also obvious to a person skilled in the art that the embodiments can be implemented without these specific details. Furthermore, known features may be omitted or simplified so as not to obscure the described embodiment.

[0005] The systems and procedures described here can be used by, among others, non-autonomous vehicles or machines, semi-autonomous or autonomous vehicles or machines (e.g., in one or more advanced driver assistance systems (ADAS), one or more in-vehicle infotainment systems, one or more emergency vehicle detection systems), controlled and uncontrolled robots or robotic platforms, warehouse vehicles, all-terrain vehicles, vehicles coupled with one or more trailers, flying ships, boats, shuttles, emergency vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, trains, underwater vehicles, remotely controlled vehicles such as drones, and / or other types of vehicles.Furthermore, the systems and methods described here can be used for a variety of purposes, including but not limited to machine control, machine locomotion, machine driving, synthetic data generation, generative AI, model training or updating, perception, augmented reality, virtual reality, mixed reality, robotics, security and surveillance, simulation and digital twinning, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, data center processing, conversational AI, light transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation for 3D assets, generative AI, cloud computing, and / or any other suitable applications.

[0006] Disclosed embodiments may be included in a variety of different systems, such as automotive systems (e.g.an in-vehicle infotainment system for an autonomous or semi-autonomous machine, a perception system for an autonomous or semi-autonomous machine, systems implemented using a robot, air systems, medical systems, boat systems, intelligent area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twinning operations, systems implemented using an edge device, systems incorporating one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in a data center, systems for performing conversational AI operations, systems implementing one or more language models - such as large language models (LLMs), visual language models (VLMs), multimodal language models, etc., systems for performing generative AI operations (e.g., using one or more language models, transformer models, etc.), systems for performing light transport simulation, systems for performing collaborative content creation for 3D assets, systems that are implemented at least partially using cloud computing resources, and / or other types of systems.

[0007] Approaches according to various embodiments can be used to generate one or more parameters for a content generation environment. In at least one embodiment, a trained machine learning (ML) and / or artificial intelligence (AI) system, such as a large language model (LLM) or a vision language model (VLM), can be used to generate parameters for the content generation environment, such as, but not limited to, camera settings, scene lighting, video parameters, and / or the like, used to display objects within a scene. The parameters can be based on input provided by a user or a user proxy to a trained language model (e.g., LLM, VLM, etc.), which can then generate one or more settings according to the input.Various embodiments can be used to generate settings in two-dimensional (2D) or three-dimensional (3D) environments. For embodiments that include one or more language models—that is, one or more LLMs, one or more VLMs, or a combination of LLMs and VLMs—the language model(s) can receive input (e.g., a prompt, a request, a query, etc.) that is parsed or otherwise formatted to produce deterministic output. For example, the input provided to the language model might include a specific format for the output results, an example of desired output results, a specific list of parameters and their respective formatting, and the like. An input generator (e.g.,A prompt generator, which can be driven or otherwise guided by one or more AI and / or ML systems, can be used to generate this input based on an initial input received from a user, device, proxy, and / or the like. A modified input generated by the prompt generator can then be provided to the language model, which produces an output set of parameters. This output can further be evaluated by a validator or other system to ensure its appropriateness. Subsequently, a configuration file can be generated, and / or the parameters can be directly provided to an environment to configure various components (e.g., camera settings, lighting, etc.) based on the parameters generated by the language model.

[0008] In some examples, the machine learning model(s) described here (e.g., deep neural networks, language models, LLMs, VLMs, multimodal language models, perceptual models, tracking models, fusion models, transformer models, diffusion models, encoder-only models, decoder-only models, encoder-decoder models, neural render field (NERF) models, etc.) may be packaged as a microservice—such as an inference microservice (e.g., NVIDIA NIMs)—which may contain a container (e.g., an operating system (OS)-level virtualization package) that may include an application programming interface (API) layer, a server layer, a runtime layer, and / or at least one model “engine.” In some cases, such as when the machine learning model(s) is small enough (e.g., has a small number of parameters), the model(s) can be contained within the container itself.In other examples—such as when the model(s) is / are large—the model(s) may be hosted / stored in the cloud (e.g., in a data center) and / or may be hosted on-premises and / or at the edge (e.g., on a local server or compute unit, but outside the container). In such embodiments, the model(s) may be accessible via one or more APIs—such as REST APIs. Thus, and in some embodiments, the machine learning model(s) described here may be used as an inference microservice to accelerate the deployment of a model(s) on any cloud, data center, or edge computing system, while ensuring data security. For example, the inference microservice may include one or more APIs, a pre-configured container for simplified deployment, an optimized inference engine (e.g., a REST API), or a REST API.created using a standardized AI model deployment, execution software such as NVIDIA's Triton inference server, and / or one or more APIs for high-performance deep learning inference, which may include inference runtime and model optimizations that deliver low latency and high throughput for production applications - such as NVIDIA's TensorRT), and / or enterprise management data for telemetry (e.g., including identity, metrics, integrity checks, and / or monitoring).

[0009] The machine learning model(s) described here can be included as part of the microservice, along with an accelerated infrastructure capable of single-command deployment and / or orchestration and automatic scaling using a container orchestration system on an accelerated infrastructure (e.g., from a single device to data center scale). Thus, the inference microservice can include the machine learning model(s) (optimized, for example, for high-performance inference), inference runtime software for executing the machine learning model(s) and providing outputs / responses to inputs (e.g., user queries, prompts, etc.), and enterprise management software for providing integrity checks, identity verification, and / or other monitoring.In some implementations, the inference microservice may include software for performing an on-premises exchange and / or update of the machine learning model(s). During the exchange or update, the software performing the exchange / update may maintain user configurations of the inference runtime software and the enterprise management software.

[0010] Approaches according to various embodiments of the disclosure enable an increase in frame rate and visual smoothness in real-time video streams through multi-frame interpolation. Conventional methods typically require the execution of computationally intensive procedures to generate each intermediate frame, which can lead to increased playback time and higher computational loads. Such challenges frequently arise in dynamic and high-performance environments such as video games, where maintaining both visual quality and low latency is critical. Approaches according to various embodiments of the present disclosure can address these and other such challenges by performing intensive computations partially once per pair of input frames and then reusing the results in subsequent frame generation to improve processing efficiency.

[0011] In one embodiment, a neural network (e.g., a classifier or classification network) is used to analyze two consecutive images from a video stream. This network can determine motion properties associated with each pixel, where, as used here, the motion properties can refer to the movement and behavior of a pixel between images, such as whether the movement follows motion vectors or should be treated differently (e.g., is not well described by the motion vectors). In one embodiment, a movement that is not well described by motion vectors can be termed static. "Static," as used here, can refer not only to pixels that are fixed or remain immobile between images, but also to pixels whose movement is not accurately described by the motion vectors.This can include both truly static pixels and dynamic pixels—such as dynamic shadows, reflections, particles, or UI elements—that exhibit motion, but for which the motion vectors do not accurately describe their movement. The neural classification network can output confidence scores for at least one (e.g., each) pixel, indicating the reliability of the motion data. This network assesses whether a pixel's motion is described by the motion vectors or whether it should be handled differently. Specifically, the classification network can output two confidence scores per pixel for an image—one for the motion vector provided by the game engine and one for assuming static motion, which can encompass any movement not well explained by the motion vectors.These classification results for the original consecutive images are then reused when generating additional intermediate images between the two original images.

[0012] Once the motion properties have been determined, two candidate images can be generated by warping the original images according to the classified motion properties. During this warping process, pixels can be moved to intermediate positions based on their calculated motion properties. The motion vector confidence values, along with the input pixels, can be warped to an intermediate point using the estimated motion vectors. The static confidence values—those that assume zero motion or no motion and do not require warping—can be combined with the confidence values ​​of the warped motion vectors to form pixel-wise blended weights. These blended weights are calculated by applying a softmax function to the confidence values, for example, without limitation.The generated blend weights are used to blend the warped images with the original images. For example, image 1, which was created with warped motion vectors, is blended with the original image 1 based on the blend weights, and similarly, image 2, which was created with warped motion vectors, is blended with the original image 2, resulting in two candidate images. A second neural network is then used to generate an intermediate image by further refining the alignment and blending the candidate images. Such a refinement process may involve estimating the motion of any dynamic pixels that were initially classified as static by the first neural network because their motion was not well described by the game engine's motion vectors.

[0013] Approaches according to at least one embodiment can provide several technical advantages and improvements over conventional methods. One improvement lies in the ability to reuse motion classification results across multiple intermediate frames. By running a neural network to perform motion classification only once for a pair of input images, motion properties are generated that describe the motion of a pixel (e.g., each pixel). These properties can include confidence scores indicating whether the motion of the respective pixel can be accurately described by motion vectors or is better represented as static, reflecting the reliability of the motion data. These results are stored and subsequently reused when all intermediate frames between the input pair of images are generated.Such an approach eliminates the need to re-execute the motion classification procedure for each intermediate frame, which can be a computationally intensive task and / or unsuitable for low-latency, real-time, or near-real-time applications. Instead, embodiments and implementations of the present disclosure directly apply the motion vectors and confidence values ​​after the initial classification to determine how each pixel should be warped for each intermediate frame, without recalculating the motion vectors from scratch. This reuse of motion classification data can reduce computational costs and latency, resulting in high-quality image interpolation without the overhead of repetitive processing.

[0014] Furthermore, disclosed approaches—including various methods, processing units, and systems—allow for the inclusion of additional motion candidates beyond the motion vectors and classifications of static motion. These motion candidates can represent alternative sources of motion data, such as those derived from optical flow models or other motion estimation methods. Each candidate offers a perspective on pixel motion between frames. For example, while the primary motion vector from the game engine might describe expected motion based on game physics, an optical flow model can provide a more detailed analysis by considering the actual pixel intensity changes across frames.By integrating these various motion candidates, such an approach performs a versatile evaluation of pixel motion, resulting in more accurate and smoother interpolated images. This ability to expand and integrate different motion candidates allows the method to effectively handle complex, nonlinear motion patterns that occur in dynamic scenes, such as fast-moving objects, reflections, or rapidly changing lighting.

[0015] Furthermore, the reuse of computational results extends beyond motion classification and can include any intermediate data generated during the initial analysis of the input images. For example, if a neural network identifies and classifies objects within a scene, such as recognizing a tree or a box, this information can be stored and reused in subsequent image processing. Knowing something about the original images, such as the presence of specific objects, can be useful for intermediate blending estimation. By recognizing an object as a tree, specific warping or blending techniques can be consistently applied across all intermediate images, ensuring that the tree's motion and appearance remain accurate and coherent.Additionally, other intermediate calculations, such as lighting settings, color correction, or depth estimation, derived, for example, from various neural networks, can be stored and reapplied when additional intermediate frames are generated. Thus, the disclosed methods and systems not only reduce processing power by avoiding redundant calculations but also ensure that the visual output remains coherent and consistent.

[0016] Variations of these and other such functionalities can also be used within the scope of the various embodiments, as would be obvious to a person skilled in the art in view of the teachings and proposals contained herein.

[0017] Fig. Figure 1 illustrates an exemplary system for improving the frame rate and visual smoothness of real-time video streams by multi-frame interpolation according to at least one embodiment. The in Fig. Figure 1, an illustrated exemplary embodiment, provides an overview of how input images 110 are processed to generate intermediate images that contribute to the overall quality of the output video stream. A method can begin with the input video stream 110, which represents a sequence of video images that can be processed. The input video stream 110 can include preceding images, image A, image B, and subsequent images. Preceding images can refer to the images that precede image A and image B in the sequence. Image A and image B are two consecutive images selected for analysis with the goal of generating intermediate images that lie between them. Subsequent images are those that are processed after image B.

[0018] To generate intermediate images between image A and image B, image A and image B can be passed to a neural (e.g., classification) network 120. The neural network 120 can analyze pixels within image A and image B and extract critical information such as motion properties 130, which describe how each pixel moves or behaves between the two images. The network can output confidence scores for each motion candidate, indicating how well the motion of each pixel is described by a given motion vector or by remaining static. For example, one confidence score might indicate how accurately the motion of a pixel is captured by the motion vectors provided by the game engine. Another confidence score might represent how well the motion of a pixel is described by a static or dynamic position.No movement is described, which can apply to areas containing reflections, UI elements, or other non-moving objects. Two confidence values ​​can be generated for each pixel, with each confidence value reflecting the probability that the movement is characterized by the respective motion candidate.

[0019] In some embodiments, additional data is provided to the neural network 120. For example, in addition to the two input images (e.g., grayscale versions of the images representing brightness), quality masks are computed to help evaluate the reliability of the motion data in image A and image B. A quality mask can be a type of data that indicates how well the motion vectors represent the actual movement of pixels between images. To compute these quality masks, one input image (e.g., image A) is warped using the motion vectors to align with the other image (e.g., image B). Based on the warped image A and image B, a difference is calculated as an indication of how reliable the motion vectors are.Such a procedure can be performed in both directions—from image A to image B and from image B to image A—as well as from the image that immediately precedes image A. The difference between image A and image B can also be calculated to provide an indicator of the quality of the assumption of static motion, assuming that certain pixels do not move between the images.

[0020] Additionally, occlusion masks can be generated to identify which pixels are hidden or obscured between images. Occlusion occurs when an object in one image obscures the view of another object in the next image, causing some pixels to disappear or become invisible. For example, a pixel in image B can be considered obscured by image A if none of the motion vectors from image A point to that pixel in image B. These occlusion masks are used as input to the neural classification network to improve the accuracy of the motion prediction process.

[0021] In some embodiments, confidence values ​​from image B of the previous iteration can be reused in image A of the current iteration if the previous image B is identical to the current image A. To explain this more clearly, when generating intermediate images, the method typically moves sequentially through pairs of images, starting with image A and image B. Once one or more intermediate images are generated between these two images, such a method moves on to the next pair, which includes the original image B, now serving as the new image A, and the subsequent image C, which also serves as the new image B. Since image B from the previous iteration becomes the new image A in the current iteration, the motion properties and confidence values ​​already calculated for image B (in the previous iteration) remain valid for image A in the current iteration.Such reuse of confidence values ​​further optimizes computational efficiency by reducing redundant calculations, which improves both the processing speed and the accuracy of the generated intermediate images.

[0022] Based on the specified motion properties 130, a candidate image A and a candidate image B are generated. Candidate images A 131 and B 132 can be generated by warping the original images (e.g., image A and image B) to an intermediate point between image A 131 and image B 132. The warping process adjusts the pixels in the original images according to their motion properties and places them in positions that reflect their expected locations at the intermediate point. Once candidate images A and B are generated, the next step involves generating the final intermediate image. This image is created by blending candidate images A and B.The blending process involves aligning the pixels of both candidate images and merging their motion data to create a coherent visual representation that effectively captures the transition between image A and image B. In one embodiment, a neural network can be used to generate the intermediate image 133, where the candidate images can be aligned and blended. Finally, the processed video stream is generated, which includes the newly generated intermediate image 133 alongside the original images. By integrating these intermediate images, the output video stream 140 achieves smoother motion and improved visual quality.

[0023] Fig. 2A and Fig. Figure 2B further illustrates how intermediate frames are generated in a timeline or over time during real-time video stream processing, whereby Fig. 2A illustrates the method for generating a single intermediate image between two successive images and Fig. Figure 2B illustrates the method for generating multiple intermediate images to further increase the frame rate according to at least one embodiment. Each figure is described in more detail below.

[0024] Fig. Figure 2A illustrates an example timing diagram of image processing for generating an intermediate frame 280 between two consecutive frames, frame A 220 and frame B 230. In digital video processing, frames are discrete snapshots of visual data captured at specific times (e.g., frames 210, 240, 250, captured at times t0, t3, and t4). These frames are not recorded continuously over time but are instead captured at regular intervals (e.g., 30 frames per second). To improve perceived motion smoothness, especially in scenes with high motion, it is necessary to generate additional frames within these intervals. As shown in Figure 2A, the interpolation of these frames is performed to create an intermediate frame 280 between two consecutive frames, 220 and 230. Fig. As illustrated in 2A, the timeline begins in Fig. Image 2A is generated at time t0 with image t0, a preceding image that precedes image A 220. At time t1, image A 220 is positioned, followed by image B 230, which is positioned at time t2, with a time interval between the two. To generate an intermediate image 280 at time t1.5 between image A 220 and image B 230, such a system can first derive the motion properties 290 and 291 of images A and B, respectively, using a neural classification network. The motion properties can include details about how each pixel in the images moves or behaves over the time interval between t1 and t2, for example, whether it follows motion vectors or remains static.

[0025] Based on these motion properties, candidate images A 260 and B 270 are generated. These candidate images correspond to the same time, t1.5, which is the desired point for the intermediate image. The candidate images are generated by warping image A and image B with respect to time t1.5 according to the motion properties determined by the neural network. After candidate images A and B have been generated, they are aligned and blended to produce the intermediate image 280 at time t1.5. In one embodiment, the intermediate images 280 are generated by a second neural network that takes candidate images A and B as input. The second neural network can blend and merge the pixel data from both candidate images. Fig. Figure 2A illustrates the generation of a single intermediate frame 280. The use of motion properties to generate multiple intermediate frames is described in section 280. Fig. 2B discussed further.

[0026] Fig. Figure 2B illustrates an example method that generates multiple intermediate frames between two original frames to further increase the frame rate. In scenarios where even higher frame rates are desired, generating only one intermediate frame may not be sufficient. Fig. 2B shows how multiple intermediate frames can be generated. The time axis in Fig. 2B begins with image A 211, positioned at time t0, and image B 221, positioned at time t3. The motion properties 231 and 241 can be extracted by the neural network. These motion properties are used to generate candidate images A 251 and B 271 at time t1, representing the intermediate point for a first intermediate image (e.g., at time 1 / 3 between t0 and t3). Candidate images A 251 and B 271 are then blended to generate intermediate image 272. To generate the next intermediate image 282, the same motion properties 231 and 241 are used to create additional candidate images.For example, at time t2, another set of candidate images, candidate image A 261 and candidate image B 281, is generated based on the same motion properties derived from the original images, but adjusted to align at a different intermediate time, t2. Mixing these candidate images results in another intermediate image 282.

[0027] In one embodiment, such a method can be repeated for as many intermediate frames as needed, with each frame being derived from a pair of candidate frames generated using the same set of motion properties derived during the initial processing of the original frames (e.g., frame A and frame B). By reusing motion data across multiple frames, such a system significantly reduces computational overhead. The output is a higher frame rate with smoother transitions.

[0028] Fig. Figure 3A illustrates a detailed procedure for generating candidate images according to one embodiment. In such a procedure, image A 301 and image B 302 can each be processed by a neural network 310, such as a neural classifier or classification network. The neural network 310 can analyze each pixel within image A and image B to determine motion properties that describe how each pixel behaves or moves between image A and image B. In one embodiment, the neural network 310 can output two confidence values ​​for pixels in the input images. For example, for a pixel in image A, the neural network can generate confidence values ​​for motion vectors 321 and confidence values ​​for static or no motion 322. For a corresponding pixel in image B, confidence values ​​for motion vectors 322 and for static or no motion 332 can be generated.The confidence values ​​for motion vectors can represent the probability that a pixel's movement can be matched to the provided motion vectors, such as those generated by a game engine or derived from video content. The confidence values ​​for static or no movement indicate how well a pixel remains consistent with an assumption of zero movement, which is particularly relevant for elements expected to remain stationary, such as user interface elements, background features, or reflections.

[0029] Each pixel in image A 301 can be assigned confidence values ​​321 and 322, while each pixel in image B 302 can be assigned confidence values ​​331 and 332. These confidence values ​​are used in determining blend weights 340 in the subsequent steps. The blend weights 340 can refer to the values ​​that indicate how much influence each motion property—such as motion vectors or static / no motion—should have on the final position and / or color of a pixel in the generated intermediate image. In other words, the blend weights can control how the original image and its warped version are combined to produce a smooth transition. In one embodiment, the blend weights 340 are computed by applying a SoftMax function to the confidence values.The SoftMax function can normalize confidence values ​​and transform them into probabilities that determine the influence of motion vectors and static (no motion) on each pixel during blending. For example, if a pixel's motion is predominantly matched to the motion vector, the blend weight will shift in the direction of the motion vector. Conversely, if the confidence value of static (no motion) is higher, the blend weight will reflect this, keeping the pixel in place to maintain consistency with static elements in the scene.

[0030] The mixing weights 340 and 350 are then used to generate candidate image A 360 and candidate image B 370, respectively. Candidate image A 360 can be generated by blending the warped version of image A with the original image A using the calculated mixing weights. Similarly, candidate image B 370 is generated by blending the motion vector-modified version of image B with the original image B. These candidate images represent intermediate versions of image A and image B, adjusted to align at an intermediate point between t1 and t2.

[0031] After generating candidate images A 360 and B 370, these candidate images are processed by another neural network 380. This neural network 380 further refines and aligns the candidate images to generate an intermediate image 390 that represents the transition between image A and image B. The neural network can predict any remaining motion properties not fully captured in the initial blending process, such as lighting changes or reflections. The neural network 380 can also compute additional blending weights as needed to ensure a seamless merging of the candidate images.

[0032] In one embodiment, neural network 380 can be designed as a smaller, more efficient model compared to the initial neural network 310 used for classification, since neural network 380 utilizes the stored results from the earlier processing stages. Examples of neural networks that can be used as neural network 380 include, but are not limited to, convolutional neural networks (CNNs), vision transformers (ViTs), recurrent neural networks (RNNs), transformer models, or any neural network tailored for image processing tasks. Neural network 380 can predict any remaining motion characteristics that were not fully captured in the initial mixing process.For example, the neural network 380 can address both fixed-position elements and dynamic elements that were initially classified as "static" because their movement was not well described by the game engine's motion vectors. For instance, the neural network 380 can detect and correct subtle lighting changes, reflections, contextual information, or minor inconsistencies in object movement that were not fully addressed during the creation of candidate images A and B. In one embodiment, the static elements (or elements not well described by motion vectors) may exhibit significant movement between images A and B, such as the shifting of dynamic shadows, reflections, or moving user interface elements.

[0033] In addition to refining motion characteristics, the neural network 380 can also compute additional blending weights as needed. These additional weights can ensure a smooth merging of the candidate images by adjusting how pixels are combined based on the latest motion predictions. For example, if the initial blending missed a slight shift in an object's shadow due to changing light sources, the neural network 380 can adjust the blending weights to correct this, so that the shadow is properly aligned in the final intermediate image. In one embodiment, the neural network can compute additional blending weights to adjust how pixels are combined based on the updated motion predictions.For example, if the shadow of an object has moved significantly due to changing light sources and this was not captured during the initial mixing, the neural network 380 can correct the mixing weights to ensure proper shadow alignment in the final intermediate frame. Therefore, the corrections performed by the neural network 380 may not be limited to subtle adjustments but can also include significant refinements where the motion characteristics of dynamic elements were not accurately captured. The neural network 380 can output an intermediate frame 390, which is integrated into the video stream.

[0034] Fig. Figure 3B illustrates another exemplary embodiment in which the generation of intermediate frames includes additional motion properties derived from additional motion candidates. As shown in Fig. As illustrated in Figure 3B, the method can further include generating multiple motion candidates beyond the pure motion vectors and static or no motion. For example, additional motion properties can be derived from optical flow models that can analyze changes in pixel intensity between images to provide a more detailed understanding of the motion. In such embodiments, the neural network can output additional confidence scores for an optical flow, and the mixing procedure can involve combining motion vectors, static or no motion, and optical flow components to generate the candidate images.

[0035] The in Fig. The procedure illustrated in 3B can begin with input images 311 containing image A 312 and image B 313, which represent successive images from a video stream. These images are processed by a neural network 314 and an optical flow model 315. The neural network 314 can be similar to the one in Fig. The neural network 310 described in 3A operates. The neural network 314 can analyze pixels within image A 312 and image B 313 to generate confidence values. For image A, the neural network 314 can output confidence values ​​for motion vectors 316 and confidence values ​​for static assumptions 317 (e.g., any motion not well described by motion vectors). For image B, the neural network similarly generates confidence values ​​for motion vectors 341 and confidence values ​​for static assumptions 342 (e.g., any motion not well described by motion vectors).

[0036] Additionally, the optical flow model 315 can generate confidence values ​​for optical flow 318 for image A and confidence values ​​for optical flow 343 for image B. The optical flow model 315 can estimate these motion patterns between image A and image B by analyzing the changes in pixel intensity across the images. The optical flow model 315 can provide an understanding of pixel motion, especially in complex scenes where the motion is non-linear or uncomplicated. The optical flow model can generate additional confidence values ​​that complement the assessments of the motion vectors and the static assumptions generated by the neural network.

[0037] While the optical flow model 315 is illustrated as one example of a candidate for generating motion properties, other candidates can also be included. Additional candidates can represent different motion properties or extract other types of information from the original images that can be reused in the intermediate image generation process. For example, alternative candidates can include models that analyze depth information, texture patterns, or semantic content within the images, such as identifying specific objects or regions that require different handling during the blending process. By including a variety of candidates, such a system can utilize multiple sources of motion-related data or other image attributes, which can improve the flexibility and accuracy of intermediate image generation.

[0038] Once the confidence values ​​for image A 312 and image B 313 have been generated, the procedure proceeds with the mixture weight determination 344 and 345. Here, the mixture weights are calculated using the combined confidence values ​​from the motion vectors, the static assumptions, and the optical flow. The mixture weights are determined using a SoftMax (or similar) function that can transform confidence values ​​into probabilities. These weights are then used to generate candidate image A 346 and candidate image B 347. These candidate images can integrate motion properties from three sources: motion vectors, static assumptions, and optical flow. After the candidate images have been generated, a second neural network 348 further refines and adjusts candidate image A 346 and candidate image B 347.The neural network 348 can function similarly to the neural network 380 and predict any remaining motion or alignment problems or adaptation problems that were not fully addressed by the mixing.

[0039] Fig. Figure 4 illustrates a detailed method for generating an intermediate image from two consecutive images, image A and image B, within a video stream according to at least one embodiment. It is understood that for this and other methods disclosed herein, additional, fewer, or alternative steps may be performed in similar or alternative sequences, or at least partially in parallel, within the scope of the various embodiments, unless otherwise specified. The method can begin with obtaining the two consecutive images, which serve as input for the subsequent analysis and interpolation.

[0040] The procedure continues with steps 420 and 421, which use a neural network to analyze each pixel within image A and image B. For image A, the neural network evaluates the pixel data to generate two confidence scores: one representing the probability that the pixel's movement matches the provided motion vectors, and the other representing a static assumption that the pixel will remain in a fixed position. A similar analysis is performed for image B, resulting in two corresponding confidence scores for each pixel, as illustrated in steps 430 and 431.

[0041] Once the confidence values ​​for both images have been determined, the procedure involves warping the motion vector confidence values ​​for image A and image B, as shown in steps 440 and 441. This warping adjusts the pixels and their associated confidence values ​​to reflect their expected positions at an intermediate point between image A and image B.

[0042] After warping, the procedure calculates blend weights based on the warped confidence values ​​for both image A and image B, as illustrated in steps 450 and 451. These blend weights determine how the warped motion vector data and the original pixel data should be combined. Using the blend weights, candidate image A and candidate image B are generated, as illustrated in steps 460 and 461. Candidate image A is created by combining the image produced with the warped motion vectors with the original image A, applying the blend weights to ensure that the resulting candidate image accurately reflects the motion between the original images.The candidate image B is generated in a similar way, combining the image produced with the warped motion vectors with the original image B.

[0043] The candidate images are then processed by a second neural network. This network can take the candidate images as input and align and / or blend them, as described in step 470. The second neural network can ensure that any remaining discrepancies in motion or visual properties are resolved. The second neural network can adjust the blending based on additional predictions, such as fine-tuning the orientation of moving objects or taking into account subtle changes in lighting.

[0044] Once the candidate images have been refined and aligned, they are merged to create the final intermediate frame. This intermediate frame, representing the transition between image A and image B, is then integrated into the video stream to improve overall smoothness and image quality. The process concludes with the output of the generated intermediate frame, ready for display or further processing.

[0045] Fig. Figure 500 is a flowchart illustrating an exemplary generation of an intermediate frame between two successive frames from a video stream according to at least one embodiment. The method can begin with step 510, in which a first frame and a second frame are received from a sequence of frames in a video stream. These frames are selected to generate an intermediate frame that is positioned between them. As specified in step 520, a neural network can be used to determine a variety of motion properties for at least a first set of pixels in the first frame and a corresponding second set of pixels in the second frame. These motion properties can describe how the pixels move or behave between the two frames.

[0046] Step 530 involves generating a first candidate image based on the motion properties. This first candidate image can be generated based on motion properties, such as weighted motion vectors, applied to the pixels of the first image in a first direction corresponding to the expected motion toward, or relative to, the intermediate image. Similarly, in step 540, a second candidate image is generated by applying the motion properties, such as weighted motion vectors, to the pixels of the second image in an opposite direction. The second candidate image reflects the expected motion of pixels from the second image toward the intermediate image.

[0047] In step 550, the intermediate image is generated by aligning and blending the pixel values ​​of the first and second candidate images using a second neural network. The generated intermediate image is then provided for display on a display device (560) to improve the smoothness and visual quality of the video stream and contribute to an enhanced user experience.

[0048] In a possible additional embodiment, an alternative approach could involve generating intermediate frames based solely on the color information of the input images. In this embodiment, instead of relying on motion vectors or optical flow models, the method can analyze the color patterns across successive images to determine how pixels can be effectively blended to generate the intermediate frames. For example, using color images as the only input, a neural network can be trained to recognize the color gradients and textures present in the images and interpolate between them, effectively estimating the intermediate states of the visual content. This can be particularly advantageous in situations where motion vectors are unavailable, such as in pre-rendered video content, or where motion vectors might introduce artifacts.

[0049] In addition to their applications in generating intermediate frames, the methods described here can also be used for video upsampling to improve the resolution or frame rate of video content. Video upsampling is particularly valuable in situations where the original video content has a lower resolution or frame rate and there is a need to improve the visual quality for display on higher-resolution screens or in more demanding visual environments. For example, in video streaming services, where bandwidth limitations often result in the distribution of lower-resolution content, video upsampling can be used to improve the perceived quality of the content by instantly increasing its resolution or frame rate.This can be particularly effective when combined with the image interpolation methods described here, as the generation of additional images makes the upsampled video appear smoother and more natural.

[0050] Similarly, in the context of virtual reality (VR) or augmented reality (AR), where maintaining high resolution and frame rate is crucial for an immersive experience, video upsampling techniques can be integrated with the image generation methods described. This integration can ensure that even content originally recorded or played back at a lower quality can be enhanced to meet the high standards required by VR and AR systems.

[0051] Fig. Figure 6 illustrates an exemplary networked system 600 that includes an intermediate frame generation system according to various embodiments. The exemplary networked system 600 can be used to provide, generate, modify, encode, process, and / or transmit data or other content. The exemplary networked system 600 can include a client device 602, another client device 603, a network 614, a third-party service 660, and a provider environment 616 that includes an intermediate frame generation system 630.

[0052] The client device 602 can generate or receive data for a session using components of an application 607 on the client device 602 and data stored locally on that client device 602. For example, a user can use a client device 602 to generate intermediate images using the application 607. Although only one client device 602 is illustrated in detail, the exemplary networked system 600 may contain one or more other client devices 603 that can communicate with the provider environment 616 over the network 614.A client device 602 can be any suitable computing device capable of enabling a user to perform tasks related to real-time multi-image interpolation, as discussed herein, such as a desktop computer, notebook computer, computer workstation, game console, set-top box, streaming device, smartphone, tablet computer, VR headset, AR glasses, portable computer, or smart television. In at least one embodiment, a user can access real-time multi-image interpolation functionality using a user interface (UI) 606 running on a client device 602, although at least some functionality can also be performed on a remote device, a networked device, or via a cloud computing platform.In at least one embodiment, a user can provide input to the UI 606, such as via a touch-sensitive display 604 or by moving a mouse pointer displayed on a screen. In another embodiment, a user can provide input such as preferences and configuration data for an application 607. The application 607 can be provided to the user by the provider environment 616 for download to the client device 602. In at least one embodiment, a client device can include at least one processor 608 (e.g., a CPU or GPU), memory 612, and storage 610 to run the application 607 and / or perform tasks on behalf of the application 607.

[0053] In one embodiment, each client device 602 can make a request over at least one wired or wireless network, such as the Internet, Ethernet, a local area network (LAN), or a cellular network, among other such options. In this example, these requests can be made to an address associated with a cloud provider, which can operate or control one or more electronic resources in a cloud provider environment, such as a data center or server farm. In at least one embodiment, the request can be received or processed by at least one edge server located at the network edge and outside of at least one security layer associated with the cloud provider environment.This reduces latency by allowing client devices to interact with servers that are closer together, while also improving the security of resources in the cloud provider environment.

[0054] The network 614 can represent the communication paths between the client device 602, the provider environment 616, the other client device 603, and the third-party service 660. Through the network 614, the client device 602 can send input information associated with stream data processing over the network 614. The information can be received by a remote computing system, such as one that may be part of a resource provider environment 616. In one embodiment, the network 614 is the Internet. The network 614 can include any suitable network, including an intranet, the Internet, a cellular network, a local area network (LAN), or any other such network or combination thereof, and communication over a network can be enabled via wired and / or wireless connections.Network 614 can also use dedicated or private communication links that are not necessarily part of the Internet. In one embodiment, Network 614 uses standard communication technologies and / or protocols. Thus, Network 614 can include links that use technologies such as Ethernet, Wi-Fi, Integrated Services Digital Network (ISDN), Digital Subscriber Lines (DSL), Asynchronous Transfer Mode (ATM), etc. Likewise, the network protocols used in Network 614 can include Multiprotocol Label Switching (MPLS), Transmission Control Protocol / Internet Protocol (TCP / IP), Hypertext Transport Protocol (HTTP), Simple Mail Transfer Protocol (SMTP), File Transfer Protocol (FTP), etc. In one embodiment, at least some of the links use mobile network technologies, such as Long Term Evolution (LTE).The data exchanged over the 614 network can be represented using various technologies or formats, including Hypertext Markup Language (XML), Wireless Access Protocol (WAP), Short Message Service (SMS), etc. Additionally, all or some of the connections can be encrypted using conventional encryption technologies, such as Secure Sockets Layer (SSL), Secure HTTP, or Virtual Private Networks (VPNs). In another embodiment, the 602 client device can use customer-specific and / or dedicated data communication technologies instead of, or in addition to, those described above.

[0055] The provider environment 616 can include any suitable components for receiving requests and sending back information or performing actions in response to those requests. In the Fig. In the embodiment illustrated in Figure 6, the provider environment 616 can include an interface 618 and a server 620, which contain various components for performing tasks associated with real-time multi-image interpolation. In at least one embodiment, the provider environment 616 can include web servers and / or application servers for receiving and processing requests and then sending back data or other content or information in response to the request.

[0056] Interface 618 can receive communications to server 620. In at least one embodiment, interface 618 can include application programming interfaces (APIs) or other exposed interfaces that allow a user to make requests to server 620. In at least one embodiment, interface 618 can also include other components, such as at least one web server, routing components, or load balancers. In at least one embodiment, components of interface 618 can determine the type of request or communication and can route the request to a suitable system or service, such as a frame interpolation system 630.

[0057] The server 620 can include a transfer manager 622, a content application 624, an object repository 634, and a user database 636. The server 620 can receive requests and data from the client device 602, perform tasks associated with the requests, and send results or other data to the client device 602. In at least one embodiment, a content application 624 running on the server 620 (for example, a cloud server or edge server) can initiate a session associated with the client device 602, such as using a session manager and user data stored in a user database 636, and can cause content, such as one or more object representations from an object repository 634, to be selected by a content manager 626 for processing.At least some of the generated content, such as results from stream data processing, can be transferred to the client device 602 using a suitable transfer manager 622 for transmission by downloading, streaming, or another such transmission channel. An encoder can be used to encode and / or compress at least some of this data before it is transferred to the client device 602. In at least one embodiment, the client device 602 receiving such content can provide this content to a suitable application 607 for selecting, providing, synthesizing, modifying, or using the content for presentation (or other purposes) on or through the client device 602.A decoder can also be used to decode data received via the network 614 for presentation via the client device 602, such as image or video content displayed on a touchscreen 604. In at least one embodiment, at least part of the content can already be stored, played back, or accessible on the client device 602, so that transmission via the network 614 is not necessary for at least this part of the content, for example, if the content has been previously downloaded or stored locally on a hard disk or optical disk. In at least one embodiment, a transmission mechanism, such as data streaming, can be used to transfer the content from the server 620 or the user database 636 to the client device 602.In at least one embodiment, at least part of this content can be obtained, enhanced, and / or streamed from another source, such as a third-party service 660 or another client device 603, which may also include a content application 662 for generating, enhancing, or providing content. In at least one embodiment, parts of this functionality can be performed using multiple computing devices or multiple processors within one or more computing devices, which may include, for example, a combination of CPUs and GPUs.

[0058] In at least one embodiment, the Server 620 can include a processor, such as a central processing unit (CPU). However, in at least one embodiment, resources in such environments can utilize GPUs to process data for at least certain types of requirements. In at least one embodiment, GPUs with thousands of cores are designed to handle substantial parallel workloads and have therefore become popular in deep learning for training neural networks and generating predictions.While using GPUs for offline builds has enabled faster training of larger and more complex models in at least one embodiment, offline prediction generation implies that either request-time input features cannot be used or predictions for all permutations of features must be generated and stored in a lookup table to serve real-time requirements. If, in at least one embodiment, a deep learning framework supports a CPU mode and a model is small and simple enough to perform forward coupling to a CPU with reasonable latency, then a service could host a model on a CPU instance. In at least one embodiment, training can be performed offline on a GPU and inference can be performed in real time on a CPU.If, in at least one embodiment, a CPU-based approach is not a practical option, then a service can run on a GPU instance. However, since GPUs have different performance and cost characteristics than CPUs, in at least one embodiment, running a service that offloads a runtime algorithm to a GPU may require it to be designed differently than a CPU-based service.

[0059] The server 620 can contain a content application 624, which includes a content manager 626 and an intermediate picture generator 630. As discussed earlier, the content manager 626 can send objects, such as records and instructions, from the object repository 634, along with requests and other data, from the client device 602 to an intermediate picture generator 630 for stream data processing. The intermediate picture generator 630 can process input data and provide the results to the transfer manager 622 for return to the client device 602. The intermediate picture generator 630 can also use local records or records provided by the third-party service 660 for stream data processing. INFERENCE AND TRAINING LOGIC

[0060] Fig. Figure 7A illustrates an inference and / or training logic 715 used to perform inference and / or training operations associated with one or more embodiments. Details regarding the inference and / or training logic 715 are given below in conjunction with Fig. 7A and / or 7B provided.

[0061] In at least one embodiment, the inference and / or training logic 715 can, without limitation, include code and / or data memory 701 to store forward and / or output weighting and / or input / output data and / or other parameters for configuring neurons or layers of a neural network that is trained in aspects of one or more embodiments and / or used for inference. In at least one embodiment, the training logic 715 can include or be coupled to code and / or data memory 701 to store graph code or other software for controlling the timing and / or order in which weighting and / or other parameter information is loaded to configure logic that includes integer and / or floating-point units (collectively, arithmetic logic units (ALUs)).In at least one embodiment, a code, such as a graph code, loads weighting or other parameter information based on the architecture of a neural network to which the code corresponds into the processor ALU. In at least one embodiment, the code and / or data memory 701 stores the weighting parameters and / or input / output data of each layer of a neural network that is trained or used in conjunction with one or more embodiments during the forward propagation of input / output data and / or weighting parameters during training and / or inference using aspects of one or more embodiments. In at least one embodiment, any portion of the code and / or data memory 701 may be included in another on-chip or off-chip data memory, including the L1, L2, or L3 cache or system memory of a processor.

[0062] In at least one embodiment, any section of the code and / or data memory 701 can be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data memory 701 can be a cache memory, a dynamic randomly addressable memory (“DRAM”), a static randomly addressable memory (“SRAM”), a non-volatile memory (e.g., flash memory), or another type of memory.In at least one embodiment, the choice of whether the code and / or data storage 701 is, for example, internal or external to a processor, or comprises DRAM, SRAM, Flash or another type of memory, may depend on the available chip-internal versus chip-external memory, latency requirements of executed training and / or inference functions, batch size of data used in inferencing and / or training a neural network, or a combination of these factors.

[0063] In at least one embodiment, the inference and / or training logic 715 may, without limitation, include a code and / or data store 705 to store backward and / or output weights and / or input / output data corresponding to neurons or layers of a neural network that is trained and / or used for inference in aspects of one or more embodiments. In at least one embodiment, the code and / or data store 705 stores weighting parameters and / or input / output data of each layer of a neural network that is trained or used in conjunction with one or more embodiments during backward propagation of input / output data and / or weighting parameters during training and / or inference using aspects of one or more embodiments.In at least one embodiment, the training logic 715 can include or be coupled to a code and / or data memory 705 to store graph code or other software to control the timing and / or order in which weighting and / or other parameter information is loaded, and to configure logic that includes integer and / or floating-point units (collectively, arithmetic logic units (ALUs)). In at least one embodiment, a code, such as a graph code, loads weighting or other parameter information into the processor ALU based on the architecture of a neural network to which the code corresponds. In at least one embodiment, any portion of the code and / or data memory 705 can be contained in another on-chip or off-chip data memory, including the L1, L2, or L3 cache or system memory of a processor.In at least one embodiment, any section of the code and / or data memory 705 can be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data memory 705 can be a cache memory, a DRAM, an SRAM, a non-volatile memory (e.g., flash memory), or another type of memory. In at least one embodiment, the choice of whether the code and / or data memory 705 is, for example, internal or external to a processor, or consists of a DRAM, an SRAM, a flash memory, or some other type of memory, can depend on the available on-chip memory compared to the available off-chip memory, the latency requirements of training and / or inference functions performed, the batch size of data used in inferencing and / or training a neural network, or a combination of these factors.

[0064] In at least one embodiment, the code and / or data memory 701 and the code and / or data memory 705 can be separate memory structures. In at least one embodiment, the code and / or data memory 701 and the code and / or data memory 705 can be the same memory structure. In at least one embodiment, the code and / or data memory 701 and the code and / or data memory 705 can be partly the same memory structure and partly separate memory structures. In at least one embodiment, any part of the code and / or data memory 701 and the code and / or data memory 705 can be contained in another chip-internal or chip-external data memory, which includes an L1, L2, or L3 cache or system memory of a processor.

[0065] In at least one embodiment, the inference and / or training logic 715 may, without limitation, include one or more arithmetic logic units (“ALU(s)”) 710, including integer and / or floating-point units, to perform logical and / or mathematical operations that are based at least in part on or specified by training and / or inference code (e.g., graph code), wherein a result thereof may produce activations (e.g., output values ​​of layers or neurons within a neural network) that are stored in an activation memory 720 and are functions of input / output and / or weighting parameter data that are stored in the code and / or data memory 705 and / or the code and / or data memory 701.In at least one embodiment, activations stored in the activation memory 720 are generated according to algebraic and / or matrix-based mathematics performed by the ALU(s) 710 in response to the execution of instructions or other code, wherein weight values ​​stored in the code and / or data memory 705 and / or the code and / or data memory 701 are used as operands together with other values, such as systematic error values, gradient information, dynamic values, or other parameters or hyperparameters, one or all of which may be stored in the code and / or data memory 701 or the code and / or data memory 705 or in another on-chip or off-chip memory.

[0066] In at least one embodiment, the ALU(s) 710 are contained within one or more processors or other hardware logic devices or circuits, whereas in another embodiment, the ALU(s) 710 may be external to a processor or other hardware logic device or circuit that uses them (e.g., a coprocessor). In at least one embodiment, the ALU(s) 710 may be contained within the execution units of a processor or otherwise within a bank of ALUs that the execution units of a processor can access, either within the same processor or distributed across different processors of different types (e.g., central processing units, graphics processing units, fixed function units, etc.).In at least one embodiment, the code and / or data memory 701, the code and / or data memory 705, and the activation memory 720 can be located on the same processor or other hardware logic device or circuit, whereas in another embodiment, they can be located on different processors or other hardware logic devices or circuits, or a combination of identical and different processors or other hardware logic devices or circuits. In at least one embodiment, any portion of the activation memory 720 can be included in another on-chip or off-chip data memory, including the L1, L2, or L3 cache or system memory of a processor.Furthermore, inference and / or training code can be stored with other code that a processor or other hardware logic or circuitry can access and be retrieved and / or processed using the retrieval, decoding, scheduling, execution, shutdown, and / or other logical circuitry of a processor.

[0067] In at least one embodiment, the activation memory 720 can be a cache memory, a DRAM, an SRAM, a non-volatile memory (e.g., flash memory), or another type of memory. In at least one embodiment, the activation memory 720 can be located wholly or partially inside or outside one or more processors or other logic circuits. In at least one embodiment, the choice of whether the activation memory 720 is, for example, internal or external to a processor, or whether it consists of a DRAM, an SRAM, flash memory, or another type of memory, can depend on the available on-chip memory compared to the available external memory, the latency requirements of training and / or inference functions performed, the batch size of data used in inferencing and / or training a neural network, or a combination of these factors.In at least one embodiment, the in . Fig. 7A illustrated inference and / or training logic 715 can be used in conjunction with an application-specific integrated circuit (“ASIC”), such as a Google Tensorflow® Processing Unit, a Graphcore™ inference processing unit (IPU), or an Intel Corp. Nervana®-type processor (e.g., “Lake Crest”). In at least one embodiment, the Fig. Figure 7A illustrates inference and / or training logic 715 used in conjunction with central processing unit (CPU) hardware, graphics processing unit (GPU) hardware, or other hardware such as field programmable gate arrays (FPGAs).

[0068] Fig. Figure 7B illustrates inference and / or training logic 715 according to at least one or more embodiments. In at least one embodiment, the inference and / or training logic 715 may, without limitation, include hardware logic in which computing resources are dedicated or otherwise used exclusively in conjunction with weight values ​​or other information corresponding to one or more layers of neurons within a neural network. In at least one embodiment, the Fig. 7B illustrated inference and / or training logic 715 can be used in conjunction with an application-specific integrated circuit (ASIC), such as Google's Tensorflow® processing unit, a Graphcore™ inference processing unit (IPU), or an Intel Corp. Nervana® processor (e.g., "Lake Crest"). In at least one embodiment, the inference and / or training logic illustrated in Figure 7B can be used in conjunction with an application-specific integrated circuit (ASIC), such as Google's Tensorflow® processing unit, a Graphcore™ inference processing unit (IPU), or an Intel Corp. Nervana® processor (e.g., "Lake Crest"). In at least one embodiment, the logic illustrated in Figure 7B can be used in conjunction with an application-specific integrated circuit (ASIC), such as Google's Tensorflow® processing unit, a Graphcore™ inference processing unit (IPU), or an Intel Corp. Nervana® processor (e.g., "Lake Crest"). Fig. Figure 7B illustrates inference and / or training logic 715 used in conjunction with central processing unit (CPU) hardware, graphics processing unit (GPU) hardware, or other hardware, such as field-programmable gate arrays (FPGAs). In at least one embodiment, the inference and / or training logic 715 includes, without limitation, code and / or data memory 701 and code and / or data memory 705, which can be used to store code (e.g., graph code), weight values, and / or other information, including distortion values, gradient information, instantaneous values, and / or other parameter or hyperparameter information. In at least one embodiment, which is illustrated in Fig. As illustrated in Figure 7B, each of the code and / or data memory 701 and the code and / or data memory 705 is assigned to a dedicated computing resource, such as the computing hardware 702 and the computing hardware 706, respectively. In at least one embodiment, each of the computing hardware 702 and the computing hardware 706 comprises one or more ALUs that perform mathematical functions, such as linear algebraic functions, only on information stored in the code and / or data memory 701 and the code and / or data memory 705, respectively, with the result being stored in the activation memory 720.

[0069] In at least one embodiment, each of the code and / or data storage units 701 and 705 and the corresponding computing hardware 702 and 706 corresponds to different layers of a neural network, such that the resulting activation from a "memory / computing pair 701 / 702" of the code and / or data storage unit 701 and the computing hardware 702 is provided as input for the "memory / computing pair 705 / 706" of the code and / or data storage unit 705 and the computing hardware 706, in order to reflect the conceptual organization of a neural network. In at least one embodiment, each of the memory / computing pairs 701 / 702 and 705 / 706 can correspond to more than one layer of a neural network. In at least one embodiment, additional memory / computing pairs (not shown) may be included after or in parallel to the memory / computing pairs 701 / 702 and 705 / 706 in the inference and / or training logic 715. DATA CENTER

[0070] Fig. Figure 8 illustrates an exemplary data center 800, in which at least one embodiment can be used. In at least one embodiment, the data center 800 includes a data center infrastructure layer 810, a framework layer 820, a software layer 830, and an application layer 840.

[0071] In at least one embodiment, as in Fig. As shown in Figure 8, the data center infrastructure layer 810 can include a resource orchestrator 812, clustered compute resources 814, and node compute resources (“node CR”) 816(1) to 816(N), where “N” is any positive integer. In at least one embodiment, the node CRs 816(1)-816(N) can include, but are not limited to, any number of central processing units (“CPUs”) or other processors (including accelerators, field-programmable gate arrays (“FPGAs”), graphics processing units, etc.), memory devices (e.g., dynamic read-only memory), data storage devices (e.g., solid-state or disk drives), network input / output devices (“NW I / O” devices), network switches, virtual machines (“VMs”), power modules, and cooling modules, etc. In at least one embodiment, one or more node CRs can be controlled by the node CRs.816(1)-816(N) a server that has one or more of the aforementioned computing resources.

[0072] In at least one embodiment, the grouped compute resources 814 can include separate groupings of node CRs housed in one or more racks (not shown), or many racks housed in data centers at different geographic locations (also not shown). Separate groupings of node CRs within grouped compute resources 814 can include grouped compute, network, storage, or memory resources that can be configured or allocated to support one or more workloads. In at least one embodiment, multiple node CRs, including CPUs or processors, can be grouped in one or more racks to provide compute resources to support one or more workloads.In at least one embodiment, one or more racks can also include any number of power modules, cooling modules and network switches in any combination.

[0073] In at least one embodiment, the resource orchestrator 812 can configure or otherwise control one or more node CR 816(1) to 816(N) and / or grouped computing resources 814. In at least one embodiment, the resource orchestrator 812 can include a management unit of a software design infrastructure (“SDI”) for the data center 800. In at least one embodiment, the resource orchestrator 812 can include hardware, software, or a combination thereof.

[0074] In at least one embodiment, as in Fig. As shown in Figure 8, the framework layer 820 includes a task scheduler 822, a configuration manager 824, a resource manager 826, and a distributed file system 828. In at least one embodiment, the framework layer 820 can include a framework to support software 832 of software layer 830 and / or one or more applications 842 of application layer 840. In at least one embodiment, the software 832 or the application(s) 842 can each be web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. In at least one embodiment, the framework layer 820 can, without restriction, be a type of web application framework for free and open-source software, such as Apache Spark™ (hereinafter “Spark”), which can use the distributed file system 828 for large-scale data processing (e.g., “Big Data”).In at least one embodiment, the task scheduler 822 can include a Spark driver to facilitate the scheduling of workloads supported by various layers of the data center 800. In at least one embodiment, the configuration manager 824 can be able to configure various layers, such as the software layer 830 and the framework layer 820, including Spark and the distributed file system 828, to support large-scale data processing. In at least one embodiment, the resource manager 826 can be able to manage clustered or grouped compute resources allocated or assigned to support the distributed file system 828 and the task scheduler 822. In at least one embodiment, the clustered or grouped compute resources can include a grouped compute resource 814 on the data center infrastructure layer 810.In at least one embodiment, the resource manager 826 can coordinate with the resource orchestrator 812 to manage these allocated or assigned computing resources.

[0075] In at least one embodiment, the software 832, which is included in software layer 830, may include software that is used by at least parts of node CR 816(1) to 816(N), grouped computing resources 814, and / or the distributed file system 828 of framework layer 820. One or more types of software may include, but are not limited to, web browsing software, email scanning software, database software, and streaming video content software.

[0076] In at least one embodiment, the application(s) 842 included in the application layer 840 may include one or more types of applications used by at least parts of node CR 816(1) to 816(N), grouped compute resources 814, and / or the distributed file system 828 of the framework layer 820. One or more types of applications may include any number of a genomics application, a cognitive compute application, and a machine learning application, including, but not limited to, training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), or other machine learning applications used in conjunction with one or more embodiments.

[0077] In at least one embodiment, the configuration manager 824, the resource manager 826, and the resource orchestrator 812 can implement any number and type of self-modifying actions based on any amount and type of data acquired in any technically feasible manner. In at least one embodiment, self-modifying actions can relieve a data center operator of the data center 800 from potentially making poor configuration decisions and potentially avoiding underutilized and / or underperforming sections of a data center.

[0078] In at least one embodiment, the Data Center 800 may include tools, services, software, or other resources for training one or more machine learning models, or for predicting or deriving information using one or more machine learning models according to one or more embodiments described herein. For example, in at least one embodiment, a machine learning model may be trained by calculating weighting parameters according to a neural network architecture using software and computing resources previously described with reference to the Data Center 800.In at least one embodiment, trained machine learning models corresponding to one or more neural networks can be used to infer or predict information using resources previously described in relation to the Computing Center 800 and weighting parameters calculated using one or more training techniques described in this document.

[0079] In at least one embodiment, the data center can use CPUs, application-specific integrated circuits (ASICs), GPUs, FPGAs, or other hardware to perform training and / or inference using the resources described above. Furthermore, one or more of the software and / or hardware resources described above can be configured as a service to allow users to train or perform information inference, such as image recognition, speech recognition, or other artificial intelligence services.

[0080] The inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. Details regarding the inference and / or training logic 715 are provided below in conjunction with Fig. 7A and / or 7B are provided. In at least one embodiment, the inference and / or training logic 715 can be provided in the system from Fig. 8 for inference or prediction operations at least partially based on weighting parameters calculated using neural network training operations, neural network functions and / or neural network architectures or neural network use cases described in this document.

[0081] Such components can enable multi-image interpolation for an improved user experience. COMPUTER SYSTEMS

[0082] Fig. Figure 9 is a block diagram illustrating an exemplary computer system, which may be a system with interconnected devices and components, a system-on-a-chip (SoC), or a certain combination thereof, and is formed with a processor that may include execution units for executing an instruction, according to at least one embodiment. In at least one embodiment, the computer system may, without limitation, include a component, such as a processor, to employ execution units that include logic for performing algorithms on process data, according to the present disclosure, as in the embodiment described in this document.In at least one embodiment, the Computer System 900 may include processors such as the PENTIUM® processor family, Xeon™, Itanium®, XScale™ and / or StrongARM™, Intel® Core™ or Intel® Nervana™ microprocessors available from Intel Corporation in Santa Clara, California, although other systems (including PCs with other microprocessors, technical workstations, set-top boxes, and the like) may also be used. In at least one embodiment, the Computer System 900 may run a version of the WINDOWS operating system available from Microsoft Corporation in Redmond, Washington, although other operating systems (for example, UNIX and Linux), embedded software, and / or graphical user interfaces may also be used.

[0083] The embodiments can be used on other devices, such as handheld devices and embedded applications. Some examples of handheld devices are mobile phones, Internet Protocol devices, digital cameras, personal digital assistants (PDAs), and portable PCs. In at least one embodiment, embedded applications can include a microcontroller, a digital signal processor (DSP), a system-on-a-chip, network computers (NetPCs), set-top boxes, network hubs, wide-area network switches (WANs), or any other system capable of executing one or more instructions according to at least one embodiment.

[0084] In at least one embodiment, the computer system 900 can, without limitation, include a processor 902, which can, without limitation, include one or more execution units 908 for training and / or inferring a machine learning model according to the techniques described in this document. In at least one embodiment, the computer system 900 is a single-processor desktop or server system, but in another embodiment, the computer system 900 can be a multi-processor system.In at least one embodiment, the processor 902 can, without restriction, include a complex instruction set computing (“CISC”) microprocessor, a reduced instruction set computing (“RISC”) microprocessor, a very long instruction word (“VLIW”) microprocessor, a processor implementing a combination of instruction sets, or any other processing device, such as a digital signal processor. In at least one embodiment, the processor 902 can be coupled to a processor bus 910, which can transmit data signals between the processor 902 and other components in the computer system 900.

[0085] In at least one embodiment, the processor 902 can, without limitation, include a Level 1 (“L1”) internal cache memory (“cache”) 904. In at least one embodiment, the processor 902 can have a single internal cache or multiple levels of internal cache. In at least one embodiment, the cache memory can be located outside the processor 902. Other embodiments, depending on the specific implementation and requirements, can also include a combination of both internal and external caches. In at least one embodiment, the register file 906 can store various types of data in various registers, including, without limitation, integer registers, floating-point registers, status registers, and instruction pointer registers.

[0086] In at least one embodiment, the execution unit 908, which includes without limitation logic for performing integer and floating-point operations, is also located in the processor 902. In at least one embodiment, the processor 902 may also include a microcode ("uCode") read-only memory (ROM) that stores microcode for certain macro instructions. In at least one embodiment, the execution unit(s) 908 may include logic for handling a packed instruction set 909. In at least one embodiment, by including a packed instruction set 909 in an instruction set of a general-purpose processor 902, together with associated circuitry for executing instructions, operations used by numerous multimedia applications can be performed using compressed data in a general-purpose processor 902.In one or more embodiments, numerous multimedia applications can be accelerated and executed more efficiently by using the full width of a processor's data bus to perform operations on compressed data, thereby potentially eliminating the need to transfer smaller data units across the processor's data bus to perform one or more operations on only one data element at a time.

[0087] In at least one embodiment, the execution unit 908 can also be used in microcontrollers, embedded processors, graphics devices, DSPs, and other types of logic circuits. In at least one embodiment, the computer system 900 can include a memory 920 without restriction. In at least one embodiment, the memory 920 can be implemented as a dynamic random access memory (“DRAM”) device, a static random access memory (“SRAM”) device, a flash memory device, or another type of memory device. In at least one embodiment, the memory 920 can store instructions 919 and / or data 921 represented by data signals that can be executed by the processor 902.

[0088] In at least one embodiment, a system logic chip can be coupled to the processor bus 910 and the memory 920. In at least one embodiment, the system logic chip can include, without restriction, a memory controller hub (“MCH”) 916, and the processor 902 can communicate with the MCH 916 via the processor bus 910. In at least one embodiment, the MCH 916 can provide a high-bandwidth memory path 918 for the memory 920 for storing instructions and data, and for storing graphics instructions, data, and textures. In at least one embodiment, the MCH 916 can route data signals between the processor 902, the memory 920, and other components in the computer system 900, and bridge data signals between the processor bus 910, the memory 920, and a system I / O 922. In at least one embodiment, the system logic chip can provide a graphics port for coupling to a graphics controller.In at least one embodiment, the MCH 916 can be coupled to the memory 920 via a high-bandwidth memory path 918, and a graphics / video card 912 can be coupled to the MCH 916 via an Accelerated Graphics Port (“AGP”) interconnection 914.

[0089] In at least one embodiment, the computer system 900 can use the system I / O 922, which is a proprietary node interface bus, to couple the MCH 916 to the I / O control hub (“ICH”) 930. In at least one embodiment, the ICH 930 can provide direct connections to some I / O devices via a local I / O bus. In at least one embodiment, the local I / O bus can, without restriction, include a high-speed I / O bus for connecting peripheral devices to the memory 920, the chipset, and the processor 902. Examples can include, without limitation, an audio control 929, a firmware hub (“Flash BIOS”) 928, a wireless transceiver 926, a data storage device 924, a legacy I / O control 923 containing user input and keyboard interfaces 925, a serial expansion port 927, such as a universal serial bus (“USB”) port, and a network control 934.The 924 data storage device can include a hard disk drive, a floppy disk drive, a CD-ROM device, a flash memory device, or another mass storage device.

[0090] Illustrated in at least one embodiment Fig. 9 a system that includes interconnected hardware devices or “chips”, whereas in other embodiments Fig. 9 illustrates an exemplary system on a chip (“SoC”). In at least one embodiment, the devices can be interconnected using proprietary interconnects, standardized interconnects (e.g., PCIe), or a combination thereof. In at least one embodiment, one or more components of the computer system 900 are interconnected using Compute Express Link (CXL) interconnects.

[0091] The inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. Details regarding the inference and / or training logic 715 are provided below in conjunction with Fig. 7A and / or 7B are provided. In at least one embodiment, the inference and / or training logic 715 can be provided in the system from Fig. 9 for inference or prediction operations at least partially based on weighting parameters used with the neural network training operations, neural network functions and / or architectures or neural network use cases described herein.

[0092] Such components can enable multi-image interpolation for an improved user experience.

[0093] Fig. Figure 10 is a block diagram illustrating an electronic device 1000 for using a processor 1010 according to at least one embodiment. In at least one embodiment, the electronic device 1000 can be, for example, and without limitation, a notebook, a tower server, a rack server, a blade server, a laptop, a desktop computer, a tablet, a mobile device, a telephone, an embedded computer, or any other suitable electronic device.

[0094] In at least one embodiment, the system 1000 can include, without restriction, the processor 1010, which is communicatively coupled to any suitable number or type of components, peripherals, modules, or devices. In at least one embodiment, the processor 1010 is coupled using a bus or interface, such as a 1°C bus, a system management bus (“SMBus”), a low-pin-count (LPC) bus, a serial peripheral interface (“SPI”), a high-definition audio (“HDA”) bus, a serial advance technology attachment (“SATA”) bus, a universal serial bus (“USB”) (version 1, 2, 3), or a universal asynchronous receiver / transmitter (“UART”) bus. In at least one embodiment, the following is illustrated: Fig. 10 a system comprising interconnected hardware devices or “chips”, whereas in other embodiments Fig. 10. An exemplary system on a chip (“SoC”) can be illustrated. In at least one embodiment, the Fig. The devices illustrated in 10 are interconnected using proprietary interconnects, standardized interconnects (e.g., PCIe), or a certain combination thereof. In at least one embodiment, one or more components are made of Fig. 10 interconnected using Compute Express Link (CXL) interconnections.

[0095] In at least one embodiment, Fig. 10 a display 1024, a touchscreen 1025, a touchpad 1030, a near field communications unit (NFC) 1045, a sensor hub 1040, a thermal sensor 1046, an Express chipset (EC) 1035, a trusted platform module (TPM) 1038, a BIOS / firmware / flash memory (BIOS, FW flash) 1022, a DSP 1060, a drive 1020, such as a solid state drive (SSD) or a hard disk drive (HDD), a wireless local area network (WLAN) 1050, a Bluetooth unit 1052, a wireless wide area network (WWAN) 1056, a global Positioning system (global positioning system - GPS) 1055, a camera (“USB 3.0 camera”) 1054, such as a USB 3.The device may include a 0-camera and / or a low-power double-data-rate ("LPDDR") storage unit ("LPDDR3") 1015, implemented, for example, in an LPDDR3 standard. These components may each be implemented in any suitable manner.

[0096] In at least one embodiment, other components can be communicatively coupled to the processor 1010 via previously discussed components. In at least one embodiment, an accelerometer 1041, an ambient light sensor (“ALS”) 1042, a compass 1043, and a gyroscope 1044 can be communicatively coupled to the sensor hub 1040. In at least one embodiment, the thermal sensor 1039, a fan 1037, a keyboard 1036, and a touchpad 1030 can be communicatively coupled to the EC 1035. In at least one embodiment, one or more loudspeakers 1063, headphones 1064, and a microphone (“Mic”) 1065 can be communicatively coupled to an audio unit (“Class D audio codec and amplifier”) 1062, which in turn can be communicatively coupled to the DSP 1060. In at least one embodiment, the audio unit 1062 can, for example and without limitation, include an audio encoder / decoder (“codec”) and a class-D amplifier.In at least one embodiment, a SIM card (“SIM”) 1057 can be communicatively coupled with the WWAN unit 1056. In at least one embodiment, components such as the WLAN unit 1050 and the Bluetooth unit 1052, as well as the WWAN unit 1056, can be implemented in a next-generation form factor (“NGFF”).

[0097] The inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. Details regarding the inference and / or training logic 715 are provided below in conjunction with Fig. 7A and / or 7B are provided. In at least one embodiment, the inference and / or training logic 715 can be provided in the system from Fig. 10 for inference or prediction operations, at least partly based on weighting parameters calculated using neural network training operations, neural network functions and / or neural network architectures or neural network use cases described in this document.

[0098] Such components can enable multi-image interpolation for an improved user experience.

[0099] Fig. Figure 11 is a block diagram of a processing system according to at least one embodiment. In at least one embodiment, the system 1100 includes one or more processors 1102 and one or more graphics processors 1108 and can be a single-processor desktop system, a multi-processor workstation system, or a server system comprising a large number of processors 1102 or processor cores 1107. In at least one embodiment, the processing system 1100 is a processing platform integrated into an integrated circuit as a system-on-a-chip (SoC) for use in mobile, portable, or embedded devices.

[0100] In at least one embodiment, the System 1100 can comprise or be integrated into a server-based gaming platform, a game console comprising a game and media console, a mobile gaming console, a handheld game console, or an online game console. In at least one embodiment, the System 1100 is a mobile phone, a smartphone, a tablet computing device, or a mobile internet device. In at least one embodiment, the Processing System 1100 can also include, be coupled to, or be integrated into a wearable device, such as a smartwatch wearable device, a smart eyewear device, an augmented reality device, or a virtual reality device.In at least one embodiment, the processing system 1100 is a television or set-top box device comprising one or more processor(s) 1102 and a graphical user interface generated by one or more graphics processor(s) 1108.

[0101] In at least one embodiment, one or more processor(s) 1102 each include one or more processor core(s) 1107 for processing instructions that, when executed, perform operations for system and user software. In at least one embodiment, each of the processor core(s) 1107 is configured to process a specific instruction set 1109. In at least one embodiment, the instruction set 1109 can enable Complex Instruction Set Computing (CISC), Reduced Instruction Set Computing (RISC), or computation using a Very Long Instruction Word (VLIW). In at least one embodiment, the processor core(s) 1107 can each process a different instruction set 1109, which may contain instructions to facilitate the emulation of other instruction sets. In at least one embodiment, the processor core(s) 1107 can...The processor core(s) 1107 also include other processing devices, such as a digital signal processor (DSP).

[0102] In at least one embodiment, the processor(s) 1102 include a cache memory 1104. In at least one embodiment, the processor(s) 1102 may have a single internal cache or multiple levels of internal cache. In at least one embodiment, the cache memory is shared by different components of the processor(s) 1102. In at least one embodiment, the processor(s) 1102 also use an external cache (e.g., a Level 3 ("L3") cache or Last-Level Cache ("LLC")) (not shown), which can be shared by the processor core(s) 1107 using known cache coherence techniques. In at least one embodiment, the processor(s) 1102 additionally includes a register file 1106, which may contain different types of registers for storing different types of data (e.g.,(Integer register, floating-point register, status register, and an instruction pointer register). In at least one embodiment, the register file 1106 may contain universal registers or other registers.

[0103] In at least one embodiment, one or more processors 1102 are coupled to one or more interface buses 1110 to transmit communication signals, such as address, data, or control signals, between the processor 1102 and other components in the system 1100. In at least one embodiment, the interface bus(s) 1110 can be a processor bus, such as a version of a Direct Media Interface (DMI) bus. In at least one embodiment, the interface bus(s) 1110 is not limited to a DMI bus and can include one or more peripheral component interconnection buses (e.g., PCI, PCI Express), memory buses, or other types of interface buses. In at least one embodiment, the processor(s) 1102 include an integrated memory controller 1116 and a platform control hub 1130.In at least one embodiment, the storage controller 1116 enables communication between a storage device and other components of the system 1100, while the platform controller hub (“PCH”) 1130 provides connections to input / output (“I / O”) devices via a local I / O bus.

[0104] In at least one embodiment, the storage device 1120 can be a dynamic random-access memory (DRAM) device, a static random-access memory (SRAM) device, a flash memory device, a phase-change memory device, or some other storage device that has suitable performance to serve as process memory. In at least one embodiment, the storage device 1120 can operate as system memory for the system 1100 to store data 1122 and instructions 1121 that are used when one or more processor(s) 1102 execute an application or procedure. In at least one embodiment, the memory controller 1116 is also coupled with an optional external graphics processor 1112 that can communicate with one or more graphics processor(s) 1108 in the processors 1102 to perform graphics and media operations.In at least one embodiment, a display device 1111 can be connected to the processor(s) 1102. In at least one embodiment, the display device 1111 can include one or more internal display devices, such as in a mobile electronic device or a laptop device, or external display devices connected via a display interface (e.g., DisplayPort, etc.). In at least one embodiment, the display device 1111 can include a head-mounted display (HMD), such as a stereoscopic display device for use in virtual reality (VR) or augmented reality (AR) applications.

[0105] In at least one embodiment, the platform control hub 1130 enables peripheral devices to connect to the storage device 1120 and the processor 1102 via a high-speed I / O bus. In at least one embodiment, the I / O peripheral devices include, without limitation, an audio controller 1146, a network controller 1134, a firmware interface 1128, a wireless transceiver 1126, touch sensors 1125, and a data storage device 1124 (e.g., a hard disk drive, flash memory, etc.). In at least one embodiment, the data storage device 1124 can be connected via a storage interface (e.g., SATA) or via a peripheral bus, such as a peripheral component interconnection bus (e.g., PCI, PCI Express). In at least one embodiment, the touch sensors 1125 can include touchscreen sensors, pressure sensors, or fingerprint sensors.In at least one embodiment, the wireless transceiver 1126 can be a WiFi transceiver, a Bluetooth transceiver, or a mobile network transceiver, such as a 3G, 4G, or Long-Term Evolution (LTE) transceiver. In at least one embodiment, the firmware interface 1128 enables communication with the system firmware and can, for example, be a Unified Expandable Firmware Interface (UEFI). In at least one embodiment, the network controller 1134 can enable a network connection to a wired network. In at least one embodiment, a high-performance network controller (not shown) is coupled to the interface bus(s) 1110. In at least one embodiment, the audio controller 1146 is a high-resolution, multi-channel audio controller.In at least one embodiment, the system 1100 includes an optional legacy I / O controller 1140 for coupling conventional devices (e.g., Personal System 2 (PS / 2)) to the system. In at least one embodiment, the platform control hub 1130 can also be connected to one or more universal serial bus (“USB”) controller(s) 1142, connection input devices such as keyboard and mouse 1143 combinations, a camera 1144, or other USB input devices.

[0106] In at least one embodiment, an instance of the memory controller 1116 and the platform control hub 1130 can be integrated into a discrete external graphics processor, such as an external graphics processor 1112. In at least one embodiment, the platform control hub 1130 and / or the memory controller 1116 can be external to the one or more processors 1102. For example, in at least one embodiment, the system 1100 can include an external memory controller 1116 and a platform control hub 1130, which can be configured as a memory control hub and peripheral control hub within a system chipset that communicates with the processor(s) 1102.

[0107] The inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. Details regarding the inference and / or training logic 715 are provided below in conjunction with Fig. 7A and / or 7B are provided. In at least one embodiment, sections of or the entire inference and / or training logic 715 can be integrated into a graphics processing unit 1500. For example, in at least one embodiment, the training and / or inference techniques described herein can utilize one or more of the ALUs implemented in a graphics processing unit. Furthermore, in at least one embodiment, the inference and / or training operations described herein can be performed using logic other than that provided in Fig. The logic illustrated in Figures 7A and / or 7B can be performed. In at least one embodiment, weighting parameters can be stored in a memory and / or registers on or off a chip (shown or not shown), which configure the ALU of a graphics processor to execute one or more machine learning algorithms, neural network architectures, use cases, or training techniques described in this document.

[0108] Such components can enable multi-image interpolation for an improved user experience.

[0109] Fig. Figure 12 is a block diagram of a processor 1200 comprising one or more processor core(s) 1202A-1202N, an integrated memory controller 1214, and an integrated graphics processor 1208 according to at least one embodiment. In at least one embodiment, the processor 1200 may include additional cores, up to and including the additional core(s) 1202N, which are represented by dashed boxes. In at least one embodiment, each of the processor cores 1202A-1202N includes one or more internal cache units 1204A-1204N. In at least one embodiment, each processor core also has access to one or more shared cached units 1206.

[0110] In at least one embodiment, the internal cache unit(s) 1204A-1204N and the shared cache unit(s) 1206 constitute a cache memory hierarchy within the processor 1200. In at least one embodiment, the cache memory unit(s) 1204A-1204N can include at least one level of instruction and data cache within each processor core and one or more levels of shared mid-level cache, such as a Level 2 (L2), Level 3 (L3), Level 4 (L4), or other cache levels, wherein a highest cache level in front of the external memory is classified as LLC. In at least one embodiment, cache coherence logic maintains coherence between the various cache units 1206 and 1204A-1204N.

[0111] In at least one embodiment, the processor 1200 can also include a set of one or more bus control units 1216 and a system agent core 1210. In at least one embodiment, one or more bus control units 1216 manage a set of peripheral buses, such as one or more PCI or PCI Express buses. In at least one embodiment, the system agent core 1210 provides a management function for various processor components. In at least one embodiment, the system agent core 1210 includes one or more integrated memory controllers 1214 to manage access to various external storage devices (not shown).

[0112] In at least one embodiment, one or more of the processor cores 1202A-1202N include support for simultaneous multithreading. In at least one embodiment, the system agent core 1210 includes components for coordinating the processor core(s) 1202A-1202N during multithreaded processing. In at least one embodiment, the system agent core 1210 may additionally include a power control unit (PCU) containing logic and components for regulating one or more power states of the processor core(s) 1202A-1202N and the graphics processor 1208.

[0113] In at least one embodiment, the processor 1200 additionally includes a graphics processor 1208 for performing graphics processing operations. In at least one embodiment, the graphics processor 1208 is coupled with a shared cache unit(s) 1206 and the system agent core 1210, which includes one or more integrated memory controllers 1214. In at least one embodiment, the system agent core 1210 also includes a display controller 1211 for driving the graphics processor output to one or more coupled displays. In at least one embodiment, the display controller 1211 can also be a separate module coupled to the graphics processor 1208 via at least one interconnection, or it can be integrated within the graphics processor 1208.

[0114] In at least one embodiment, a ring-based interconnection unit 1212 is used to couple internal components of the processor 1200. In at least one embodiment, an alternative interconnection unit can be used, such as a point-to-point connection, a switched connection, or other methods. In at least one embodiment, the graphics processor 1208 is coupled to the ring interconnection unit 1212 via an I / O connection 1213.

[0115] In at least one embodiment, the I / O connection 1213 represents at least one of several versions of I / O interconnects, including an in-package I / O interconnect that enables communication between different processor components and an embedded high-performance memory module 1218, such as an eDRAM module. In at least one embodiment, each of the processor cores 1202A-1202N and the graphics processor 1208 uses embedded memory modules 1218 as a shared last-level cache.

[0116] In at least one embodiment, the processor core(s) 1202A-1202N are homogeneous cores executing a common instruction set architecture. In at least one embodiment, the processor core(s) 1202A-1202N are heterogeneous in terms of the instruction set architecture (ISA), wherein one or more processor core(s) 1202A-1202N execute a common instruction set, while one or more other cores of the processor core(s) 1202A-1202N execute a subset of a common instruction set or a different instruction set. In at least one embodiment, the processor core(s) 1202A-1202N are heterogeneous in terms of the microarchitecture, wherein one or more cores with relatively higher power consumption are coupled with one or more performance cores with lower power consumption.In at least one embodiment, the 1200 processor can be implemented on one or more chips or as an integrated SoC circuit.

[0117] The inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. Details regarding the inference and / or training logic 715 are provided below in conjunction with Fig. 7A and / or 7B are provided. In at least one embodiment, sections of or all of the inference and / or training logic 715 can be integrated into the processor 1200. For example, in at least one embodiment, the training and / or inference techniques described in this document can use one or more of the ALUs located in a graphics processor 1208, one or more graphics core(s) 1202A-1202N, or other components in Fig. 12 are realized. Furthermore, in at least one embodiment, the inference and / or training operations described herein can be performed using logic other than that described in Fig. The logic illustrated in Figures 7A and / or 7B can be performed. In at least one embodiment, weighting parameters can be stored in a memory and / or registers on or off a chip (shown or not shown), which configure the ALU of a 1200 graphics processor to execute one or more machine learning algorithms, neural network architectures, use cases, or training techniques described in this document.

[0118] Such components can enable multi-image interpolation for an improved user experience. VIRTUALIZED COMPUTING PLATFORM

[0119] Fig. Figure 13 is an exemplary data flow diagram for a method 1300 for generating and deploying an image processing and inference pipeline according to at least one embodiment. In at least one embodiment, the method 1300 can be deployed for use with imaging devices, processing devices, and / or other types of devices in one or more facilities 1302. The method 1300 can be executed in a training system 1304 and / or in a deployment system 1306. In at least one embodiment, the training system 1304 can be used to perform the training, deployment, and implementation of machine learning models (e.g., neural networks, object recognition algorithms, computer vision algorithms, etc.) for use in a deployment system 1306.In at least one embodiment, the deployment system 1306 can be configured to offload processing and computing resources to a distributed computing environment in order to reduce the infrastructure requirements in the facility 1302. In at least one embodiment, one or more applications in a pipeline can use or access services (e.g., inference, visualization, computing, AI, etc.) of the deployment system 1306 during application execution.

[0120] In at least one embodiment, some of the applications used in advanced processing and inference pipelines may employ machine learning models or other AI to perform one or more processing steps. In at least one embodiment, the machine learning models in the facility(ies) 1302 may be trained using data 1308 (such as imaging data) generated in the facility(ies) 1302 (and stored in one or more picture archiving and communication system (PACS) servers in the facility(ies) 1302), or may be trained using imaging or sequencing data 1308 from one or more other facilities or a combination thereof.In at least one embodiment, the training system 1304 can be used to provide applications, services and / or other resources for generating working, deployable machine learning models for the deployment system 1306.

[0121] In at least one embodiment, the model register 1324 can be secured by an object store that can support versioning and object metadata. In at least one embodiment, the object store can be accessed, for example, via a cloud-storage-compatible application programming interface (API) within a cloud platform. In at least one embodiment, the machine learning models within the model register 1324 can be uploaded, listed, edited, or deleted by developers or partners of a system that interacts with an API. In at least one embodiment, an API can provide access to procedures that allow users with appropriate credentials to associate models with applications such that models can be executed as part of the execution of containerized application instantiations.

[0122] In at least one embodiment, the training system 1304 ( Fig. 13) include a scenario in which the facilities 1302 train their own machine learning model or have an existing machine learning model that needs to be optimized or updated. In at least one embodiment, imaging data 1308 generated by one or more imaging devices, sequencing devices, and / or other types of devices can be received. Once the imaging data 1308 has been received, in at least one embodiment, an AI-assisted annotation 1310 can be used to assist in generating annotations corresponding to the imaging data 1308 to be used as ground-truth data for a machine learning model. In at least one embodiment, an AI-assisted annotation 1310 can be used to train one or more machine learning models (e.g.,Convolutional neural networks (CNNs) that can be trained to generate annotations corresponding to certain types of imaging data 1308 (e.g., from certain devices) are included. In at least one embodiment, the AI-assisted annotation 1310 can then be used directly or can be adapted or tuned using an annotation tool to generate ground-truth data. In at least one embodiment, the AI-assisted annotation 1310, the labeled data 1312, or a combination thereof can be used as ground-truth data for training a machine learning model. In at least one embodiment, a trained machine learning model can be designated as the output model(s) 1316 and can be used by the deployment system 1306 as described in this document.

[0123] In at least one embodiment, a training pipeline may include a scenario in which the facilities 1302 require a machine learning model for use in performing one or more processing tasks for one or more applications in the deployment system 1306, but the facility(ies) 1302 may not currently have such a machine learning model (or may not have a model that is optimized, efficient, or effective for such purposes). In at least one embodiment, an existing machine learning model may be selected from a model register 1324. In at least one embodiment, the model register 1324 may contain machine learning models that have been trained to perform a variety of different inference tasks on imaging data.In at least one embodiment, the machine learning models in the model register 1324 may have been trained on imaging data from facilities other than the facilities 1302 (e.g., remote facilities). In at least one embodiment, machine learning models may have been trained on image data from one site, two sites, or any number of sites. In at least one embodiment, when trained on imaging data from a specific site, the training may take place at that site or at least in a manner that protects the confidentiality of the imaging data or restricts the transmission of the imaging data out of the facility. Once a model has been trained—or partially trained—at a site, a machine learning model may, in at least one embodiment, be added to the model register 1324.In at least one embodiment, a machine learning model can then be retrained or updated on any number of other facilities, and a retrained or updated model can be made available in the model register 1324. In at least one embodiment, a machine learning model can then be selected from the model register 1324—and designated as output model(s) 1316—and can be used in the deployment system 1306 to perform one or more processing tasks for one or more applications of a deployment system.

[0124] In at least one embodiment, a scenario may include facilities 1302 that require a machine learning model for use in performing one or more processing tasks for one or more applications in the deployment system 1306, but the facilities 1302 may not currently have such a machine learning model (or may not have a model that is optimized, efficient, or effective for such purposes). In at least one embodiment, a machine learning model selected from the model register 1324 may not be matched or optimized for imaging data 1308 generated in the facilities 1302 due to differences in the populations, the robustness of the training data used to train a machine learning model, the diversity of anomalies in the training data, and / or other problems with the training data.In at least one embodiment, an AI-assisted annotation 1310 can be used to help generate annotations corresponding to the imaging data 1308, which are to be used as ground-truth data for retraining or updating a machine learning model. In at least one embodiment, the labeled data 1312 can be used as ground-truth data for training a machine learning model. In at least one embodiment, the retraining or updating of a machine learning model can be referred to as model training 1314. In at least one embodiment, the model training 1314—e.g., AI-assisted annotation 1310, labeled data 1312, or a combination thereof—can be used as ground-truth data for retraining or updating a machine learning model.In at least one embodiment, a trained machine learning model can be designated as output model(s) 1316 and can be used by the deployment system 1306 as described in this document.

[0125] In at least one embodiment, the deployment system 1306 can include software 1318, services 1320, hardware 1322, and / or other components, features, and functionality. In at least one embodiment, the deployment system 1306 can include a software "stack" such that the software 1318 can be built on top of services 1320 and can use the services 1320 to perform some or all of the processing tasks and services 1320, and the software 1318 can be built on top of hardware 1322 and can use the hardware 1322 to perform processing, storage, and / or other computational tasks of the deployment system 1306. In at least one embodiment, the software 1318 can include any number of different containers, each container being capable of executing an instantiation of an application.In at least one embodiment, each application can perform one or more processing tasks in an advanced processing and inference pipeline (e.g., inference, object detection, feature recognition, segmentation, image enhancement, calibration, etc.). In at least one embodiment, an advanced processing and inference pipeline can be defined based on selections of various containers that are desired or required for processing imaging data 1308, in addition to containers that receive and configure imaging data for use by each container and / or for use by the facility 1302 after processing via a pipeline (e.g., to convert outputs back into a usable data type).In at least one embodiment, a combination of containers within the software 1318 (which, for example, forms a pipeline) can be referred to as a virtual instrument (as will be described in more detail in this document), and a virtual instrument can utilize services 1320 and hardware 1322 to perform one or all of the processing tasks of applications instantiated in containers.

[0126] In at least one embodiment, a data processing pipeline can receive input data (e.g., imaging data 1308) in a specific format in response to an inference request (e.g., a request from a user of the deployment system 1306). In at least one embodiment, the input data can be representative of one or more images, video footage, and / or other data representations generated by one or more imaging devices. In at least one embodiment, the data can be preprocessed as part of the data processing pipeline to prepare the data for processing by one or more applications.In at least one embodiment, post-processing can be performed on the output of one or more inference tasks or other processing tasks of a pipeline to prepare output data for a subsequent application and / or to prepare output data for transmission and / or use by a user (e.g., as a response to an inference request). In at least one embodiment, the inference tasks can be performed by one or more machine learning models, such as trained or provisioned neural networks, which may include output model(s) 1316 of the training system 1304.

[0127] In at least one embodiment, the tasks of the data processing pipeline can be encapsulated in (one) container(s), each representing a discrete, fully functional instantiation of an application and a virtualized computing environment capable of referencing machine learning models. In at least one embodiment, the containers or applications can be published to a private area (e.g., with restricted access) of a container register (described in more detail below in this document), and trained or deployed models can be stored in the model register 1324 and assigned to one or more applications. In at least one embodiment, images of applications (e.g.,Container images) are available in a container registry, and once they have been selected by a user from a container registry for use in a pipeline, an image can be used to create a container for instantiating an application for use by a user's system.

[0128] In at least one embodiment, developers (e.g., software developers, clinicians, physicians, etc.) can develop, publish, and store applications (e.g., in the form of containers) for performing image processing and / or inference on provided data. In at least one embodiment, the development, publication, and / or storage can be performed using a software development kit (SDK) associated with a system (e.g., to ensure that a developed application and / or container is compliant or compatible with a system). In at least one embodiment, a developed application can be tested locally (e.g., in a first facility with data from a first facility) using an SDK that includes at least some of the services 1320 as a system (e.g., the system 1200 from Fig. 12) can support. In at least one embodiment, since DICOM objects can contain any number between one and hundreds of images or other data types, and due to variations in the data, a developer may be responsible for managing (e.g., setting constructs for, incorporating preprocessing into an application, etc.), extracting, and preparing incoming data. In at least one embodiment, once validated by System 1300 (e.g., for accuracy), an application may be available in a container register for selection and / or implementation by a user to perform one or more processing tasks with respect to data in a user's facility (e.g., a second facility).

[0129] In at least one embodiment, developers can then make applications or containers accessible and usable by users of a system (e.g., the System 1300 from) via a network. Fig. 13) share. In at least one embodiment, completed and validated applications or containers can be stored in a container register, and associated machine learning models can be stored in the model register 1324. In at least one embodiment, a requesting entity—providing an inference or image processing request—can search a container register and / or a model register 1324 for an application, container, dataset, machine learning model, etc., select a desired combination of elements to include in the data processing pipeline, and submit an image processing request.In at least one embodiment, a request may include input data (and in some examples, associated patient data) required to execute a request, and / or a selection of machine learning applications and / or models to be executed when processing a request. In at least one embodiment, a request may then be passed to one or more components of the deployment system 1306 (e.g., a cloud) to perform processing in the data processing pipeline. In at least one embodiment, the processing by the deployment system 1306 may involve referencing selected elements (e.g., applications, containers, models, etc.) from a container register and / or model register 1324. In at least one embodiment, once results are generated by a pipeline, results may be provided to a user as a reference (e.g.,(for viewing in a viewing application suite running on a local workstation or terminal).

[0130] In at least one embodiment, services 1320 can be used to assist in the processing or execution of applications or containers in pipelines. In at least one embodiment, the services 1320 can include computing services, artificial intelligence (AI) services, visualization services, and / or other types of services. In at least one embodiment, the services 1320 can provide functionality that is common to one or more applications in the software 1318, such that the functionality can be abstracted to a service that can be called or used by the applications. In at least one embodiment, the functionality provided by the services 1320 can be executed dynamically and more efficiently, and is also highly scalable by allowing the applications to process data in parallel (e.g., using a parallel computing platform 1230). Fig. 12)) to process. In at least one embodiment, instead of each application sharing the same functionality offered by Services 1320 having its own instance of Services 1320, Services 1320 can be shared by various applications. In at least one embodiment, the Services can include an inference server or inference engine that can be used to perform detection or segmentation tasks, as non-limiting examples. In at least one embodiment, a model training service can be included that can provide the ability to train and / or retrain machine learning models. In at least one embodiment, a data augmentation service can also be included that enables the extraction, resizing, scaling, and / or other enhancement of GPU-accelerated data (e.g.,DICOM data, RIS data, CIS data, REST-compliant data, RPC data, raw data, etc.) can be provided. In at least one embodiment, a visualization service can be used that can add image rendering effects—such as ray tracing, rasterization, denoising, sharpening, etc.—to make two-dimensional (2D) and / or three-dimensional (3D) models more realistic. In at least one embodiment, services for virtual instruments can be included that provide beam shaping, segmentation, inference, imaging, and / or support for other applications within virtual instrument pipelines.

[0131] In at least one embodiment, where Services 1320 include an AI service (e.g., an inference service), one or more machine learning models can be executed by invoking an inference service (e.g., an inference server) (e.g., as an API call) to execute one or more machine learning models or their processing as part of application execution. In at least one embodiment, where another application includes one or more machine learning models for segmentation tasks, an application can invoke an inference service to execute machine learning models to perform one or more of the operations associated with segmentation tasks.In at least one embodiment, the software 1318, which implements the advanced processing and inference pipeline that includes a segmentation application and anomaly detection application, can be streamlined because each application can call the same inference service to perform one or more inference tasks.

[0132] In at least one embodiment, the hardware 1322 can include a GPU, CPU, graphics cards, an AI / deep learning system (e.g., an AI supercomputer such as NVIDIA's DGX), a cloud platform, or a combination thereof. In at least one embodiment, different types of hardware 1322 can be used to provide efficient, custom-built software 1318 and services 1320 in the deployment system 1306. In at least one embodiment, the use of GPU processing for local processing (e.g., in the facility 1302) within an AI / deep learning system, in a cloud system, and / or in other processing components of the deployment system 1306 can be implemented to improve the efficiency, accuracy, and effectiveness of image processing and generation.In at least one embodiment, the software 1318 and / or services 1320 can be optimized for GPU processing with respect to deep learning, machine learning, and / or high-performance computing, as non-limiting examples. In at least one embodiment, at least part of the computing environment of the deployment system 1306 and / or the training system 1304 can be run in a data center, on one or more supercomputers, or high-performance computing systems with GPU-optimized software (e.g., a hardware and software combination of NVIDIA's DGX system). In at least one embodiment, the hardware 1322 can include any number of GPUs that can be called upon to perform parallel data processing as described in this document.In at least one embodiment, the cloud platform may further include GPU processing for GPU-optimized execution of deep learning tasks, machine learning tasks, or other computational tasks. In at least one embodiment, the cloud platform (e.g., NVIDIA's NGC) may be implemented using one or more AI / deep learning supercomputers and / or GPU-optimized software (e.g., NVIDIA's DGX systems) as a hardware abstraction and scaling platform. In at least one embodiment, the cloud platform may integrate an application container clustering or orchestration system (e.g., Kubernetes) across multiple GPUs to enable seamless scaling and load balancing.

[0133] Fig. Figure 14 is a system diagram for an exemplary system 1400 for generating and providing an imaging deployment pipeline according to at least one embodiment. In at least one embodiment, the system 1400 can be used to implement the method 1300 from Fig. 13 and / or other methods to implement advanced processing and inference pipelines. In at least one embodiment, the system 1400 may include the training system 1304 and the deployment system 1306. In at least one embodiment, the training system 1304 and the deployment system 1306 may be implemented using software 1318, services 1320, and / or hardware 1322, as described in this document.

[0134] In at least one embodiment, the System 1400 (e.g., the Training System 1304 and / or the Deployment System 1306) can be implemented in a cloud computing environment (e.g., using Cloud 1426). In at least one embodiment, the System 1400 can be implemented locally with respect to a healthcare facility or as a combination of both cloud and local computing resources. In at least one embodiment, access to APIs in Cloud 1426 can be restricted to authorized users by mandated security measures or protocols. In at least one embodiment, a security protocol can include web tokens that can be signed by an authentication service (e.g., AuthN, AuthZ, Gluecon, etc.) and carry appropriate authorization.In at least one embodiment, the APIs of virtual instruments (described in this document) or other instantiations of System 1400 can be restricted to a set of public IPs that have been verified or authorized for interaction.

[0135] In at least one embodiment, various components of the System 1400 can communicate with each other using one of several different network types, including, without limitation, local area networks (LANs) and / or wide area networks (WANs), via wired and / or wireless communication protocols. In at least one embodiment, communication between devices and components of the System 1400 (e.g., for transmitting inference requests, receiving results of inference requests, etc.) can be carried out via one or more data buses, wireless data protocols (WiFi), wired data protocols (e.g., Ethernet), etc.

[0136] In at least one embodiment, the training system 1304 can execute training pipelines 1404, similar to those described in this document with respect to Fig. 13 are described. In at least one embodiment, in which one or more machine learning models are to be used in the deployment pipeline(s) 1410 by the deployment system 1306, the training pipelines 1404 can be used to train or retrain one or more (e.g., pre-trained) models, and / or to implement one or more of the pre-trained models 1406 (e.g., without requiring retraining or an update). In at least one embodiment, output model(s) 1316 can be generated as a result of the training pipelines 1404. In at least one embodiment, the training pipelines 1404 can include any number of processing steps, such as, without limitation, the conversion or adaptation of imaging data (or other input data).In at least one embodiment, different training pipelines 1404 can be used for different machine learning models employed by the deployment system 1306. In at least one embodiment, a training pipeline (or pipelines) 1404 similar to a first example, which is described with respect to . Fig. As described in section 13, a training pipeline (or pipelines) 1404, similar to a second example, can be used for a first machine learning model, which is similar to a second example that is similar to Fig. 13, can be used for a second machine learning model, and training pipelines 1404, which are similar to a third example, can be used in relation to Fig. The training system 1304, as described in section 13, can be used for a third machine learning model. In at least one embodiment, any combination of tasks within the training system 1304 can be used, depending on the requirements of each machine learning model. In at least one embodiment, one or more of the machine learning models can already be trained and ready for deployment, so that the machine learning models may not require processing by the training system 1304 and can be implemented by the deployment system 1306.

[0137] In at least one embodiment, output models 1316 and / or pretrained model(s) 1406 may include any type of machine learning model(s) depending on the implementation or embodiment. In at least one embodiment and without limitation, the machine learning models used by the System 1400 may include one or more machine learning models that employ linear regression, logistic regression, decision trees, support vector machines (SVMs), Naive Bayes, k-nearest neighbors (Knn), K-means clustering, random forest, dimensionality reduction algorithms, gradient enhancement algorithms, neural networks (e.g., self-encoder, convolution, recursive, perceptron, long-term / short-term memory (LSTM), Hopfield, Boltzmann, deep belief, unfolding, generative contradictory, fluid state machine, etc.) and / or other types of machine learning models.

[0138] In at least one embodiment, the training pipelines 1404 can include AI-supported annotation, as described in this document at least with regard to Fig. 14 described in more detail. In at least one embodiment, the labeled data 1312 (e.g., conventional annotation) can be generated by any number of techniques. In at least one embodiment, labels or other annotations can be generated in a drawing program (e.g., an annotation program), a CAD (Computer Aided Design) program, a marking program, or another type of program suitable for generating annotations or labels for ground truth, and / or they can be drawn by hand in some examples. In at least one embodiment, ground truth data can be generated synthetically (e.g., generated from computer models or renderings), generated in reality (e.g., designed and generated from real data), or generated automatically by machine (e.g.,using feature analysis and learning to extract features from data and then generate labels), annotated by humans (e.g., labeler or annotation expert, defining the position of labels), and / or a combination thereof. In at least one embodiment, for each instance of imaging data 1308 (or other data type used by machine learning models), there can be corresponding ground-truth data generated by the training system 1304. In at least one embodiment, AI-assisted annotation can be performed as part of the deployment pipelines 1410, either in addition to or instead of AI-assisted annotation included in the training pipelines 1404. In at least one embodiment, the system 1400 can include a multi-layered platform that includes a software layer (e.g.,The software 1318) of diagnostic applications (or other types of applications) that can perform one or more medical imaging and diagnostic functions. In at least one embodiment, the system 1400 can be communicatively coupled to PACS server networks of one or more facilities (e.g., via encrypted links). In at least one embodiment, the system 1400 can be configured to access and reference data from PACS servers in order to perform operations such as training machine learning models, deploying machine learning models, image processing, inference, and / or other operations.

[0139] In at least one embodiment, a software layer can be implemented as a secure, encrypted, and / or authenticated API through which applications or containers can be accessed (e.g., called) by one or more external environments (e.g., facility 1302). In at least one embodiment, the applications can then call or execute one or more services 1320 to perform computational, AI, or visualization tasks assigned to the respective applications, and the software 1318 and / or services 1320 can utilize hardware 1322 to perform processing tasks effectively and efficiently. In at least one embodiment, communications sent to or received from a training system 1304 and a deployment system 1306 can be conducted using a pair of DICOM adapters 1402A, 1402B.

[0140] In at least one embodiment, the deployment system 1306 can execute deployment pipeline(s) 1410. In at least one embodiment, the deployment pipeline(s) 1410 can include any number of applications that can be applied sequentially, non-sequentially, or otherwise to imaging data (and / or other types of data) generated by imaging devices, sequencing devices, genomics devices, etc. – including AI-assisted annotation, as described above. In at least one embodiment, as described in this document, deployment pipeline(s) 1410 for a single device can be designated as a virtual instrument for a device (e.g., a virtual ultrasound instrument, a virtual CT scan instrument, a virtual sequencing instrument, etc.).In at least one embodiment, there can be more than one deployment pipeline 1410 for a single device, depending on the information desired from the data generated by the device. In at least one embodiment where anomaly detection from an MRI machine is desired, there can be a first deployment pipeline 1410, and where image enhancement from the output of an MRI machine is desired, there can be a second deployment pipeline 1410.

[0141] In at least one embodiment, an image generation application may include a processing task that involves the use of a machine learning model. In at least one embodiment, a user may choose to use their own machine learning model or select a machine learning model from model register 1324. In at least one embodiment, a user may implement their own machine learning model or select a machine learning model for inclusion in an application to perform a processing task. In at least one embodiment, applications may be selectable and customizable, and by defining application constructs, the provisioning and implementation of applications for a particular user are presented as a more seamless user experience.In at least one embodiment, by utilizing other features of the system 1400 - such as the services 1320 and the hardware 1322 - the deployment pipeline(s) 1410 can be even more user-friendly, provide simpler integration and deliver more accurate, efficient and timely results.

[0142] In at least one embodiment, the deployment system 1306 may include a user interface (“UI”)1414 (e.g., a graphical user interface, a web interface, etc.) that can be used to select applications for inclusion in one or more deployment pipeline(s) 1410, to arrange applications, to edit or modify applications or parameters or constructs thereof, to use and interact with one or more deployment pipeline(s) 1410 during setup and / or deployment, and / or to otherwise interact with the deployment system 1306.In at least one embodiment, which is not illustrated with respect to the training system 1304, the user interface 1414 (or another user interface) can be used to select models for use with the deployment system 1306, to select models for training or retraining in the training system 1304, and / or to otherwise interact with the training system 1304.

[0143] In at least one embodiment, the pipeline manager 1412 can be used in addition to an application orchestration system 1428 to manage the interaction between applications or containers of the deployment pipeline(s) 1410 and services 1320 and / or hardware 1322. In at least one embodiment, the pipeline manager 1412 can be configured to allow application-to-application, application-to-services 1320, and / or application-or-service-to-hardware 1322 interactions. In at least one embodiment, although illustrated as being included in the software 1318, this is not intended to be restrictive, and in some examples, the pipeline manager 1412 can be included in the services 1320. In at least one embodiment, the application orchestration system 1428 (e.g., Kubernetes, Docker, etc.) can be configured to allow application-to-application, application-to-services 1320, and / or application-to-services 1322 interactions.) include a container coordination system that can group applications into containers as logical units for coordination, management, scaling, and deployment. In at least one embodiment, by assigning applications from the deployment pipeline(s) 1410 (e.g., a reconstruction application, a segmentation application, etc.) to individual containers, each application can be run in a self-contained environment (e.g., at a kernel level) to increase speed and efficiency.

[0144] In at least one embodiment, each application and / or container (or an image thereof) can be developed, modified, and deployed individually (e.g., a first user or developer can develop, modify, and deploy a first application, and a second user or developer can develop, modify, and deploy a second application separately from the first user or developer), making it possible to focus on and attend to the task of a single application and / or one or more containers without being hindered by tasks of one or more other applications or containers. In at least one embodiment, communication and interaction between different containers or applications can be supported by the pipeline manager 1412 and the application orchestration system 1428.In at least one embodiment, as long as an expected input and / or output of each container or application is known to a system (e.g., based on constructs of applications or containers), the application orchestration system 1428 and / or the pipeline manager 1412 can enable communication and resource sharing between each of the applications or containers. In at least one embodiment, since one or more of the applications or containers in the deployment pipeline(s) 1410 can share the same services and resources, the application orchestration system 1428 can coordinate services or resources between various applications or containers, balance their load, and determine their shared use.In at least one embodiment, a scheduler can be used to track the resource requirements of applications or containers, the current or planned use of these resources, and resource availability. In at least one embodiment, a scheduler can thus allocate resources to different applications and distribute them between and among the applications, taking into account the requirements and availability of a system. In some examples, a scheduler (and / or another component of the application orchestration system 1428) can determine the availability and distribution of resources based on constraints imposed on a system (e.g., user constraints), such as quality of service (QoS), urgency of the need for data output (e.g., to determine whether real-time or delayed processing is required), and so on.

[0145] In at least one embodiment, service(s) 1320, which are used and shared by applications or containers in the deployment system 1306, can include computing service(s) 1416, AI service(s) 1418, visualization service(s) 1420, and / or other service types. In at least one embodiment, applications can call (e.g., execute) one or more of the services 1320 to perform processing operations for an application. In at least one embodiment, computing services 1416 can be used by applications to perform superdata processing or other high-performance computing (HPC) tasks. In at least one embodiment, one or more computing services 1416 can be used to perform parallel processing (e.g.,using a parallel computing platform 1430) for processing data across one or more applications and / or performing one or more tasks of a single application substantially simultaneously. In at least one embodiment, the parallel computing platform 1430 (e.g., NVIDIA's CUDA) can enable general-purpose computing on GPUs (GPGPU) (e.g., GPU / graphics cards 1422). In at least one embodiment, a software layer of a parallel computing platform 1430 can provide access to virtual instruction sets and parallel computing elements of GPUs for executing computing kernels. In at least one embodiment, the parallel computing platform 1430 can include memory, and in some embodiments, memory can be shared by multiple containers and / or among different processing tasks within a single container.In at least one embodiment, inter-process communication (IPC) calls can be generated for multiple containers and / or for multiple processes or procedures within a container to use the same data from a shared memory segment of the Parallel Computing Platform 1430 (e.g., where several different stages of an application or multiple applications process the same information). In at least one embodiment, instead of creating a copy of data and moving data to different memory locations (e.g., a read / write operation), the same data can be used in the same memory location for any number of processing tasks (e.g., concurrently, at different times, etc.).In at least one embodiment, since data is used to generate new data as a result of processing, this information can be stored in a new data location and shared by different applications. In at least one embodiment, the location of data and the location of updated or modified data can be part of a definition of how a payload is understood within containers.

[0146] In at least one embodiment, AI services 1418 can be used to perform inference services for executing one or more machine learning models assigned to applications (e.g., tasked with executing one or more processing tasks of an application). In at least one embodiment, the AI ​​service(s) 1418 can support an AI system 1424 to execute one or more machine learning models (e.g., neural networks, such as CNNs) for segmentation, reconstruction, object detection, feature recognition, classification, and / or other inference tasks. In at least one embodiment, the applications of the deployment pipeline(s) 1410 can use one or more output models 1316 from the training system 1304 and / or other application models to perform inference on imaging data.In at least one embodiment, two or more examples of inference using the Application Orchestration System 1428 (e.g., a scheduler) may be available. In at least one embodiment, a first category may include a high-priority / low-latency path that can achieve higher Service Level Agreements, such as for performing inferences on urgent requests during an emergency or for a radiologist during diagnosis. In at least one embodiment, a second category may include a standard-priority path that can be used for requests that may not be urgent or where analysis can be performed at a later time. In at least one embodiment, the Application Orchestration System 1428 may allocate resources (e.g.,Distribute services 1320 and / or hardware 1322) based on priority paths for different inference tasks of AI service(s) 1418.

[0147] In at least one embodiment, a shared memory for AI service(s) 1418 can be installed in the system 1400. In at least one embodiment, the shared memory can function as a cache (or other type of device) and be used to process requests from applications. In at least one embodiment, when an inference request is made, a request can be received by a set of API instances of the deployment system 1306, and one or more instances can be selected (e.g., for best fit, load balancing, etc.) to process a request.In at least one embodiment, to process a request, a request can be entered into a database; a machine learning model can be located from the model register 1324 if it is not already in the cache; a validation step can ensure that the appropriate machine learning model is loaded into a cache (e.g., shared memory); and / or a copy of a model can be stored in a cache. In at least one embodiment, a scheduler (e.g., of the pipeline manager 1412) can be used to start an application referenced in a request if an application is not already running or if there are not enough instances of an application. In at least one embodiment, an inference server can be started if it has not already been started to execute a model.Any number of inference servers can be started per model. In at least one embodiment, models can be cached in a pull model where inference servers form a cluster, if load balancing is advantageous. In at least one embodiment, the inference servers can be statically loaded onto corresponding distributed servers.

[0148] In at least one embodiment, inference can be performed using an inference server running in a container. In at least one embodiment, an instance of an inference server can be associated with a model (and optionally a plurality of versions of a model). In at least one embodiment, if an instance of an inference server does not exist when a request to perform inference on a model is received, a new instance can be loaded. In at least one embodiment, when an inference server is started, a model can be passed to an inference server, so that the same container can be used to handle different models, as long as the inference server is running as a separate instance.

[0149] In at least one embodiment, an inference request for a given application can be received during application execution, and a container (which, for example, hosts an instance of an inference server) can be loaded (if not already done) and a start procedure invoked. In at least one embodiment, the preprocessing logic in a container can load incoming data, decode it, and / or perform any additional preprocessing on it (e.g., using CPU(s) and / or GPU(s)). In at least one embodiment, once the data is prepared for inference, a container can perform the inference on the data as needed. In at least one embodiment, this can involve a single inference call on a single image (e.g., a hand X-ray) or require inference on hundreds of images (e.g., a breast CT scan).In at least one embodiment, an application can summarize the results before completion, which can include, without limitation, a single confidence score, pixel-level segmentation, voxel-level segmentation, the generation of a visualization, or the generation of text summarizing findings. In at least one embodiment, different models or applications can be assigned different priorities. For example, some models can have a real-time priority (TAT < 1 min), while others can have a lower priority (e.g., TAT < 10 min). In at least one embodiment, model execution times can be measured by the requesting institution or organization and can include partner network traversal time as well as execution by an inference service.

[0150] In at least one embodiment, the transmission of requests between Services 1320 and inference applications can be hidden behind a software development kit (SDK), and robust transport can be provided via a queue. In at least one embodiment, a request via an API is placed in a queue for a unique application / tenant ID combination, and an SDK pulls a request from the queue and sends it to an application. In at least one embodiment, a queue name can be provided in an environment from which an SDK retrieves it. In at least one embodiment, asynchronous communication via a queue can be beneficial because it allows each instance of an application to begin work as soon as it becomes available. Results can be pushed back via a queue to ensure that no data is lost.In at least one embodiment, queues can also provide the ability to segment work, since the highest-priority work can go to a queue with the most instances of an associated application, while the lowest-priority work can go to a queue with a single associated instance that processes tasks in a received order. In at least one embodiment, an application can run on a GPU-accelerated instance generated in Cloud 1426, and an inference service can perform the inference on a GPU.

[0151] In at least one embodiment, a visualization service (or services) 1420 can be used to generate visualizations for visualizing application output and / or a deployment pipeline (or pipelines) 1410. In at least one embodiment, the GPUs / graphics cards 1422 can be used by a visualization service (or services) 1420 to generate visualizations. In at least one embodiment, rendering effects, such as ray tracing, can be implemented by a visualization service (or services) 1420 to generate higher-quality visualizations. In at least one embodiment, visualizations can include, without limitation, 2D image rendering, 3D volume rendering, 3D volume reconstruction, 2D tomographic sections, virtual reality displays, augmented reality displays, etc. In at least one embodiment, virtualized environments can be used to provide a virtually interactive display or environment (e.g.,to generate a virtual environment for interaction by users of a system (e.g., doctors, nurses, radiologists, etc.). In at least one embodiment, visualization service(s) 1420 may include an internal visualizer, film technology, and / or other rendering or image processing capabilities or functionality (e.g., ray tracing, rasterization, internal optics, etc.).

[0152] In at least one embodiment, the hardware 1322 may include GPUs / graphics cards 1422, an AI system 1424, a cloud 1426, and / or any other hardware used to run a training system 1304 and / or a deployment system 1306. In at least one embodiment, the GPUs / graphics cards 1422 (e.g., NVIDIA's TESLA and / or QUADRO GPUs) may include any number of GPUs that can be used to perform processing tasks of compute service(s) 1416, AI service(s) 1418, visualization service(s) 1420, other services, and / or any of the features or functionality of the software 1318.For example, with respect to AI service(s) 1418, the GPUs / graphics cards 1422 can be used to perform preprocessing on imaging data (or other types of data used by machine learning models), postprocessing on outputs from machine learning models, and / or to perform inference (e.g., to run machine learning models). In at least one embodiment, the cloud 1426, the AI ​​system 1424, and / or other components of the system 1400 can use the GPUs / graphics cards 1422. In at least one embodiment, the cloud 1426 can include a GPU-optimized platform for deep learning tasks. In at least one embodiment, the AI ​​system 1424 can use GPUs, and the cloud 1426—or at least a part of it that is tasked with deep learning or inference—can run using one or more AI systems 1424.Although the Hardware 1322 is depicted as discrete components, this is not intended as a limitation, and any components of Hardware 1322 can be combined or used with any other components of Hardware 1322.

[0153] In at least one embodiment, the AI ​​system 1424 may include a specially manufactured computer system (e.g., a supercomputer or an HPC) configured for inference, deep learning, machine learning, and / or other artificial intelligence tasks. In at least one embodiment, the AI ​​system 1424 (e.g., NVIDIA's DGX) may include GPU-optimized software (e.g., a software stack) that can be run using a variety of GPUs / graphics cards 1422 in addition to the CPU, RAM, memory, and / or other components, features, or functionality. In at least one embodiment, one or more AI systems 1424 may be deployed in a cloud 1426 (e.g., in a data center) to perform one or all of the AI-based processing tasks of the system 1400.

[0154] In at least one embodiment, Cloud 1426 can include a GPU-accelerated infrastructure (e.g., NVIDIA NGC) that can provide a GPU-optimized platform for performing processing tasks of System 1400. In at least one embodiment, Cloud 1426 can include an AI system 1424 for performing one or more of the AI-based tasks of System 1400 (e.g., as a hardware abstraction and scaling platform). In at least one embodiment, Cloud 1426 can be integrated with the application orchestration system 1428 and utilize multiple GPUs to enable seamless scaling and load balancing among the applications and services 1320. In at least one embodiment, Cloud 1426 can be tasked with performing at least some of the services 1320 of System 1400, including the compute service(s) 1416, the AI ​​service(s) 1418, and / or the...The visualization service(s) 1420 are included, as described in this document. In at least one embodiment, the cloud 1426 can perform small and large batch inference (running, for example, NVIDIA's TENSOR RT), provide an accelerated parallel computing API and platform 1430 (e.g., NVIDIA's CUDA), run the application orchestration system 1428 (e.g., Kubernetes), provide a graphics rendering API and platform (e.g., for ray tracing, 2D graphics, 3D graphics, and / or other rendering techniques to produce higher-quality movies), and / or provide other functionality for the system 1400.

[0155] Fig. Figure 15A illustrates a data flow diagram for a method 1500 for training, retraining, or updating a machine learning model according to at least one embodiment. In at least one embodiment, the method 1500 can be described as a non-limiting example using the system 1400 from Fig. 14. In at least one embodiment, the method 1500 can utilize the services and / or hardware as described in this document. In at least one embodiment, refined models 1512 generated by the method 1500 can be executed by a deployment system for one or more containerized applications in the deployment pipelines.

[0156] In at least one embodiment, the model training 1514 can involve retraining or updating an initial model 1504 (e.g., a pre-trained model) using new training data (e.g., new input data, such as a customer record 1506, and / or new ground-truth data associated with the input data). In at least one embodiment, retraining or updating the initial model 1504 can involve resetting, deleting, and / or replacing output or loss layer(s) of the initial model 1504 with updated or new output or loss layer(s). In at least one embodiment, the initial model 1504 can have previously fine-tuned parameters (e.g.,Weightings and / or systematic errors) left over from previous training, so that training or retraining 1514 does not take as long or require as much processing power as training a model from scratch. In at least one embodiment, during model training, by resetting or replacing the output or loss layer(s) of the initial model 1504, the parameters for a new dataset can be updated and recalibrated based on loss calculations associated with the accuracy of the output or loss layer(s) when generating predictions on a new customer dataset 1506.

[0157] In at least one embodiment, pre-trained models 1506 may be stored in a data archive or register. In at least one embodiment, the pre-trained models 1506 may have been trained, at least partially, in one or more facilities other than the facility performing the method 1500. In at least one embodiment, to protect the privacy and rights of patients, test subjects, or customers from different facilities, the pre-trained models 1506 may have been trained on-site using customer or patient data generated on-site.In at least one embodiment, the pre-trained models 1506 can be trained using a cloud and / or other hardware, but confidential, privacy-protected patient data must not be transmitted to, used by, or accessible to any component of a cloud (or other hardware outside the user's own premises). In at least one embodiment, in which the pre-trained models 1506 are trained using patient data from more than one institution, the pre-trained models 1506 may have been trained individually for each institution before training on patient or customer data from another institution. In at least one embodiment, such as when customer or patient data is used for privacy reasons (e.g., by waiver, for experimental purposes, etc.),) have been released, or if customer or patient data is included in a public dataset, customer or patient data can be used by any number of facilities to train the pre-trained 1506 models on-site and / or off-site, such as in a data center or other cloud computing infrastructure.

[0158] In at least one embodiment, when selecting applications for use in the deployment pipelines, a user can also select machine learning models to be used for specific applications. In at least one embodiment, a user may not have a model available for use, allowing the user to select the pre-trained model for use with an application. In at least one embodiment, the pre-trained model may not be optimized to generate accurate results on a user's facility's customer dataset 1506 (e.g., based on patient diversity, demographic data, types of medical imaging devices used, etc.).In at least one embodiment, the pre-trained model can be updated, retrained and / or fine-tuned before being deployed in a deployment pipeline for use with one or more applications in a given facility.

[0159] In at least one embodiment, a user can select the pre-trained model(s) to be updated, retrained, and / or fine-tuned, and this pre-trained model can be referred to as the initial model 1504 for a training system within the method 1500. In at least one embodiment, a customer dataset 1506 (e.g., imaging data, genomic data, sequencing data, or other types of data generated by devices in a facility) can be used to perform model training (which may, without limitation, include transfer learning) on ​​the initial model 1504 to generate a refined model 1512. In at least one embodiment, ground-truth data corresponding to the customer dataset 1506 can be generated by the training system 1304.In at least one embodiment, ground truth data can be generated at least partially by clinicians, scientists, physicians, practitioners, in an institution.

[0160] In at least one embodiment, AI-assisted annotation can be used in some examples to generate ground-truth data. In at least one embodiment, the AI-assisted annotation 1310 (e.g., implemented using an AI-assisted annotation SDK) can utilize machine learning models (e.g., neural networks) to generate suggested or predicted ground-truth data for a customer dataset. In at least one embodiment, the user can use annotation tools within a user interface (a graphical user interface (GUI)) on the computing device.

[0161] In at least one embodiment, the user 1510 can interact with a GUI via the computing device 1508 to edit or fine-tune (automatic) annotations. In at least one embodiment, a polygon editing feature can be used to move the vertices of a polygon to more accurate or fine-tuned positions.

[0162] In at least one embodiment, the ground truth data (e.g., from AI-assisted annotation, manual labeling, etc.) can be used during model training to generate the refined model 1512 once the customer dataset 1506 has associated ground truth data. In at least one embodiment, the customer dataset 1506 can be applied to an initial model 1504 any number of times, and the ground truth data can be used to update the parameters of the initial model 1504 until an acceptable level of accuracy is achieved for the refined model 1512. In at least one embodiment, once the refined model 1512 has been generated, it can be deployed within one or more deployment pipelines in a facility to perform one or more processing tasks related to medical imaging data.

[0163] In at least one embodiment, the refined model 1512 can be uploaded to pretrained models in a model register for selection by another facility. In at least one embodiment, this method can be extended with any number of facilities, allowing the refined model 1512 to be further refined on new datasets any number of times to generate a more universal model.

[0164] Fig. Figure 15B is an exemplary illustration of a client-server architecture 1532 for enhancing annotation tools with pre-trained annotation models, according to at least one embodiment. In at least one embodiment, an AI-assisted annotation tool 1536 can be instantiated based on a client-server architecture 1532. In at least one embodiment, the annotation tool 1536 can, for example, assist radiologists in imaging applications by helping them identify organs and anomalies. In at least one embodiment, the imaging applications can include software tools that help the user 1510, as a non-restrictive example, to identify a few extreme points on a specific organ of interest in raw images 1534 (e.g., in a 3D MRI or CT scan) and to obtain automatically labeled results for all 2D slices of a specific organ.In at least one embodiment, the results can be stored in a data archive as training data 1538 and (by way of example and without limitation) used as ground-truth data for training. In at least one embodiment, if the computing device 1508 sends extreme points for AI-assisted annotation, a deep learning model can, for example, receive this data as input and return inference results of a segmented organ or anomaly. In at least one embodiment, pre-instantiated annotation tools, such as the AI-assisted annotation tool 1536, can be used. Fig.15B, can be improved by making API calls (e.g., an API call 1544) to a server, such as an annotation assistant server 1540, which may contain a set of pre-trained models 1542 stored, for example, in an annotation model register. In at least one embodiment, an annotation model register can store pre-trained models 1542 (e.g., machine learning models, such as deep learning models) that are pre-trained to perform AI-assisted annotation on a specific organ or anomaly. These models can be further updated using training pipelines. In at least one embodiment, pre-installed annotation tools can be improved over time as new labeled data is added.

[0165] Such components can enable multi-image interpolation for an improved user experience.

[0166] Other variations are within the spirit of the present disclosure. Thus, while various modifications and alternative constructions can be made with respect to the disclosed methods, certain illustrated embodiments are shown in the drawings and have been described in detail above. However, it is understood that the intention is not to limit the disclosure to the specific disclosed form or forms, but rather, on the contrary, to cover all modifications, alternative constructions, and equivalents that fall within the spirit and scope of the disclosure as defined in the attached claims.

[0167] The use of the terms "a," "an," "the," and similar referents in the context of describing disclosed embodiments (particularly in the context of the following claims) is to be interpreted as covering both the singular and the plural unless otherwise specified herein or the context clearly contradicts this, and not as defining an expression. The terms "comprising," "having," "including," and "containing" are to be interpreted as open expressions (i.e., in the sense of "including without being limited to") unless otherwise specified. "Connected" is to be interpreted as partially or completely contained within, attached to, or joined to one another when it is used unmodified and refers to physical connections, even if an element is inserted between them.The mention of value ranges herein is intended merely as a quick method of individually referring to each separate value falling within the range, unless otherwise stated herein, and each separate value is included in the description as if it were individually reproduced herein. The use of the term "set" (e.g., "a set of objects") or "subset" in at least one embodiment is to be understood as a non-empty compilation comprising one or more elements, unless otherwise noted or the context contradicts this. Furthermore, unless otherwise stated or the context contradicts this, the term "subset" of a corresponding set does not necessarily mean a proper subset of the corresponding set, but the subset and the corresponding set may be the same.

[0168] Unless specifically stated otherwise or the context clearly contradicts it, connective language, such as phrases of the form "at least one of A, B, and C" or "at least one of A, B, and C," is otherwise to be understood in the context in which it is generally used to indicate that an object, expression, etc., can be either A, B, or C, or any non-empty subset of the sentence consisting of A, B, and C. For example, in the illustrated example of a sentence containing three elements, the connective phrases "at least one of A, B, and C" and "at least one of A, B, and C" refer to any one of the following: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}.Thus, such connecting expressions are generally not intended to express that certain embodiments require the presence of at least one of A, at least one of B, and at least one of C. Additionally, unless otherwise stated or contradicted by the context, the term "plurality" also denotes a state of plurality (e.g., "a plurality of elements" denotes multiple elements). In at least one embodiment, a plurality of elements consists of at least two, but may also include more if this is either explicitly stated or indicated by the context. Furthermore, unless otherwise stated or evident from the context, the phrase "based on" means "at least partially based on" and not "exclusively based on."

[0169] The operations of methods described herein may be performed in any suitable order unless otherwise specified herein or the context clearly precludes it. In at least one embodiment, a method such as the methods described herein (or variations and / or combinations thereof) is performed under the control of one or more computer systems configured with executable instructions, and it is implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) executed together on one or more processors, by hardware or combinations thereof.In at least one embodiment, code is stored on a computer-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions that can be executed by one or more processors. In at least one embodiment, a computer-readable storage medium is a non-transient computer-readable storage medium that excludes transitory signals (e.g., a propagating transient electrical or electromagnetic transmission) but includes non-transient data storage circuits (e.g., buffers, caches, and queues) within transient signal transmitters / receivers. In at least one embodiment, the code (e.g.,Executable code or source code) is stored on a set of one or more non-transitory, computer-readable storage media on which executable instructions are stored (or other storage for executable instructions) which, when executed (i.e., as a result of execution) by one or more processors of a computer system, cause the computer system to perform the operations described herein. In at least one embodiment, a set of non-transitory, computer-readable storage media comprises multiple non-transitory storage media, and one or more of the individual non-transitory storage media do not contain the entire code, while multiple non-transitory, computer-readable storage media collectively store the entire code.In at least one embodiment, the executable instructions are implemented such that different instructions are executed by different processors—for example, a non-transient computer-readable storage medium stores instructions, and a central processing unit (“CPU”) executes some of the instructions, while a graphics processing unit (“GPU”) executes other instructions. In at least one embodiment, different components of a computer system have separate processors, and different processors execute different subsets of instructions.

[0170] In at least one embodiment, an arithmetic logic unit is a set of combinational logic circuits that receives one or more inputs to produce a result. In at least one embodiment, an arithmetic logic unit is used by a processor to perform mathematical operations such as addition, subtraction, or multiplication. In at least one embodiment, an arithmetic logic unit is used to implement logical operations such as a logical AND / OR or XOR. In at least one embodiment, an arithmetic logic unit is stateless and made of physical switching components such as semiconductor transistors arranged to form logic gates. In at least one embodiment, an arithmetic logic unit can operate internally as a stateful logic circuit with an associated clock.In at least one embodiment, an arithmetic logic unit can be implemented as an asynchronous logic circuit with an internal state that is not maintained in an associated register set. In at least one embodiment, an arithmetic logic unit is used by a processor to combine operands stored in one or more registers of the processor and to generate an output that can be stored by the processor in another register or in memory location.

[0171] In at least one embodiment, the processor, as a result of processing an instruction retrieved by the processor, presents one or more inputs or operands to an arithmetic logic unit (ALU), whereby the ALU generates a result that is at least partially based on an instruction code provided as input to the ALU. In at least one embodiment, the instruction codes provided by the processor to the ALU are at least partially based on the instruction executed by the processor. In at least one embodiment, combinational logic in the ALU processes the inputs and generates an output that is placed on a bus within the processor.In at least one embodiment, the processor selects a destination register, memory location, output device or output memory location on the output bus, so that by clocking the processor the results generated by the ALU are sent to the desired location.

[0172] Within the scope of this application, the term "arithmetic logic unit" or "ALU" is used to denote any computational logic circuit that processes operands to produce a result. For example, the term "ALU" in this description may refer to a floating-point unit, a DSP, a tensor core, a shader core, a coprocessor, or a CPU.

[0173] Accordingly, computer systems in at least one embodiment are configured to implement one or more services that, individually or collectively, perform operations of the methods described herein, and such computer systems are configured with applicable hardware and / or software that enables the execution of operations. Furthermore, a computer system implementing at least one embodiment of the present disclosure is a single device, and in another embodiment, a distributed computer system comprising several devices that operate differently, such that the distributed computer system performs the operations described herein and such that a single device does not perform all operations.

[0174] The use of any and all examples or illustrative language (e.g., "such as") provided in this document is intended solely to better clarify the embodiments of the disclosure and does not constitute a limitation of the scope of the disclosure unless claimed otherwise. No wording in the description should be interpreted as indicating any unclaimed element as essential to the implementation of the disclosure.

[0175] In the description and claims, the terms "coupled" and "connected," along with their derivatives, may be used. It is understood that these terms are not intended to be synonymous. Rather, in specific examples, "connected" or "coupled" can be used to indicate that two or more elements are in direct or indirect physical or electrical contact with each other. "Coupled" can also mean that two or more elements are not in direct contact with each other, but nevertheless interact or work together.

[0176] Unless expressly stated otherwise, terms such as "processing", "calculating", "calculating", "determining" or the like throughout this description are understood to refer to actions and / or procedures or processes of a computer or computing system or similar electronic computing device that manipulate and / or convert data represented as physical, e.g. electronic, quantities in the registers and / or memory of the computing system into other data represented in a similar manner as physical quantities in the memory, registers or other such information storage, transmission or display devices of the computing system.

[0177] Similarly, the term "processor" can refer to any device or section of a device that processes electronic data from registers and / or memories and converts that electronic data into other electronic data that can be stored in registers and / or memories. As non-restrictive examples, the "processor" can be a CPU or a GPU. A "computing platform" can include one or more processors. As used herein, "software" processes or "software" procedures can, for example, include software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Furthermore, each procedure can refer to multiple methods for executing instructions sequentially or in parallel, continuously or intermittently.In at least one embodiment, the terms "system" and "method" are used interchangeably here insofar as a system can embody one or more methods and the methods can be considered as a system.

[0178] This document may refer to the acquisition, capture, reception, or input of analog or digital data into a subsystem, computer system, or computer-implemented machine. In at least one embodiment, a method for acquiring, capturing, receiving, or inputting analog and digital data can be accomplished in various ways, such as receiving data as a parameter of a function call or a call to an application programming interface. In at least one embodiment, the method of acquiring, capturing, receiving, or inputting analog or digital data can be achieved by transmitting data over a serial or parallel interface.In at least one embodiment, methods for obtaining, acquiring, receiving, or inputting analog or digital data can be performed by transmitting data over a computer network from the providing entity to the acquiring entity. In at least one embodiment, reference can also be made to providing, outputting, transmitting, sending, or displaying analog or digital data. In various examples, the methods for providing, outputting, transmitting, sending, or displaying analog or digital data can be performed by transmitting data as input or output parameters of a function call, a parameter of an application programming interface, or an interprocess communication mechanism.

[0179] Although the descriptions presented here are exemplary implementations of the described procedures, other architectures may also be used to implement the described functionality, and these are intended to be within the scope of this disclosure. Furthermore, although specific distributions of responsibilities may be defined above for the purpose of discussion, various functions and responsibilities may be distributed and divided differently depending on the circumstances.

[0180] Although the subject matter has been further described in language specific to structural features and / or process steps, it is understood that the subject matter claimed in the appended claims is not necessarily limited to the specific features or steps described. Rather, specific features and steps are disclosed as exemplary ways of implementing the claims.

Claims

[1] Computer-implemented method, comprising: Determine, using a neural network (120), a variety of motion properties (130) for at least a first set of pixels in a first image and a corresponding second set of pixels in a second image from a sequence of images in a video stream (110), including: Analyzing pixels in the first image (301) and the second image (302) and generating, for each pixel, two confidence values, where one confidence value (321 / 331) corresponds to a probability that a movement of the pixel matches given movement vectors, and the other confidence value (322 / 332) represents a static assumption; Warping of motion vector confidence values ​​for the first image and the second image; Determining mixed weights (340 / 350) based on the warped confidence values ​​for the first image and the second image; Generating, for an intermediate image to be generated at a point in the sequence of images between the first image and the second image, a first candidate image (360) based partly on the motion properties applied in a first direction to one or more pixels of the first image; Generating, for the intermediate image, a second candidate image (370) based partly on the motion properties applied in a direction opposite to the direction of the second image; Generating the intermediate image (390) between the first image and the second image by aligning and blending values ​​for one or more pixels of the first candidate image and the second candidate image; and Providing the generated intermediate image for display on a display device. [2] Computer-implemented method according to claim 1, wherein generating the first candidate image (260) and the second candidate image (270) comprises warping the first set of pixels from the first image (220) and the second set of pixels from the second image (230) to an intermediate position (280) based on the plurality of motion properties (290 / 291). [3] Computer-implemented method according to claim 1 or 2, wherein an optical flow model (315) is used to determine additional motion properties, and wherein the first candidate image (346) and the second candidate image (347) are further generated based on the additional motion properties determined based on the optical flow model. [4] Computer-implemented method according to claim 3, wherein the mixing of the first candidate image and the second candidate image comprises applying a plurality of mixing weights calculated on the basis of confidence values ​​of the pixels, the confidence values ​​indicating a probability that the pixels correspond to a static content. [5] Computer-implemented method according to any of the preceding claims, wherein generating the intermediate image (349) further comprises: Aligning the first candidate image (346) and the second candidate image (347) using a second neural network (348); and Mixing the aligned first candidate image and the second candidate image using the second neural network to generate the intermediate image. [6] Computer-implemented method according to any one of the preceding claims, further comprising: Storing data associated with a classification and motion characteristics for reuse across multiple intermediate frames between the first and second frames; and Generating additional intermediate images between the first image and the second image, with each additional intermediate image being generated using the stored data. [7] At least one processor, comprising: one or more processing units, set up to carry out a process, comprising: Determine, using a neural network (120), a variety of motion properties (130) for at least a first set of pixels in a first image and a corresponding second set of pixels in a second image of a video sequence (110), including: Analyzing pixels in the first image (301) and the second image (302) and generating, for each pixel, two confidence values, where one confidence value (321 / 331) corresponds to a probability that a movement of the pixel matches given movement vectors, and the other confidence value (322 / 332) represents a static assumption; Warping of motion vector confidence values ​​for the first image and the second image; Determining mixed weights (340 / 350) based on the warped confidence values ​​for the first image and the second image; Generating, for an intermediate image to be generated at a point in the sequence of images between the first image and the second image, a first candidate image (360) based partly on the motion properties applied in a first direction to pixels of the first image; Generating, for the intermediate image, a second candidate image (370) based on the motion properties applied in a direction opposite to the direction of the second image; Generating the intermediate image (390) between the first image and the second image by aligning and mixing the first candidate image and the second candidate image; and providing the generated intermediate image for display on a display device. [8] Processor according to claim 7, wherein generating the first candidate image (260) and the second candidate image (270) comprises warping the first set of pixels from the first image (220) and the second set of pixels from the second image (230) to an intermediate position based on the plurality of motion properties (290 / 291). [9] Processor according to claim 7 or 8, wherein an optical flow model (315) is used to determine additional motion properties, and the first candidate image (346) and the second candidate image (347) are further generated based on the additional motion properties determined based on the optical flow model. [10] Processor according to claim 9, wherein the mixing of the first candidate image and the second candidate image comprises applying a plurality of mixing weights calculated on the basis of confidence values ​​of the pixels, the confidence values ​​indicating a probability that the pixels correspond to a static content. [11] Processor according to any one of claims 7 to 10, wherein generating the intermediate image (349) further comprises: Aligning the first candidate image (346) and the second candidate image (347) using a second neural network (348); and Mixing the aligned first candidate image and the aligned second candidate image using the second neural network to generate the intermediate image. [12] Processor according to any one of claims 7 to 11, wherein the processor is included in a system comprising at least one of: a system for performing simulation operations; a system for performing simulation operations to test or validate autonomous machine applications; a system for performing digital twin operations; a system for performing a light transport simulation; a system for displaying a graphical output; a system for performing deep learning operations; a system implemented using an edge device; a system for generating or displaying virtual reality (VR) content; a system for generating or displaying augmented reality (AR) content; a system for generating or displaying content of mixed reality, MR; a system that includes one or more virtual machines, VMs; a system that is at least partially implemented in a data center; a system for performing hardware tests using a simulation; a system for generating synthetic data; a system for performing generative AI operations; a system implemented using one or more large language models, LLMs; a system that is implemented using one or more Vision Language Models (VLMs); a system that is implemented using one or more multimodal language models; a system that uses or employs one or more inference microservices; a system that includes one or more machine learning models deployed in a service or microservice along with an OS-level virtualization package, e.g., a container; a collaborative content creation platform for 3D assets; or a system that is implemented at least partially using cloud computing resources.

Images