Hierarchical base model

Abstract

This disclosure relates to hierarchical foundational models, specifically disclosing apparatus, systems, methods, and techniques for using hierarchically arranged machine learning processes (e.g., neural networks) to generate at least one latent representation of two or more input data streams. In at least one embodiment, one or more tasks are performed at least in part based on the at least one latent representation.

Description

Technical Field

[0001] At least one embodiment involves multiple neural networks that generate latent representations of a data stream that transmits multimodal, multivariate, multicomponent, and / or multi-source data. In at least one embodiment, the multiple neural networks are components of a base model that can be pre-trained to encode data from the data stream and fine-tuned to perform a specific task. In at least one embodiment, the multiple neural networks are arranged hierarchically to align and merge the data stream to generate latent representations. In at least one embodiment, a computing system (e.g., within a data center) implements various novel techniques described herein. Background Technology

[0002] Data centers process and / or generate large amounts of data. For example, data related to one or more devices within the data center (such as logs, telemetry data, etc.) is transmitted within the data center as data streams. The data transmitted in these streams can contain various formats, including different values ​​for various parameters, collected at different times, collected at different frequencies, and / or using different units of measurement. This heterogeneity in data streams makes aggregation and / or analysis difficult. Technologies for managing and analyzing data collected in data centers can be improved. Attached Figure Description

[0003] Figure 1 The diagram illustrates an example architecture that can be used to build a data center, based on at least one embodiment. Figure 2A This is a block diagram illustrating an example of a tree-like basic model (FM) with a hierarchical structure according to at least one embodiment; Figure 2B A block diagram illustrating a plurality of tree-like FMs arranged to produce an output, according to at least one embodiment; Figure 3A The diagram illustrates an example data structure according to at least one embodiment; Figure 3B The illustration shows a diagram of a structure according to at least one embodiment. Figure 2A A flowchart illustrating the reasoning process performed by the leaf FMs in a tree-like FM; Figure 3C The illustration shows a diagram of a structure according to at least one embodiment. Figure 2A A diagram illustrating the reasoning process performed by FM nodes (which may differ from leaf FMs) in a tree-structured FM; Figure 4 A block diagram depicting a training process according to at least one embodiment is shown; Figure 5The diagram illustrates a flowchart of a method using hierarchical FM according to at least one embodiment; Figure 6 The flowchart illustrates a training method for training leaf FMs in a hierarchical FM, according to at least one embodiment. Figure 7 The flowchart illustrates a training method for training FM nodes in a hierarchical FM according to at least one embodiment. Figure 8 An example of a system including a driver and / or runtime according to at least one embodiment is shown, the driver and / or runtime including one or more libraries to provide one or more application programming interfaces (APIs); Figure 9 A block diagram illustrating examples of processors and modules according to at least one embodiment is shown; Figure 10A The logic according to at least one embodiment is shown; Figure 10B The logic according to at least one embodiment is shown; Figure 11 An example data center system according to at least one embodiment is shown; and Figure 12 A block diagram of a computer system according to at least one embodiment is shown. Detailed Implementation

[0004] Various techniques have been described in the preceding and following sections. For illustrative purposes, specific configurations and details have been elaborated to provide a thorough understanding of the possible ways to implement these techniques. However, it will be apparent that the techniques described below can be practiced in different configurations without specific details. Furthermore, well-known features may be omitted or simplified to avoid obscuring the techniques described.

[0005] A foundational model (FM) is a machine learning model that provides the "foundation" from which more specialized models are generated. FM can be pre-trained on general data and then fine-tuned for specific applications. Hierarchical FM (e.g., Figure 2A The tree-structured FM 200 illustrated in the diagram can be used to generate a latent representation (e.g., a sequence of vectors) of combinations of different data streams carrying different data. This latent representation can then be used to perform one or more analyses, which can be characterized as being performed simultaneously on different data streams.

[0006] A hierarchical FM can have a hierarchical structure comprising nodes (called FM nodes) arranged in one or more layers. Each FM node is a separate, independent FM, and a hierarchical FM can be characterized as an aggregation or combination of these independent FMs. These layers can be described as stacked on top of each other, such that some layers are on top of others. However, this stacking arrangement is for illustrative and descriptive purposes only and is not a requirement of the hierarchical FM. Each layer comprises one or more FM nodes.

[0007] One or more data streams may be received and / or generated by the data center and provided to at least some of the independent FM nodes in a hierarchical FM. In a hierarchical FM, any FM node that does not receive output from a preceding FM node in the same hierarchical FM is called a leaf FM. Each leaf FM may receive a single data stream as input. In a hierarchical FM, the output of a leaf FM that can be characterized as a parent node may be input to one or more FM nodes that can be characterized as one or more child nodes relative to that leaf FM. In a hierarchical FM, the outputs of two or more leaf FMs may be provided to another FM node, which may align and / or fuse the outputs to generate a combined latent representation, which may be provided to another FM node and / or used to perform one or more tasks (e.g., pattern recognition, prediction, and / or anomaly detection). The output of another hierarchical FM may be provided as input to a leaf FM or an FM node (not a leaf FM). When a specific output of another hierarchical FM is provided to a specific FM node (not a leaf FM), that specific FM node can combine that specific output with the output of the hierarchical FM's FM (e.g., a leaf FM or an FM node that is not a leaf FM). The hierarchical FM can be trained in a self-supervised manner on different data streams (such as multivariate, multi-source, multimodal, and / or multi-component data streams) to align and / or fuse data from different data streams.

[0008] Figure 1 The diagram illustrates a block diagram of an example architecture that can be used to construct a data center 100, according to at least one embodiment. The data center 100 may include various software and hardware for processing, storing, managing, and disseminating data and information. The data center 100 may include an artificial intelligence (AI) factory data center, or other dedicated data centers designed to support the computing needs of AI applications. References below... Figure 11 Further details are provided regarding one or more components that may be included in data center 100. In at least one embodiment, at least a portion of data center 100 is used in… Figures 8-12 The description and / or about Figures 8-12 This is implemented by at least a portion of any one or more of the systems described. In at least one embodiment, at least a portion of data center 100 is used to implement... Figures 8-12 The description and / or about Figures 8-12 At least a part of any one or more systems described.

[0009] In at least one embodiment, data center 100 processes one or more data streams 101. The one or more data streams 101 may include continuous data streams that can be generated and / or transmitted in real time. Each of the one or more data streams 101 may consist of a sequence of data elements generated over a period of time (e.g., continuously), such as by one or more sensors, one or more monitors, one or more data collection processes (e.g., implemented by software and / or hardware), and / or one or more other data collection methods. Each of the data elements may have a data format, such as a structured, semi-structured, or unstructured format. The one or more data streams 101 may be processed (e.g., in real time or near real time) to extract useful information and / or insights, enabling timely decision-making and / or action execution in response to such information and / or insights.

[0010] The one or more data streams 101 may include one or more incoming data streams 101A and / or one or more data center internal data streams 101B. The one or more incoming data streams 101A are received from one or more data sources outside the data center 100. For example, the one or more incoming data streams 101A may be received at the data center 100 from one or more computing devices (e.g., one or more personal computing devices, one or more Internet of Things (IoT) devices, one or more wearable devices (e.g., augmented reality (AR) headsets, virtual reality (VR) headsets, mixed reality headsets and / or other wearable devices), one or more vehicles and / or one or more other types of devices). The one or more data center internal data streams 101B originate from within the data center 100. For example, the one or more data center internal data streams 101B may be transmitted from another component within the data center 100 to one or more of the servers 104. For example, one of the servers 104 may transmit one or more of the one or more data center internal data streams 101B to another of the servers 104. The one or more data center intra-data center data streams 101B can be transmitted and processed by the same server (e.g., one of the servers 104). For example, one or more devices of a particular server 104 can generate the one or more data center intra-data center data streams 101B, and that particular server can process the one or more data streams transmitted by those devices.

[0011] The data transmitted by the one or more data streams 101 may include user-generated content from web applications, sensor data, transactional data, multimedia content, and / or one or more other types of content. The data transmitted by the one or more data streams 101 may include telemetry data, log data, time-series data, structured data, semi-structured data, unstructured data, and / or one or more other types of data. In at least one embodiment, the one or more data streams 101 may include multimodal data, multivariate data, multicomponent data, and / or multi-source data.

[0012] Multimodal data is heterogeneous, assigned to multiple data types, and / or stored using multiple data formats. Multimodal data includes data with two or more modalities. Different modalities can be captured using different mechanisms (e.g., using different sensors and / or different methods). Different modalities can provide one or more different types of information. For example, modalities can include text data, audio data, image data, video data, sensor data, spatial data, temporal data, tactile data, and / or one or more other types of data. Regarding data center 100, examples of multimodal data can include: logs, profiles, group analysis results, central processing unit (CPU) usage metrics, memory usage metrics, disk I / O metrics, telemetry data, and / or sensor data (e.g., data collected by temperature sensors, humidity sensors, power usage sensors, surveillance cameras, thermal imaging devices, ambient noise sensors, traffic flow sensors, rack layout sensors, equipment layout sensors, and / or other types of sensors). By integrating and analyzing multimodal data (e.g., using the Basic Model (FM) function 116), a more comprehensive understanding of complex phenomena and / or situations can be obtained. FM function 116 can be used to fuse multimodal data for use in one or more machine learning processes and / or one or more artificial intelligence (AI) processes.

[0013] Multivariate data includes values ​​corresponding to multiple variables (e.g., one or more parameters, or one or more features). Each observation in a multivariate dataset includes values ​​for more than one variable. For data center 100, examples of multivariate data may include: CPU usage metrics, memory usage metrics, disk I / O metrics, traffic flow metrics, connectivity metrics, telemetry data, temperature data, humidity data, power consumption metrics, energy efficiency data, logs (e.g., access logs, alarm logs, and / or others), monitoring data, equipment status data, and / or other data. Multivariate data can be used to analyze complex relationships and / or interactions between variables. FM function 116 can be used to align and / or fuse multivariate data for processing (e.g., analysis) through one or more machine learning processes and / or one or more AIs. By analyzing multivariate data, patterns, correlations, and / or trends can be identified in the data that are not apparent when analyzing each variable individually.

[0014] Multi-component data includes metrics collected for multiple components. Components may include: one or more devices (e.g., individual GPUs), one or more groups of the one or more devices (e.g., a deep learning system including individual GPUs), one or more software processes, one or more groups of the one or more software processes, and / or other types of components.

[0015] Multi-source data includes data classified into one or more source categories. As a non-limiting example, source categories may include "computing," "networking," and / or others. Data classified as "computing" may be collected for components performing computation (e.g., processors, GPUs, CPUs, etc.) and / or may be otherwise related to computation (e.g., GPU utilization metrics). Data classified as "networking" may be collected for components performing networking (e.g., routers, network interface controllers, etc.) and / or may be otherwise related to networking (e.g., quality of service, signal-to-noise ratio, bandwidth utilization metrics, network latency metrics, etc.).

[0016] For example, the data transmitted by one or more data streams 101 may have different formats, including different parameter values, collected at different times, collected at different frequencies, including the same parameter values ​​collected using different units of measurement, etc. The diversity of the data transmitted in the one or more data streams 101 may be ingested, processed (e.g., performed by FM function 116) and / or stored by data center 100 for subsequent analysis and / or utilization (e.g., through one or more machine learning processes and / or one or more AI).

[0017] In embodiments including the one or more incoming data streams 101A, the one or more incoming data streams 101A may be received at a data center gateway 102. For example, the data center gateway 102 may include hardware (and associated software) for receiving data from the one or more incoming data streams 101A and distributing the received data to other components of the data center 100. As a non-limiting example, the hardware of the data center gateway 102 may include: one or more network interface cards (NICs) for facilitating the reception and transmission of data packets; one or more routers and / or switches for managing data flow between external sources of the one or more data streams 101A and internal components of the data center 100; one or more firewalls for filtering the one or more incoming data streams 101A to prevent unauthorized access; a load balancer for distributing incoming data traffic to one or more servers 104; and / or one or more other hardware components. In at least one embodiment, at least a portion of the data center gateway 102 is used in… Figures 8 to 12 The description and / or about Figures 8 to 12 It is implemented by at least one part of any or more systems described. In at least one embodiment, at least a part of the data center gateway 102 is used to implement in Figures 8 to 12 The description and / or about Figures 8 to 12At least one part of any or more systems described. Some embodiments may omit the one or more incoming data streams 101A, and such embodiments may also omit the data center gateway 102.

[0018] In at least one embodiment, one or more servers 104 may be implemented as one or more computing devices that host applications and / or provide services to end users. For example, one or more servers 104 may be equipped with one or more processors 105, storage devices 110, and / or memory 112. The one or more processors 105 may include one or more CPUs 106, one or more graphics processing units (GPUs) 108, one or more parallel processing units (PPUs), a main central processing unit (“CPU”) complex, one or more microprocessors, one or more microcontrollers, one or more data processing units (DPUs), one or more arithmetic logic units (“ALUs”), and / or one or more other types of processors. In at least one embodiment, at least a portion of the one or more processors 105 is used in… Figures 8 to 12 The description and / or about Figures 8 to 12 It is implemented by at least a portion of any one or more of the systems described. In at least one embodiment, at least a portion of the one or more processors 105 is used to implement... Figures 8 to 12 The description and / or about Figures 8 to 12 At least one part of any or more systems described. In at least one embodiment, at least a portion of one or more of the servers 104 is used in... Figures 8 to 12 The description and / or about Figures 8 to 12 It is implemented by at least a portion of any one or more of the described systems. In at least one embodiment, at least a portion of one or more of the servers 104 is used to implement... Figures 8 to 12 The description and / or about Figures 8 to 12 At least one part of any one or more systems described.

[0019] The one or more servers 104 can manage a variety of computing workloads. The one or more servers 104 can be interconnected via a network infrastructure (e.g., a network within data center 100) to facilitate communication and data exchange between different servers within the one or more servers 104 and / or between the one or more servers 104 and one or more other devices. The network infrastructure may include one or more network devices (not shown), such as one or more switches, one or more firewalls, one or more hubs, one or more modems, one or more access points (APs), one or more repeaters, one or more bridges, one or more gateways, one or more network interfaces (e.g., one or more network interface controllers (NICs)), one or more load balancers, and / or one or more routers that route data traffic within the internal network (e.g., between one or more servers 104).

[0020] In at least one embodiment, the one or more processors 105 (e.g., one or more CPUs 106) perform tasks such as executing instructions and / or performing one or more computations. For example, the one or more CPUs 106 can process data by performing arithmetic and / or logical operations. The one or more CPUs 106 can interface with other components of the one or more servers 104 to coordinate the execution of one or more tasks. For example, the one or more CPUs 106 can provide instructions to the one or more GPUs 108 to perform parallel processing of data. In at least one embodiment, the one or more GPUs 108 accelerate the execution of one or more tasks through parallel processing. For example, the one or more GPUs 108 can perform large-scale computations, such as those used for machine learning and data analysis. By combining the one or more GPUs 108 with the one or more CPUs 106, the computational efficiency of the one or more servers 104 can be improved.

[0021] In at least one embodiment, storage device 110 is a component of the one or more servers 104 and / or accessible by the one or more servers 104, performing data storage. For example, storage device 110 may include storage media, such as hard disk drives and solid-state drives, for providing high-capacity, high-speed data access. Storage device 110 can maintain data integrity and support fast data retrieval when data is requested for processing.

[0022] In at least one embodiment, memory 112 is a component of the one or more servers 104 and / or accessible by the one or more servers 104, providing data access. For example, memory 112 may include random access memory (RAM) that facilitates fast data access, thereby improving the speed and efficiency of tasks performed at data center 100. Memory 112 can be managed to optimize resource allocation and performance.

[0023] Storage device 110 and / or memory 112 may include one or more non-transitory machine-readable media. Storage device 110 and / or memory 112 (e.g., non-transitory machine-readable media) may store executable instructions 114 that, if executed by one or more of the one or more processors 105 (e.g., one or more of the one or more CPUs 106 and / or one or more of the one or more GPUs 108), implement FM function 116. Storage device 110 and / or memory 112 (e.g., one or more non-transitory processor-readable media) may be implemented using, for example, volatile memory (e.g., dynamic random access memory (“DRAM”)) and / or non-volatile memory (e.g., hard disk drives, solid-state devices (“SSDs”), etc.). In at least one embodiment, at least a portion of storage device 110 and / or at least a portion of memory 112 are used in… Figures 8 to 12 The description and / or about Figures 8 to 12 It is implemented by at least a portion of any one or more of the systems described. In at least one embodiment, at least a portion of storage device 110 and / or at least a portion of memory 112 are used to implement in Figures 8 to 12 The description and / or about Figures 8 to 12 At least one part of any one or more systems described.

[0024] The components of one or more servers 104 can be used to generate results based on operations performed on data processed at the one or more servers 104. For example, one or more servers 104 can perform FM function 116, which can produce results. In at least one embodiment, the result of FM function 116 is provided as output 120 (e.g., the output of data center 100 and / or one or more servers 104). Output 120 may remain within data center 100 and / or may be sent to one or more receiving destinations. For example, receiving destinations may include end users and / or other systems. Data center 100, one or more servers 104, and / or one or more external devices transmitting one or more data streams 101A to data center 100 can perform various tasks in various environments such as factories, medical facilities (e.g., hospitals), offices, homes, vehicles, robots, warehouses, and / or any suitable context or environment. The one or more data streams 101 may include data related to performing such tasks, such as information related to computing resources used to perform the task. The one or more data streams 101 may include data related to performing machine learning tasks and / or one or more other computationally intensive tasks.

[0025] As discussed above, the one or more data streams 101 may include complex data with different modalities and / or values ​​of different variables and / or parameters. In at least one embodiment, the data transmitted in the one or more data streams 101 is collected and / or combined (e.g., aggregated) by one or more of the one or more servers 104 for further processing. FM function 116 can be used to aggregate two or more of the one or more data streams 101, such as transmitting multimodal data and / or multivariate data, and use the aggregated data to identify and / or characterize patterns in the one or more data streams 101. Such pattern identification can be applied to tasks such as predicting future events and / or detecting anomalous activity (e.g., anomalies), allowing for appropriate adjustments to any affected system (e.g., data center 100, one or more servers 104, and / or other systems inside or outside data center 100). By accurately identifying data patterns used for downstream decision-making and / or anomaly detection, the performance of one or more systems (such as data center 100) can be improved.

[0026] If FM function 116 is executed by at least one of the one or more processors 105 (e.g., at least one of the one or more CPUs 106 and / or at least one of the one or more GPUs 108), then it implements hierarchical FM (e.g., Figure 2AThe tree-structured FM 200 illustrated in the figure is used to generate a latent representation (e.g., a sequence of vectors) representing combinations of different data streams (e.g., the one or more data streams 101) that deliver different data (e.g., multimodal data, multivariate data, multicomponent data, and / or multisource data). This latent representation (e.g., Figure 2A The FM tree output 210 shown in the figure can then be used to perform one or more analyses (e.g., using one or more machine learning processes, such as one or more neural networks) on different data streams (e.g., the one or more data streams 101).

[0027] The one or more data streams 101 may be received at and / or generated by the data center 100 and provided to at least some of the various FM nodes of the hierarchical FM. For example, the one or more data streams 101 may be provided only to a set of leaf FMs at the first or lowest level of the hierarchical FM. However, any level of the hierarchical FM may include one or more leaf FMs. Each leaf FM may process one or more data streams generated at one or more computing devices within the data center 100 (e.g., data streams generated by the GPU and / or NIC of a particular computing device).

[0028] Figure 2A This is a block diagram illustrating an example of a tree-like FM 200 with a hierarchical structure according to at least one embodiment. The FM function 116 can implement the tree-like FM 200 if it is executed by at least one of the one or more processors 105 (e.g., at least one CPU of the one or more CPUs 106 and / or at least one GPU of the one or more GPUs 108). The tree-like FM 200 includes a plurality of FM nodes 202 arranged in a hierarchical structure. The FM nodes 202 can be arranged in layers (such as layers 204-208) that can be described as stacked on top of each other, such that some layers are above other layers. However, such a stacking arrangement is for illustrative and descriptive purposes only and is not a requirement of the tree-like FM 200. In the tree-like FM 200, each subsequent (e.g., higher in order) layer may include fewer FM nodes than a previous (e.g., lower) layer. However, this is not mandatory, and in some embodiments, one or more layers may include more FM nodes than at least one previous layer.

[0029] Tree-like FM 200 may include a first (e.g., bottom) layer 204, one or more intermediate layers (e.g., intermediate layer 206), and a final (e.g., top) layer 208. However, tree-like FM 200 may include any number of layers, including a single layer. For example, tree-like FM 200 may omit intermediate layer 206. Furthermore, tree-like FM 200 may include one or more layers above the final layer 208, and / or one or more layers below the first layer 204. For example, although… Figure 2A Only a single intermediate layer 206 is shown; the tree-like FM 200 can include any number of intermediate layers located between the first layer 204 and the final layer 208. Furthermore, the number of FM nodes 202 included in the first layer 204, intermediate layer 206, and / or final layer 208 can be [related to the previous sentence]. Figure 2A The difference is not shown. For example, the first layer 204, each intermediate layer (e.g., intermediate layer 206), and / or the final layer 208 may include any number of FM nodes 202, including a single FM node.

[0030] As described in this article, an FM node in a tree FM 200 that does not receive the output from another FM node in the tree FM 200 as input is called a leaf FM. Figure 2A In the tree FM 200, FM nodes 202-1-1 to 202-1-B are leaf FMs. In the tree FM 200, the output of one or more leaf FMs, which can be represented as one or more parent nodes, is input to one or more FM nodes, which can be represented as one or more child nodes of that leaf FM. For example, in... Figure 2A In this example, FM node 202-2-1 is a child node relative to parent leaf nodes FM 202-1-1 and 202-1-2. Furthermore, FM node 202-2-2 is a child node relative to parent leaf node FM 202-1-B and possibly one or more other leaf nodes FM. As another non-limiting example, FM node 202-3-1 is a child node relative to parent leaf nodes FM 202-2-1 and 202-2-C.

[0031] In at least one embodiment, each leaf FM of the first layer 204 receives one or more of the one or more data streams 101. For example, each leaf FM of the first layer 204 may receive data streams from a specific computing device. As a non-limiting example, leaf FM 202-1-1 of the first layer 204 may receive a first data stream 201-1, leaf FM 202-1-2 may receive a second data stream 201-2, and leaf FM 202-1-B of the first layer 204 may receive an S-th data stream 201-S. Each leaf FM in the first layer 204 may generate a potential representation of its corresponding incoming data stream, and this potential representation may be an input to the FM node of the intermediate layer 206. For example, the outputs from both leaf FM 202-1-1 and leaf FM 202-1-2 may be provided to the FM node 202-2-1 of the intermediate layer 206. Furthermore, the output from leaf FM 202-1-B and at least one other FM node of the possible first layer 204 can be provided to FM node 202-2-C of the intermediate layer 206. As a non-limiting example, the data output from the first layer 204 may include a potential representation (e.g., encoding) of multivariable device internal data, which can be provided to the FM node of the intermediate layer 206 for further processing.

[0032] In at least some embodiments, each leaf FM of the first layer 204 can generate a potential representation of data from a data stream of a single type and / or a data stream originating from a single source. For example, a first data stream 202-1 received by FM node 202-1-1 may include multivariate time-series data from a data center GPU manager (DCGM) of a specific GPU, a second data stream 202-2 received by FM node 202-1-2 may include kernel system log data, which may be semi-structured data generated by the same GPU, and the S-th data stream 202-S received at FM node 202-1-B may include another type of data from the same GPU.

[0033] The FM nodes of intermediate layer 206 can align and / or fuse data (e.g., latent representations) received from two or more leaf FMs in first layer 204, generating a latent representation of the aligned and / or fused data. The latent representations output by the FM nodes of intermediate layer 206 can be fed into one or more FM nodes 202 on final layer 208. The data output from intermediate layer 206 may include, for example, multimodal, multi-source, and device-internal data, which can be received at final layer 208 for final processing.

[0034] exist Figure 2AIn the illustrated example, the FM node 202-3-1 of the final layer 208 can align and / or fuse data received from the intermediate layer 206, generating one or more latent representations of the aligned and / or fused data. The FM node 202-3-1 of the final layer 208 can obtain data from other tree-structured FMs, align and / or fuse the obtained data with the data received from the intermediate layer 206, and generate one or more latent representations of the aligned and / or fused data.

[0035] In at least one embodiment, the structure of the tree-like FM 200 (e.g., edges connecting nodes) can be determined based on: 1) prior knowledge, and / or 2) learning. When the structure is determined based on prior knowledge, FM nodes 202 can be interconnected according to logic defined at least partially by the prior knowledge. For example, the hierarchical structure of the layers of the tree-like FM 200 can be constructed based on domain knowledge. When the structure of the tree-like FM 200 is determined based on prior knowledge, one or more intermediate layers (e.g., intermediate layer 206) can include one or more FM nodes 202 that receive data from at least a portion (e.g., two or more) of the FM nodes of the first layer 204. In at least one embodiment, the FM nodes of the intermediate layer 206 can align the latent representations received from the FM nodes of the first layer 204 based on one or more variables (e.g., time). The FM nodes of the intermediate layer 206 can further fuse or merge the latent representations received from the first layer 204, each latent representation representing a different data stream. For example, FM node 202-1-1 can generate DCGM encoding from a first data stream 202-1, FM node 202-1-2 can generate system log encoding from a second data stream 202-2 (e.g., sending system logs collected for a deep learning system such as DGX), and FM node 202-2-1 can align the DCGM encoding with the system log encoding based on a device identifier (e.g., DGX ID) and one or more defined time periods. FM node 202-2-1 can merge and / or fuse these aligned latent representations, for example, by summing the aligned latent representations, concatenating the aligned latent representations, and / or feeding the aligned latent representations as input to a trained linear layer to generate a fused latent representation. FM function 116 can then feed the fused latent representation as input to one or more neural networks (e.g., a transformer encoder) to produce a latent representation as output. This latent representation can be used as input to another FM node and / or another process. For example, FM node 202-3-1 can receive this latent representation as input. This potential representation could represent the DCGM data and system logs of a set of hosts (e.g., a set of deep learning systems) clustered according to a certain aspect (i.e., by the server room they are located in in data center 100). FM node 202-3-1 can be aggregated along the axis or D dimension of the hosts (e.g., deep learning systems).

[0036] The latent representation output by intermediate layer 206 can be clustered based on one or more aspects (e.g., the server room where the computing devices are located in data center 100), and one or more FM nodes 202 can be created for each cluster at the final layer 208 of the tree-like FM 200. In at least one embodiment, the FM nodes of the final layer 208 (e.g., FM node 202-3-1) receive the outputs of two or more FM nodes of intermediate layer 206 within the same cluster (e.g., FM node 202-2-1 and FM node 202-2-M), and align and / or fuse the outputs. The final layer 208 can combine or fuse the outputs in ways such as summing the outputs, concatenating the outputs, and / or feeding the outputs as input to a trained linear layer to generate a fused latent representation. The FM function 116 can then feed the fused latent representation as input to one or more neural networks (e.g., a transformer encoder) to produce latent representations as outputs.

[0037] For example, FM function 116 can connect any FM node on layer 204 that processes telemetry data associated with computation (e.g., DCGM, system logs) to a first FM node on layer 206, and FM function 116 can individually connect any FM node on layer 204 that processes telemetry data associated with networking (e.g., CollectX, SM logs) to a different second FM node on layer 206. Alternatively, FM function 116 can first aggregate the FM nodes of layer 204 using the FM nodes of layer 204 according to the modal type (time series or log) of the data stream processed by the FM nodes of layer 204. Then, FM function 116 can aggregate the FM nodes of layer 206 using the FM nodes of adjacent higher layers (e.g., final layer 208) by the value of the variable “source” (e.g., “computation” or “networking”).

[0038] In some embodiments, when the structure of the tree-like FM 200 is determined based on learning, at least a portion of the structure of the tree-like FM 200 can be created based on grouping the FM nodes 202 according to Neural Architecture Search (NAS). For example, FM function 116 can specify the number of FM nodes included in each layer other than the first layer 204, and determine how to group (or connect) the FM nodes of adjacent lower layers using one or more NAS algorithms. The number of FM nodes in the first layer 204 can be determined at least in part by the number of one or more data streams 101.

[0039] As another non-limiting example, FM function 116 can use NAS to identify the architecture of each FM node 202 based on the task to be performed by the FM nodes. To form the first layer 204 of the tree-like FM 200, FM nodes with similar architectures and / or assigned similar tasks can be selected and arranged as the first layer 204. For example, as described above, the FM nodes of the first layer 204 can have a neural architecture configured to receive a single data stream type and output a latent representation of the data from that data stream. FM nodes of one or more intermediate layers (e.g., intermediate layer 206) can have a neural architecture configured to receive two or more latent representations of different data types, align and / or fuse these latent representations, and output a latent representation of the aligned and / or fused data. FM nodes of the final layer 208 can be determined by grouping the outputs of two or more FM nodes 202 from previous intermediate layers (e.g., intermediate layer 206) based on computational soft clustering. For example, they can be grouped according to more than one category (e.g., the categories discussed herein). Figure 3A The output is grouped (or clustered) in terms of one or more values ​​along the N, M and / or D dimensions, rather than in a single predetermined aspect (e.g., a value along one of the N, M and / or D dimensions).

[0040] To support high-frequency telemetry, the tree-structured FM 200 can be trained and / or run in a distributed manner. For example, leaf FMs can be trained locally at the device level, while other FM nodes (e.g., nodes at the intermediate layer 206 and the final layer 208) can be located at the data center level.

[0041] The tree-based FM 200 can be trained as a whole without first pre-training the FM nodes 202, and / or can be trained after pre-training the FM nodes 202. Supervised and / or unsupervised learning can be used to train the tree-based FM 200 as a whole. For example, if the final layer 208 of the tree-based FM 200 includes a task-specific head (e.g., a classifier) ​​for generating the FM tree output 210 using the latent representation generated by the final layer 208, the tree-based FM 200 can be trained as a whole using a training dataset that includes the input dataset (e.g., one or more data streams 101) and their corresponding ground truth data (e.g., classification results), without first pre-training the individual FM nodes 202. Using such a training method, the tree-based FM can be trained to perform the objective at the lowest resolution level (e.g., at the data center level or floor level) instead of training each FM node 202 separately. As another non-limiting example, one or more layers of the tree-based FM 200 may include task-specific heads. In such embodiments, the latent representation generated by one or more such layers can be provided to an appropriate task-specific head, and the portion of the tree FM 200 terminating at that task-specific head can be trained using a training dataset comprising the input dataset (e.g., input data corresponding to the portion of the tree FM 200 terminating at that task-specific head) and its corresponding ground truth data. Using such a training method, the tree FM 200 can be trained to perform an objective at one or more resolution levels above the lowest level (e.g., the server room level of data center 100). Furthermore, one or more FM nodes 202 of the tree FM 200 can be trained individually. Regarding unsupervised learning, any unsupervised learning method can be used to train the tree FM 200, such as contrastive learning, autoregressive learning, adversarial learning, generative learning, and / or one or more other unsupervised learning techniques.

[0042] Figure 2B A block diagram illustrating a plurality of tree-like FMs arranged to generate output 230 is provided according to at least one embodiment. Each tree-like FM 220 can be constructed as described herein. For example, tree-like FM 200 (see Figure 2A This can be used to implement one or more tree-structured FM 220s. Figure 2BThe diagram illustrates a tree-structured FM 220 arranged to define three layers 222, 224, and 226. The tree-structured FM 220 can be arranged to define any number of layers. For example, the second layer 224 can be omitted, and the output of the first layer 222 can be directly (as input) provided to the third (final) layer 226. As another non-limiting example, the second layer 224 can be characterized as an intermediate layer, and two or more intermediate layers can be located between the first layer 222 and the third layer 226. In this arrangement, the first intermediate layer receives the output of the first layer 222 as input and provides its output to another intermediate layer. Furthermore, in this arrangement, the last intermediate layer receives the output of one of the intermediate layers as input and provides its output to the third layer 226. Any intermediate layer between the first and last intermediate layers receives the output of one of the intermediate layers as input and provides its output to another intermediate layer. In embodiments including only the first and last intermediate layers, the output of the first intermediate layer is provided as input to the last intermediate layer.

[0043] The base layer or first layer 222 is illustrated as comprising tree FMs 222-1 to 222-E. The variable “E” can take any integer value greater than 1. Although illustrated as different tree FMs, tree FMs 222-1 to 222-E can be implemented using a single tree FM, for example, by processing different data (e.g., serially). Each tree FM on the first layer 222 can have one or more leaf FMs, each leaf FM receiving a data stream (e.g., telemetry data, log data, etc.). The output of tree FMs 222-1 to 222-E can be a latent representation (e.g., a sequence of vectors) or other values ​​as described herein.

[0044] Although the second layer 224 is illustrated as including tree FMs 224-1 to 224-F, the second layer 224 can include any number of tree FMs, including a single tree FM. Although illustrated as different tree FMs, tree FMs 224-1 to 224-F can be implemented using a single tree FM, for example, by processing different data (e.g., serially). At least some of the tree FMs in the second layer 224 can receive the output of the first layer 222 as input. For example, in Figure 2B In the diagram, tree FM 224-1 is illustrated as receiving the outputs of tree FM 222-1 and 222-2. The outputs of tree FM 222-1 and 222-2 can be latent representations (e.g., sequences of vectors) or other values ​​as described herein.

[0045] Although the third layer 226 is illustrated as consisting only of tree FMs 226-1, the third layer 26 can include any number of tree FMs. At least some of the tree FMs on the third layer 226 can receive the output of the second layer 224 as input. For example, in Figure 2B In the diagram, the tree FM 226-1 is illustrated as receiving the outputs of the tree FM 224-1 and 224-F. The output of the tree FM 226-1 can be a latent representation (e.g., a sequence of vectors), or other values ​​as described herein.

[0046] Structures defined by tree-like FM (such as hierarchical structures, graphs, etc.) can be used to create outputs with different resolutions. For example, a first layer 222 can output a first resolution (e.g., a potential representation per device, such as DGX), a second layer 224 can output a second resolution lower than the first resolution (e.g., a potential representation per group of devices, such as devices within a specific physical location), and a third layer 226 can output a third resolution lower than the second resolution (e.g., a potential representation of a set of per group of devices, such as groups of devices within a specific physical location).

[0047] Each tree FM on layer 222 can have one or more leaf FMs, for example, receive data streams from different devices. For example, each tree FM on layer 222 can have leaf FMs that receive telemetry data streams associated with different GPUs in a particular DGX. In this example, the output of the tree FM on layer 222 can be a latent representation of telemetry data associated with GPUs in different DGXs. In other words, each tree FM on layer 222 can output a latent representation corresponding to different DGXs (e.g., encoding the telemetry data associated with the GPUs in the DGX). Each tree FM on layer 224 can receive the output of any tree FM on layer 222 that corresponds to a DGX physically located in a particular data center. In such an example, each tree FM on layer 224 can output a latent representation corresponding to different data centers in data center 100. Continuing this example, each tree FM on layer 226 can receive the output of any tree FM on layer 226 that corresponds to a data center physically located on a particular floor. In such an example, each tree FM on the third layer 226 can output a latent representation corresponding to a different floor in the data center 100. As another non-limiting example, tree FM 226-1 on the third layer 226 can receive the outputs of all tree FMs on the second layer 226 and can output a latent representation corresponding to the data center 100. The output 230 of the third layer 226 may include this latent representation and / or one or more other values ​​(such as those described herein), which are generated at least in part based on the latent representation produced by tree FM 226-1. Therefore, tree FM 220 can be used to produce outputs at different resolutions (e.g., at the device (e.g., DGX) level, server room level, floor level, and / or data center level). Furthermore, the outputs of tree FM 220 can be combined in different ways to produce one or more outputs at different resolutions. As another non-limiting example, tree FM 220 can be used to construct one or more different structures (e.g., one or more hierarchical structures, one or more graphs, etc.) that can produce one or more different outputs, and / or one or more outputs at different resolutions.

[0048] Figure 3A A block diagram illustrating an example data structure 300 according to at least one embodiment is shown. The data structure 300 (e.g., a tensor) may be used by one or more processors (e.g., at least one of one or more processors 105) and / or stored in memory (e.g., storage device 110 and / or memory 112). The data structure 300 (e.g., a tensor) stores input data 302, such as samples collected from a specific device (e.g., one of one or more servers 104). The input data 302 can be obtained from... Figure 1 One or more data streams 101 are obtained.

[0049] return Figure 3A The illustrated data structure 300 includes three dimensions: the D dimension represents internal components, referred to as "instances" (e.g., GPUs within a deep learning system); the M dimension represents different variables; and the N dimension represents time steps. Data structure 300 can be transformed so that at least one of the one or more GPUs 108 (e.g., one or more tensor kernels of the one or more GPUs 108) can perform forward passes along the D (instance) dimension, the M (variable) dimension, or the N (time) dimension. For example, a forward pass along the N dimension (called a time pass) can include D×M sequences of length N. A forward pass along the M dimension (called a variable pass) can include D×N sequences of length M. A forward pass along the D dimension (called an instance pass) can include M×N sequences of length D.

[0050] Figure 3B A block diagram illustrating an inference process 310 performed by a leaf FM in a tree-like FM 200 according to at least one embodiment is shown. The inference process 310 may be performed by a data center 100, a computing device, and / or one or more other computing systems. An FM function 116 (e.g., if performed by one or more processors 105) may perform the inference process 310. For example, the FM function 116 may implement a leaf FM, which may perform at least a portion of the inference process 310.

[0051] Inference process 310 begins with FM function 116 providing raw data 312 to tokenization process 314 (e.g., implemented by FM function 116). Raw data 312 may include multivariate data, multi-source data, multimodal data, and / or multi-component data. For ease of illustration, raw data 312 will be described as being transmitted as a data stream. However, raw data 312 can be in any form, including being stored and read from a file. Raw data 312 may be provided to tokenization process 314 in the form stored in data structure 300, or tokenization process 314 may store the data in data structure 300. Tokenization process 314 may divide the input data 302 stored in data structure 300 into segments (e.g., blocks) and flatten these segments into one-dimensional vectors (e.g., tokens). For example, when performing time-passing, tokenization process 314 may divide each D×M sequence (e.g., a matrix) into multiple blocks. If data structure 300 stores raw time-series data, then lexicalization process 314 can split a portion of input data 302 (e.g., each of a set of matrices) into multiple overlapping or non-overlapping windows (or blocks) (e.g., as used by PatchTST). If data structure 300 stores log messages, then lexicalization process 314 can split a portion of input data 302 (e.g., each of a set of matrices) into log messages, where each log message is a fragment or term (e.g., a vector). These terms (e.g., vectors) can be arranged serially, referred to as input vector sequence 316.

[0052] In at least one embodiment, FM function 116 provides an input vector sequence 316 to a latent sequence generation process 322. The latent sequence generation process 322 encodes the input sequence 316 to obtain a latent sequence 324. The latent sequence generation process 322 may include: normalizing the input sequence 316 and projecting the normalized sequence using a learned linear layer to obtain the latent sequence 324. If the input sequence 316 is encoded as a time series, the latent sequence generation process 322 normalizes the input sequence 316 (e.g., using one or more RevIn operations) and projects the input sequence 316 using a learned linear layer. If the input sequence 316 is encoded as a log (e.g., a system log), the latent sequence generation process 322 may encode the input sequence 316 and / or one or more templates using a learned layer or a pre-trained encoder (e.g., SBERT from the SentenceTransformer library).

[0053] In at least one embodiment, FM function 116 provides a latent vector sequence 324 to a neural network 326, which outputs a latent representation 328. The neural network 326 may be implemented as a transformer encoder, a language model (e.g., a large language model (LLM), a small language model (SLM), a visual language model (VLM), a multimodal language model (MMLM), etc.), a visual-language-action (VLA) model, a recurrent neural network (RNN), a long short-term memory network (LSTM), a gated recurrent unit (GRU), a convolutional neural network for sequences (CNN), a memory-enhanced neural network, a non-transformer-based attention mechanism, a graph neural network (GNN), an echo state network (ESN), pooling computation, a self-attention mechanism, a bidirectional RNN, a bidirectional LSTM, a bidirectional GRU, a temporal convolutional network (TCN), a WaveNet, a capsule network, and / or other types of machine learning processes. FM function 116 can provide a potential representation 328 (e.g., a sequence of vectors) to one or more other FM nodes 202 of tree FM 200, to one or more FM nodes of another tree FM (e.g., similar to tree FM 200), and / or to one or more other processes (e.g., machine learning processes, AI, one or more task-specific heads, etc.).

[0054] Figure 3C The illustration shows a tree-like FM 200 according to at least one embodiment (see...) Figure 2A A block diagram of inference process 330 performed by an FM node (which may be different from a leaf FM) in a data center 100. Inference process 330 may be performed by data center 100, computing devices, and / or one or more other computing systems. FM function 116 (e.g., if performed by one or more processors 105) may perform inference process 310. For example, FM function 116 may implement an FM node that can perform at least a portion of inference process 330. FM node may be located in an intermediate layer (e.g., intermediate layer 206) or a final layer 208. In at least one embodiment, FM node may be located in a first layer 204, but may receive one or more potential representations (e.g., from different tree FMs) instead of other types of data (e.g., data streams or data files).

[0055] The inference process 330 begins with FM function 116 receiving one or more latent representations 332 (e.g., each latent representation is similar to latent representation 328). Each of the one or more latent representations 332 may be stored in data structure 300, or one or more GPUs 108 (e.g., performing FM function 116) may store the one or more latent representations 332 in data structure 300.

[0056] In at least one embodiment, FM function 116 provides one or more latent representations 332 to a latent sequence generation process 338 (e.g., similar to latent sequence generation process 322), which outputs a latent vector sequence 350. The latent sequence generation process 338 may include an alignment process 340 that outputs an aligned vector sequence 342 for each of the one or more latent representations 332. The alignment process 340 may bridge gaps between data generated at different frequencies (high and low frequencies) by aligning one or more latent representations 332 by timestamps and performing interpolation (e.g., when aligning low-frequency data with high-frequency data) or pooling (e.g., when aligning high-frequency data with low-frequency data). As a non-limiting example, the data collected by DCGM may include multivariate time-series data, such as counter measurements collected over time for several hosts (e.g., DGX), and kernel system logs may include semi-structured data (logs) collected from one or more kernel applications executed from a host (e.g., DGX). For DCGM data and kernel system logs, aligned samples can be obtained by DGX id and time period. <t i , t i+1 The term is defined by >. For example, if the time period is 15 minutes, then t i+1 -t i =15 minutes. If the DCGM data and kernel system log correspond to the same time period and the same DGX identified by the DGX id, then the DCGM data and kernel system log are perfectly aligned. Use the time period (determined by t...). i and t i+1 The alignment process 340 can achieve the alignment of DCGM data with the kernel system log in the following way: extract logs from the system log table (indexed by host identifier DGX id) and extract DCGM data that corresponds to the time period and the DGX id.

[0057] As a non-limiting example, a first device (e.g., a GPU) transmits first telemetry data at a first frequency (e.g., every 30 seconds), and a second device (e.g., a network NIC) transmits second telemetry data at a different second frequency (e.g., every 5 minutes). Both the first telemetry data from the first device and the second telemetry data from the second device comprise multivariate time-series data. Alignment process 340 can achieve alignment between the first telemetry data from the first device and the second telemetry data from the second device by using a higher first frequency as the sampling rate and adding another feature vector that indicates whether a corresponding sample exists in the second telemetry data at each specific time point (e.g., a timestamp) of the first telemetry data. For example, this other feature vector could store the offset between each time point of the first telemetry data (e.g., a timestamp corresponding to the moment each data point in the first telemetry data is received) and the exact time point (e.g., a timestamp) of each data point in the second telemetry data being received. The offset vector could store values ​​normalized to the range [-0.5, 0.5]. Within this range, a value of zero indicates that a data point in the first telemetry data and a data point in the second telemetry data correspond to the same time point (e.g., a timestamp), meaning the data points were received at the same time or their times overlapped. Within this range, a value of -0.5 indicates that the arrival time of a data point in the second telemetry data is exactly midway between a specific time point (e.g., a timestamp) associated with a specific data point in the first telemetry data and a previous time point (e.g., a timestamp) associated with a previous data point in the first telemetry data. Similarly, within this range, a value of 0.5 indicates that the arrival time of a data point in the second telemetry data is exactly midway between a specific time point (e.g., a timestamp) associated with a specific data point in the first telemetry data and a next time point (e.g., a timestamp) associated with the next data point in the first telemetry data.

[0058] The latent sequence generation process 338 may provide one or more aligned vector sequences 342 to the fusion process 344, which outputs a fused vector sequence 346. The fusion process 344 may merge (e.g., concatenate) the one or more aligned vector sequences 342 obtained from other FMs of the same or different tree FMs. The fusion process 344 may merge and / or fuse the one or more aligned sequences 342 by, for example, summing the one or more aligned sequences 342, concatenating the one or more aligned sequences 342, and / or providing the one or more aligned sequences 342 as input to a trained linear layer to generate the fused vector sequence 346.

[0059] The latent sequence generation process 338 can provide the fused vector sequence 346 to the linear layer 348, which outputs a latent vector sequence 350. In some embodiments, the linear layer 348 can be omitted, and the latent sequence generation process 338 can output the fused sequence 346 as the latent sequence 350.

[0060] In at least one embodiment, FM function 116 provides a latent vector sequence 350 to a neural network 352 (e.g., similar to neural network 326), which outputs a latent representation 354. In some embodiments, neural network 352 may be implemented as at least one transformer encoder, at least one hybrid encoder, at least one decoder, at least one state-space model (SSM), and / or may use other types of models. FM function 116 may provide the latent representation 354 (e.g., a vector sequence) to one or more other FM nodes 202 of tree-structured FM 200, to one or more FM nodes of another tree-structured FM (e.g., similar to tree-structured FM 200), and / or to one or more other processes (e.g., machine learning processes, AI, one or more task-specific heads, etc.).

[0061] The tree-structured FM 200 can be trained to perform inference using one or more data streams 101 (e.g., collected by data center 100), which may include high-frequency telemetry data. Constructing the tree-structured FM 200 using FM nodes 202 can be characterized as providing an interpretable framework for the underlying model of the data center (e.g., data center 100). The tree-structured FM 200 can be evaluated using evaluation frameworks, such as one or more evaluation frameworks described herein for evaluating the underlying model of a data center. FM nodes 202 can be organized within a hierarchical FM (e.g., the tree-structured FM 200) to represent the topology of data center 100.

[0062] Figure 4 A block diagram depicting a training process 400 according to at least one embodiment is illustrated. One or more computing systems (e.g., one or more servers 104, or another computing system) can use the training process 400 to train a tree-structured FM 200 (see...). Figure 2A One or more FM nodes 202 (see) Figure 2A For example, FM function 116, when executed by one or more processors 105, can perform training process 400. For example, training process 400 can be used to train leaf FM or other FM nodes.

[0063] refer to Figure 3BEach FM node 202 of the tree-like FM 200 may include a neural network backbone 402 (e.g., neural network 326 or neural network 352), which may be implemented as a transformer encoder with K layers. The transformer encoder with K layers may include K identical layers, each performing a series of operations to transform the input data into a more abstract representation, capturing complex patterns and / or dependencies. Each of the K layers may include a multi-head self-attention process that simultaneously adjusts the weights or attention scores associated with different parts of the input sequence (e.g., blocks or segments obtained from it, referred to as lexical units) to attract attention to some parts rather than others. The multi-head self-attention process may compute an attention score for each lexical unit relative to every other lexical unit, enabling the transformer encoder to capture relationships and / or dependencies between lexical units. Each of the K layers may include a feedforward neural network that receives the output from the self-attention mechanism, performs a first linear transformation on the output, performs a nonlinear activation function on the output of the first linear transformation, and performs a second linear transformation on the output of the nonlinear activation function. Next, each of the K layers can perform layer normalization to normalize the output of the feedforward neural network. Each of the K layers can include one or more residual connections that bypass the multi-head self-attention process and / or the feedforward neural network of that layer to allow the transformer encoder to retain information from earlier layers and / or improve gradient flow during training.

[0064] exist Figure 4 In the embodiment illustrated, one or more FM nodes 202 of the tree-like FM 200 are trained to perform masked lexical prediction, which can be considered the objective of this training. However, this is not necessary, and in other embodiments, the FM can be trained to achieve other objectives. For example, one or more FM nodes can be trained to predict the next lexical. In at least one embodiment, as an alternative or addition to a transformer encoder, the neural network backbone 402 of one or more FM nodes 202 of the tree-like FM 200 may include one or more decoders that perform the next lexical prediction. In some embodiments, as an alternative or addition to a transformer encoder, the neural network backbone 402 of one or more FM nodes 202 of the tree-like FM 200 may include one or more diffusion models to achieve one or more different objectives.

[0065] When the training process 400 is executed by one or more processors 105, the input data (e.g.) Figure 3A The input data 302 illustrated can be encoded into a latent vector sequence 406. For example, see... Figure 3AData structure 300 can store input data (e.g., from...) Figure 1 (obtained from one or more data streams 101 as illustrated), one or more GPUs 108 (see...) Figure 1 The input data can be divided into segments (e.g., blocks) and flattened into an input vector sequence 404 (e.g., input sequence 316 or one or more latent representations 332). In this way, before encoding the input data, it can be split into two or more segments that can be flattened to produce two or more tokens, which can be arranged to define the input vector sequence 404. As a non-limiting example, if the input data includes logs, each log message can be a segment or token (e.g., a vector), and the tokens can be arranged to define the input vector sequence 404. For illustration purposes, only a single vector input sequence 404 is illustrated, but the one or more GPUs 108 can generate two or more vector input sequences.

[0066] Subsequently, FM function 116 can perform a masking process 408 that masks the input vector sequence 404 (e.g., using one or more masks) to produce a masked vector sequence 410. For example, one or more randomly selected terms (e.g., vectors of the input sequence 404) can be masked to form the masked sequence 410. For illustration purposes, only a single masked sequence 410 is illustrated, but this masking process can be used to generate two or more masked sequences. FM function 116 can obtain a latent sequence 406 by performing a latent sequence generation process 412 (e.g., latent sequence generation process 322 or latent sequence generation process 338) using the masked sequence 410. The latent sequence generation process 412 may include normalizing the masked sequence 410 and projecting the normalized sequence using a learned linear layer.

[0067] In at least one embodiment, the masking process 408 may be omitted. In such embodiments, the FM function 116 can obtain the latent sequence 406 by performing a latent sequence generation process 412 (e.g., latent sequence generation process 322 or latent sequence generation process 338) using the input vector sequence 404 (rather than the masked sequence 410). The latent sequence generation process 412 may normalize the input sequence 404 and project the normalized sequence using a learned linear layer. In embodiments where the input sequence 404 includes one or more latent representations output by FM nodes of tree FM 200 or another tree FM, the latent sequence generation process 412 may perform an alignment process (e.g., similar to alignment process 340), a fusion process (e.g., similar to fusion process 344), and / or a linear layer (e.g., similar to linear layer 348).

[0068] The latent vector sequence 406 is input into the neural network backbone 402, which may be, for example, a transformer encoder with K layers. Each layer may operate on each dimension encoded by the latent sequence 406 (e.g., the N (time) dimension, the M (variable) dimension of a multivariate time series, and / or the D (instance) dimension when the data is associated with multiple sources (e.g., networking or computation).

[0069] The neural network backbone 402 can learn patterns from the latent sequence 406 and output a latent representation 414 (e.g., a vector sequence). The neural network backbone 402 can be trained using self-supervision (e.g., reconstruction, contrastive learning, autoregression, and / or one or more other learning objectives). For example, during training, each FM node 202 may include a reconstruction head 420 that uses the latent representation 414 to predict a reconstructed vector sequence 422. This prediction can be compared to a ground truth sequence, and the result of this comparison is used to update the neural network backbone 402 of the FM nodes (e.g., updating the parameters used in the neural network backbone 404, such as weights). The ground truth sequence refers to the sequence input to the latent sequence generation process 412 to obtain the latent sequence 406 (e.g., a masked sequence 410 or the original input sequence 404).

[0070] The reconstruction head 420 may include one or more layers for transforming the latent representation 414 into a reconstructed vector sequence 422. These layers may include one or more dense layers (e.g., one or more fully connected layers) that map the latent representation 414 to a higher-dimensional space that matches the dimension of the ground truth sequence (e.g., the masked sequence 410 or the original input sequence 404). The dense layers may apply a linear transformation followed by a non-linear activation function. Non-limiting examples of activation functions include Rectified Linear Unit (ReLU), sigmoid, and / or tanh. The reconstruction head 420 may include one or more recurrent layers for capturing temporal dependencies and / or reconstructing sequences over time. Non-limiting examples of recurrent layers include Long Short-Term Memory (LSTM) or Gated Recurrent Unit (GRU). The reconstruction head 420 may include one or more convolutional layers, which may be used for image data and / or time-series data. The one or more convolutional layers may be used to reconstruct spatial features and / or upsample the latent representation 414 to match the dimension of the ground truth sequence (e.g., the masked sequence 410 or the original input sequence 404). The one or more convolutional layers (e.g., one or more one-dimensional convolutional layers) may be used to capture local patterns in time-series data and / or help reconstruct features that expand over time. If the latent representation 414 is smaller than the ground truth sequence (which is common in image processing), the one or more layers of the reconstruction head 420 may include one or more upsampling layers. Non-limiting examples of upsampling layers include transposed convolution or nearest-neighbor upsampling, which can be used to increase the spatial dimension. The one or more upsampling layers may perform interpolation and / or transposed convolution. The one or more layers of the reconstruction head 420 may include one or more output layers that match the dimension of the input data. For continuous data, the one or more output layers may include linear layers, while for categorical data, the one or more output layers may include softmax. The one or more layers of the reconstructed head 420 may include one or more normalization layers, such as batch normalization or layer normalization, which help stabilize training and / or improve convergence.

[0071] Loss functions, such as mean squared error (MSE) for continuous data or cross-entropy loss for categorical data, can be used to evaluate the quality of the reconstruction. For example, MSE can be used with time series data (e.g., for reconstructing time series data). As another non-limiting example, cross-entropy loss can be used to classify lexical units within a vocabulary (e.g., classifying lexical units that represent words, phrases, or parts of words). The neural network backbone 402 can be trained to minimize the loss value calculated by the loss function. Since training the FM node does not require labeled data, the FM node can be characterized as being trained using a self-supervised learning approach.

[0072] In at least one embodiment, as an alternative or addition to training FM nodes to predict ground truth sequences (e.g., masked sequence 410 or original input sequence 404), FM nodes may also be trained using contrastive objectives, adversarial objectives, and / or generative objectives. For example, the original input data (e.g., input sequence 404) input into the leaf FM may be augmented to generate positive samples and / or negative samples of latent representations for a contrastive objective, which may include making the latent representations generated by the FM nodes closer to the positive samples and further away from the negative samples.

[0073] When FM node 202 performs one or more downstream tasks (e.g., classification, regression, anomaly detection, etc.), the reconstruction head 420 of each FM node 202 can be omitted or replaced with one or more task-specific heads (e.g., heads trained to perform classification, regression, anomaly detection, etc.) used to fine-tune the FM node to perform one or more downstream tasks. For example, the final layer 208 can pool the latent representation 414 output by more than one FM node, and the FM function 116 can be trained with a dedicated classification head in a supervised manner. As another non-limiting example, the learned latent representation can be pooled (e.g., using mean pooling or taking the last element) for zero-shot and few-shot tasks.

[0074] For any FM node 202 that omits reconstruction head 420, its output may be latent representation 414. In such embodiments, the latent representation generated by an FM node at a certain layer is provided as input to child FM nodes at adjacent layers. For any FM node 202 whose reconstruction head 420 is replaced by one or more task-specific heads, its output may be the output of that one or more task-specific heads. For example, the reconstruction head 420 of the FM nodes at the first layer 204 and intermediate layers 206 may be omitted, and the reconstruction head 420 of the FM nodes at the final layer 208 may be replaced with one or more task-specific heads. In such embodiments, the FM tree output 210 of the tree-like FM 200 may include the output of that one or more task-specific heads, such as one or more classifications (e.g., by token, by latent representation, etc.), one or more regressions, identification of one or more anomalies in the latent representations 414 output by each FM node at the final layer 208, etc.

[0075] The attention mechanism of the neural network backbone 402 improves the interpretability of one or more data streams 101 by providing a way to visualize the relative importance of inputs at different levels. The tree-structured FM 200 can be used to perform simulations by augmenting the inputs (e.g., one or more data streams 101) and observing the responses in the predicted values ​​(e.g., latent representation 414, reconstructed sequence 422, and / or one or more values ​​output by one or more task-specific heads).

[0076] After FM nodes 202 are trained (e.g., using training process 400), they can be assembled into a tree-structured FM 200 and / or another FM with a hierarchical structure to perform one or more tasks. These tasks may include generating a latent representation (e.g., a sequence of vectors) as an FM tree output 210 from data collected from one or more devices in data center 100. This latent representation can be generated by omitting the reconstruction head 420 from FM nodes 202 and constructing a tree-structured FM 200 using the FM nodes. Subsequently, FM function 116 can input one or more data streams 101 into the tree-structured FM 200, which uses these data streams to generate the FM tree output 210 (which may be a latent representation). These data streams 101 may include diverse streams of high-frequency time-series and semi-structured data (logs). Such streams can transmit and include large amounts of data. The one or more data streams 101 may be output from one or more devices (e.g., included in devices in data center 100, and / or devices outside data center 100), or otherwise collected for the one or more devices.

[0077] FM node 202 enables data-driven decision-making by providing FM tree output 210 (e.g., a latent representation) to one or more machine learning processes (such as one or more task-specific heads, such as heads trained to perform classification, regression, anomaly detection, etc.). The data included in one or more data streams 101 for generating FM tree output 210 may include multivariate data from multiple sources, data with different modalities, data from one or more devices, and / or data that differs in other ways. FM function 116 can transfer FM tree output 210 (e.g., a latent representation) to one or more downstream processes (e.g., one or more machine learning processes, such as one or more neural networks) and use FM tree output 210 to perform one or more downstream tasks (e.g., prediction and anomaly detection). FM function 116 can fine-tune trained FM node 202 (e.g., using linear probing, zero-shot performance, retrieval-based zero-shot performance, and / or other techniques). One or more task-specific heads can be used with one or more FM nodes 202 to enable the one or more FM nodes to be used with new and / or evolving technologies. FM function 116 can train FM nodes 202 (whether or not they have one or more task-specific heads) using sparse and / or noisy supervisory signals. FM function 116 can train tree-like FM 200 in a hierarchical manner, and / or tree-like FM 200 can learn hierarchically from one or more data streams 101 generated and / or received by data center 100. For example, FM function 116 can use the outputs of FM nodes on first layer 204 to train and / or fine-tune FM nodes on intermediate layer 206, and FM function 116 can use the outputs of FM nodes on intermediate layer 206 to train and / or fine-tune one or more FM nodes on final layer 208.

[0078] For example, DCGM data may include time-series multivariate data collected from multiple devices of a deep learning system (e.g., DGX). As a non-limiting example, each of these devices may include several GPUs (e.g., eight GPUs). FM function 116 may receive DCGM data collected from each GPU as one or more data streams 101. The modality of the data streams transmitting such DCGM data is time-series, and each of at least a portion of the data stream may be associated with a value of a variable “source.” Examples of such values ​​include “computation” and “networking,” which may mean that the data stream includes data related to computation or networking. As a non-limiting example, the value of the variable “source” may be stored in the M (variable) dimension. In its forward pass, FM node 202 may perform attention mechanisms on the N (time) dimension, M (variable) dimension, and D (instance, e.g., GPU) dimension in an interleaved or sequential manner by shaping or transforming the data structure 300 (e.g., tensor) as described herein. Each FM node (called a leaf FM) on the first layer 204 can receive a data stream that transmits DCGM data collected from the GPU of a device in the deep learning system.

[0079] The latent representation 414 learned by each leaf FM can be aligned over time (e.g., via alignment process 340) and aggregated by the devices of the deep learning system (across variable "source" values). In other words, each FM node of the intermediate layer 206 can receive the latent representation 414 output by its corresponding leaf FM—which receives data streams from the GPU outputs of the same devices in the deep learning system—and aggregate data from such data streams for that device. For example, if the deep learning system includes two devices (each with eight GPUs), the first FM node of the intermediate layer 206 will aggregate the latent representation 414 received from its corresponding leaf FM (which processes data streams from the eight GPUs of the first device), and the second FM node of the intermediate layer 206 will aggregate the latent representation 414 received from its corresponding leaf FM (which processes data streams from the eight GPUs of the second device). Such FM nodes of the intermediate layer 206 can learn some aspect of the data center 100, such as computing and networking. Finally, FM nodes at the upper levels of the tree-like FM (e.g., at the final layer 208) can utilize representations of adjacent devices of the deep learning system (e.g., two or more devices connected in the physical data center 100).

[0080] refer to Figure 2AA hierarchical FM (similar to tree-like FM 200) can be used to perform fault classification at the FM node hierarchy. For example, a lower layer (e.g., first layer 204) may receive DCGM data and system logs collected from computational operations performed by a host (e.g., a deep learning system), as well as additional system data (e.g., CollectX data) and system management logs (e.g., SMlog) from one or more systems (e.g., in a hybrid cube architecture (HCA)). The next layer (e.g., intermediate layer 206) may align and fuse the outputs from two or more FM nodes on the lower layer. The final layer (e.g., final layer 208) may align and fuse the outputs from two or more FM nodes on the second layer. FM function 116 can use the outputs from the second layer to fine-tune the classification linear layer on the final layer (e.g., final layer 208) to classify faults. In at least one embodiment, the classification linear layer may aggregate the outputs of the FM nodes on the second layer. In at least one embodiment, the final layer may include one or more FM nodes, and the classification linear layer may be implemented using a task-specific header. A classification linear layer can receive and classify the latent representations output by one or more FM nodes of the final layer. Hierarchical FMs (e.g., tree-based FM 200) can be used to perform prediction and / or anomaly detection. In this example, the classification linear layer can be replaced with a task-specific head that receives and processes the latent representations output by one or more FM nodes of the final layer to generate predictions and / or perform anomaly detection. Furthermore, zero-shot performance can be used. For example, for anomaly detection, if the input reconstruction (e.g., Figure 4 If the reconstructed sequence 422 shown in the figure deviates from the actual observed value (e.g., the masked sequence 410 or the input sequence 404), indicating a deviation from expected and / or normal behavior, then hierarchical FM can detect the anomaly.

[0081] Hierarchical FM (similar to tree-structured FM 200) can be used to perform general monitoring of cluster health. In such embodiments, the final layer (e.g., final layer 208) may include one or more FM nodes that output one or more potential representations to a task-specific header (such as a classification header), which is fine-tuned to perform this task.

[0082] Hierarchical FM (similar to tree FM 200) can be used to fuse intra-device data streams and / or inter-device data streams. For example, in the case of a failure involving multiple devices, FM function 116 can use hierarchical FM to fuse inter-device data streams collected for multiple devices to perform troubleshooting and / or failure prediction. An example of such a failure is a switch failure. Hierarchical FM can receive data streams carrying telemetry data associated with nodes connected to the switch, fuse data from the switch and / or nodes connected to the switch, and classify the failure as related to the switch (e.g., using a classification linear layer as described herein). When multimodal data (e.g., telemetry data and / or logs) exists within the same device, hierarchical FM (e.g., tree FM 200) can be used to fuse intra-device data streams. For example, the device can fuse intra-device data streams to generate a latent representation that can be used (e.g., by another hierarchical FM) to perform inter-device fusion.

[0083] Figure 5 The illustrations of at least one embodiment illustrate the use of hierarchical FM (e.g. Figure 2A The flowchart illustrates method 500 (tree-structured FM 200). Method 500 can be executed by data center 100, computing devices, and / or one or more other computing systems. FM function 116 (e.g., if executed by one or more processors 105) can execute method 500. For example, FM function 116 can implement tree-structured FM 200, which can execute at least a portion of method 500. Before executing method 500, data (e.g., one or more data streams 101) is collected and provided to the hierarchical FM. This data may include multivariate data, multi-source data, multimodal data, and / or multi-component data. For ease of illustration, the data will be described as being transmitted as data streams. However, the data can be in any form, including being stored in and read from files.

[0084] At box 502, one or more leaf FMs of the first layer (e.g., first layer 204) of the hierarchical FM (e.g., tree FM 200) process one or more data streams, and each leaf FM produces a latent representation (e.g., latent representation 414). For example, each of the one or more leaf FMs can process a different data stream and output a different latent representation corresponding to that data stream. To process the data streams, one of the one or more leaf FMs can store a portion of the data stream in a data structure (e.g., data structure 300). One or more GPUs (e.g., Figure 1One or more GPUs (108) illustrated in the diagram can divide data stored in a data structure into fragments (e.g., blocks) and flatten these fragments into a sequence of input vectors (e.g., input sequence 404). As another non-limiting example, if the input data includes logs, each log message can be a fragment or a term (e.g., a vector), and the term can be arranged to define the sequence of input vectors (e.g., input sequence 404). Subsequently, one or more GPUs of LeafFM can perform a masking process 408 to obtain a masked sequence of vectors (e.g., masked sequence 410). Next, one or more GPUs of LeafFM can perform a latent sequence generation process 412 to obtain a sequence of latent vectors (e.g., latent sequence 406). One or more GPUs of LeafFM implement a neural network backbone (e.g., neural network backbone 402). One or more GPUs of LeafFM provide the latent vector sequence (e.g., latent sequence 406) as input to the neural network backbone, and the neural network backbone outputs a latent representation (e.g., latent representation 414). For example, at box 506, each of the leaf FMs in one or more leaf FMs of the first layer can perform inference process 310.

[0085] At decision box 504, if the hierarchical FM (e.g., tree FM 200) includes one or more intermediate layers (e.g., intermediate layer 206), the decision at decision box 504 is "yes". Otherwise, the decision at decision box 504 is "no". When the decision at decision box 504 is "yes", FM function 116 provides the latent representation of the leaf FM output in box 502 as input to the first intermediate layer of one or more intermediate layers and proceeds to box 506. Otherwise, if the decision at decision box 504 is "no", FM function 116 provides the latent representation of the output of one or more FM nodes in box 502 to the final layer (e.g., final layer 208) and proceeds to box 510.

[0086] At box 506, FM function 116 uses one or more FM nodes from a specific intermediate layer (e.g., intermediate layer 206) of one or more intermediate layers of a hierarchical FM (e.g., tree FM 200) to process one or more latent representations input to the FM nodes of that specific intermediate layer. Each FM node of that specific intermediate layer may receive two or more latent representations as input from two or more distinct adjacent (e.g., lower-level) FM nodes, align these input latent representations, fuse or otherwise combine these aligned latent representations, and output an intermediate latent representation. For example, at box 506, each FM node of that specific intermediate layer may perform inference process 330.

[0087] At decision box 508, if the hierarchical FM (e.g., tree FM 200) includes one or more additional (e.g., higher-level) intermediate layers, the decision result at decision box 508 is "yes"; otherwise, the decision result at decision box 508 is "no". When the decision result at decision box 508 is "yes", FM function 116 provides one or more latent representations output by one or more FM nodes in box 506 as input to the next intermediate layer of that one or more intermediate layers and returns to box 506. Otherwise, if the decision result at decision box 508 is "no", FM function 116 provides one or more latent representations output by one or more FM nodes in box 506 to the final layer (e.g., final layer 208) and proceeds to box 510.

[0088] At box 510, FM function 116 uses one or more FM nodes of the final layer (e.g., final layer 208) of a hierarchical FM (e.g., tree FM 200) to process one or more latent representations input to one or more FM nodes in the final layer. Each FM node in the final layer can receive two or more latent representations as input from two or more distinct adjacent (e.g., lower-layer) FM nodes, align these input latent representations, fuse or merge these aligned latent representations, and output one or more final latent representations. For example, at box 510, each FM node in the final layer can perform inference process 330.

[0089] At box 512, FM function 116 and / or another process (e.g., one or more machine learning processes and / or one or more AIs) can use one or more final latent representations generated in box 510 to perform one or more tasks. For example, FM function 116 and / or another process can provide one or more final latent representations generated in box 510 to one or more neural networks trained to perform inference using those representations. Such neural networks can analyze the final latent representations encoding multivariate data to identify patterns, correlations, trends, and / or other information present in at least a portion of the data stream. As an additional, non-limiting example, the neural networks can use the final latent representations encoded with multimodal data to infer information related to complex phenomena and / or complex situations. FM function 116 and / or another process can provide the final latent representations to one or more machine learning processes (such as one or more neural networks) to perform classification, regression, anomaly detection, and / or one or more other tasks. Method 500 can terminate after box 512.

[0090] Figure 6 The illustrations, based on at least one embodiment, illustrate what can be used to train a hierarchical FM (e.g., Figure 2A The flowchart illustrates a training method 600 for a leaf FM (in the tree-like FM 200) shown in the figure. Training method 600 can be performed by data center 100, computing devices, and / or one or more other computing systems. FM function 116 (e.g., if performed by one or more processors 105) can execute training method 600. For example, FM function 116 can implement one or more FM nodes, each of which can be trained using training method 600. Before executing method 500, data (e.g., one of one or more data streams 101) is collected and provided to the leaf FM. This data may include multivariate data, multi-source data, multimodal data, and / or multi-component data. For ease of illustration, this data will be described as being transmitted as a data stream. However, the data can be in any form, including being stored in and read from files.

[0091] At block 602, FM function 116 causes one or more processors (e.g., one or more GPUs 108) to generate an input vector sequence (e.g., input vector sequence 404) at least partially based on the input data stream. In at least one embodiment, at block 602, FM function 116 causes one or more processors (e.g., one or more GPUs 108) to perform lexicalization process 314.

[0092] At box 604, FM function 116 causes one or more processors (e.g., one or more GPUs 108) to perform masking process 408 to produce a masked vector sequence (e.g., masked vector sequence 410) based at least in part on the input vector sequence obtained at box 602.

[0093] At block 606, FM function 116 causes one or more processors (e.g., one or more GPUs 108) to perform latent sequence generation process 412 to generate a latent vector sequence (e.g., latent vector sequence 406) based at least in part on the masked vector sequence obtained at block 604. In at least one embodiment, block 604 is omitted. In such embodiments, FM function 116 causes one or more processors (e.g., one or more GPUs 108) to perform latent sequence generation process 412 to generate a latent vector sequence (e.g., latent vector sequence 406) based at least in part on the input vector sequence generated in block 602.

[0094] At box 608, FM function 116 enables one or more processors (e.g., one or more GPUs 108) to provide the latent vector sequence obtained at box 606 to a neural network encoder (e.g., neural network backbone 402), which generates a latent representation (e.g., a new latent vector sequence) based at least in part on the latent vector sequence obtained at box 606.

[0095] At box 610, FM function 116 enables one or more processors (e.g., one or more GPUs 108) to provide the latent representation (e.g., latent representation 414) obtained at box 608 to reconstruction head 420, which makes predictions on the reconstructed vector sequence (e.g., reconstructed sequence 422).

[0096] At block 612, FM function 116 calculates a loss value (using a loss function) based at least in part on the comparison between the reconstructed vector sequence and the ground truth sequence. The ground truth sequence is the sequence input to latent sequence generation process 412 and thus used to generate a latent vector sequence at block 606. In embodiments including block 604, the ground truth sequence is the masked sequence output from block 604. Otherwise, the ground truth sequence is the input vector sequence output from block 602 (e.g., input sequence 404).

[0097] At decision box 614, FM function 116 determines whether enough training iterations have been performed. When FM function 116 has completed performing training iterations, the decision at decision box 614 is "yes". Otherwise, the decision at decision box 614 is "no". When the decision at decision box 614 is "no", at box 616, FM function 116 modifies one or more parameters of the leaf FM (e.g., the weights of the neural network (e.g., neural network backbone 402)) and returns to box 608 to perform another iteration. On the other hand, when the decision at decision box 614 is "yes", at box 618, FM function 116 selects one or more parameters of the leaf FM (e.g., the weights of the neural network (e.g., neural network backbone 402)) that output a latent representation that produces the minimum loss value. FM function 116 can use the parameters selected at box 618 to configure the neural network. Training method 600 can terminate after box 618.

[0098] Figure 7 The illustrations, based on at least one embodiment, illustrate what can be used to train a hierarchical FM (e.g., Figure 2A The flowchart illustrates a training method 700 for FM nodes in a tree-structured FM (200). Training method 700 can be executed by data center 100, computing devices, and / or one or more other computing systems. FM function 116 (e.g., if executed by one or more processors 105) can execute training method 700. For example, FM function 116 can implement one or more FM nodes, each of which can be trained using training method 700. Training method 700 can be used to train any FM node at any level of a hierarchical FM (e.g., tree-structured FM 200).

[0099] At box 702, FM function 116 receives a set of latent representations and provides that set of latent representations to an FM node. For example, an FM node may be a child node, and FM function 116 may receive one or more latent representations from at least one parent node of an FM node. As another non-limiting example, an FM node may be a leaf FM, and FM function 116 may receive two or more latent representations (e.g., latent representations 332-1 to 332-P) from another hierarchical FM. The latent representation may encode data (e.g., at least one of one or more data streams 101). The data may include multivariate data, multi-source data, multimodal data, and / or multi-component data. The latent representation may be in any form, including one or more data streams, or stored in a file.

[0100] At box 704, FM function 116 causes one or more processors (e.g., one or more GPUs 108) to perform latent sequence generation process 412 to generate a sequence of latent vectors (e.g., latent vector sequence 406) based at least in part on the set of latent representations obtained at box 702. For example, at box 704, latent sequence generation process 412 may perform alignment process 340, fusion process 344, and / or linear layer 348.

[0101] At box 706, FM function 116 enables one or more processors (e.g., one or more GPUs 108) to provide a sequence of latent vectors obtained at box 704 to a neural network (e.g., neural network backbone 402), which generates new latent representations (e.g., new latent vector sequences) based at least in part on the sequence of latent vectors obtained at box 704.

[0102] At box 708, FM function 116 enables one or more processors (e.g., one or more GPUs 108) to provide the new latent representation obtained at box 706 to reconstruction head 420, and reconstruction head 420 makes predictions on the reconstructed vector sequence (e.g., reconstructed sequence 422).

[0103] At box 710, FM function 116 calculates the loss value (using a loss function) based at least in part on the comparison between the reconstructed vector sequence and the ground truth sequence. The ground truth sequence is the sequence of latent vectors obtained at box 704.

[0104] At decision box 712, FM function 116 determines whether enough training iterations have been performed. When FM function 116 has completed performing training iterations, the decision at decision box 712 is "Yes". Otherwise, the decision at decision box 712 is "No". When the decision at decision box 712 is "No", at box 714, FM function 116 modifies one or more parameters of the FM node (e.g., the weights of the neural network (e.g., neural network backbone 402)) and returns to box 706 to perform another iteration. On the other hand, when the decision at decision box 712 is "Yes", at box 716, FM function 116 selects one or more parameters of the FM node (e.g., the weights of the neural network (e.g., neural network backbone 402)) that output a latent representation that produces the minimum loss value. FM function 116 can use the parameters selected at box 716 to configure the neural network. Training method 700 can terminate after box 716.

[0105] Figure 8An example of a system 800 according to at least one embodiment is illustrated. The system 800 includes one or more drivers and / or one or more runtimes (illustrated as reference numeral 804) that include one or more libraries 806 to provide one or more application programming interfaces (“APIs”) 810. In at least one embodiment, the system 800 includes one or more drivers 804 and / or one or more runtimes 804 that include one or more libraries 806 to provide one or more APIs 810. In at least one embodiment, the one or more APIs 810 are software instruction sets that, if executed, enable one or more processors (e.g., Figure 9 One or more processors 908 illustrated herein perform one or more computational operations. In at least one embodiment, the one or more APIs 810 are distributed or otherwise provided as part of one or more libraries 806, one or more runtimes 804, one or more drivers 804, and / or one or more components of any other grouping of software and / or executable code further described herein. In at least one embodiment, the one or more APIs 810 perform one or more computational operations in response to a call to one or more software programs 802.

[0106] In at least one embodiment, the one or more software programs 802 are software modules and / or include one or more software modules. In at least one embodiment, the software modules are further configured in a non-exclusive manner. Figure 9 The diagram illustrates and describes software program 904 (or components thereof). In at least one embodiment, the one or more software programs 802 are collections of software code, commands, instructions, and / or other text sequences for instructing computing devices (e.g., one or more servers 104) to perform one or more computational operations and / or invoke one or more other sets of instructions to be performed by the computing device, such as one or more APIs 810 or one or more API functions 812. In at least one embodiment, function 812 generates, uses, and / or trains one or more hierarchical FMs (such as tree FM 200), performs inference process 310, performs inference process 330, performs training process 400, performs method 500, performs training method 600, performs training method 700, and / or otherwise performs any of the operations described herein. In at least one embodiment, the functionality provided by one or more APIs 810 includes one or more API functions 812, such as those API functions that can be used to accelerate one or more parts of one or more software programs 802 using one or more parallel processing units (PPUs) (such as graphics processing units (GPUs)).

[0107] In at least one embodiment, one or more APIs 810 are one or more hardware interfaces to one or more circuits performing one or more computational operations. In at least one embodiment, the one or more APIs 810 described herein are implemented to perform coupling... Figures 1-7 The described technology refers to one or more circuits. In at least one embodiment, one or more software programs 802 include instructions that, when executed, cause one or more hardware devices and / or circuits to perform a combination. Figures 1-7 One or more technologies are further described. In at least one embodiment, system 800 includes information regarding... Figure 1 The data system 100 described includes one or more components or all components, and / or at least one server 104, and the system 800 can perform one or more or all processes and / or operations performed by the system and components of the data center 100, and / or one or more servers 104.

[0108] In at least one embodiment, one or more software programs 802 (such as user-implemented software programs) utilize one or more APIs 810 to perform various computational operations, such as memory reservation, matrix multiplication, arithmetic operations, and / or any computational operations performed by a PPU (such as a GPU), as further described herein. In at least one embodiment, one or more functions 812 include a set of callable functions provided by one or more APIs 810, referred herein as APIs, API functions, software functions, and / or functions, which individually perform one or more computational operations, such as computational operations related to parallel computing. In at least one embodiment, one or more APIs 810 generate, use, and / or train one or more hierarchical FMs (such as tree FM 200), execute inference process 310, execute inference process 330, execute training process 400, execute method 500, execute training method 600, execute training method 700, and / or perform other operations described herein (e.g., in combination). Figures 1-7 ).

[0109] In at least one embodiment, one or more software programs 802 interact with or otherwise communicate with one or more APIs 810 to use one or more processors (e.g., Figure 9The one or more processors 908 shown herein, such as one or more PPUs (such as GPUs), are used to perform one or more computational operations. In at least one embodiment, the one or more computational operations using one or more PPUs include at least one or more group computational operations that will be accelerated at least partially by the execution of the one or more PPUs. In at least one embodiment, one or more software programs 802 interact with one or more APIs 810 to generate, use, and / or train one or more hierarchical FMs (such as tree FM 200), perform inference process 310, perform inference process 330, perform training process 400, perform method 500, perform training method 600, perform training method 700, and / or perform other operations described herein (e.g., in conjunction with...). Figures 1-7 ).

[0110] In at least one embodiment, the interface is software instructions that, if executed, provide access to one or more functions 812 provided by one or more APIs 810. In at least one embodiment, when a software developer compiles one or more software programs 802 in conjunction with one or more libraries 806, the one or more software programs 802 use a native interface, and the one or more libraries 806 include or otherwise provide access to one or more APIs 810. In at least one embodiment, one or more software programs 802 are statically compiled together with one or more pre-compiled libraries 806 and / or uncompiled source code including instructions for executing one or more APIs 810. In at least one embodiment, one or more software programs 802 are dynamically compiled, and the dynamically compiled one or more software programs are linked using a linker to one or more pre-compiled libraries 806, which include one or more APIs 810.

[0111] In at least one embodiment, when a software developer executes a software program, one or more software programs 802 use a remote interface that communicates with at least one library 806, including one or more APIs 810, via a network or other remote communication medium, utilizing or otherwise. In at least one embodiment, the one or more libraries 806 including one or more APIs 810 will be executed by a remote computing service, such as a computing resource service provider. In at least one embodiment, the one or more libraries 806 including one or more specific APIs (of one or more APIs 810) will be executed by any other computing host that provides one or more specific APIs to one or more software programs 802.

[0112] In at least one embodiment, a processor that executes or uses one or more specific software programs 802 (e.g., Figure 9 One or more processors 908 as shown call, use, execute, and / or otherwise implement one or more APIs 810 to allocate and otherwise manage memory 814 to be used by one or more specific software programs. In at least one embodiment, one or more specific software programs 802 utilize one or more APIs 810 to allocate and otherwise manage memory 814 so that one or more portions of one or more specific software programs, accelerated using one or more PPUs (e.g., GPUs) or any other accelerators or processors further described herein, can use memory 814. In at least one embodiment, one or more software programs 802 request one or more neural networks to perform signal processing using one or more functions 812 provided by one or more APIs 810. In at least one embodiment, storage device 110 and / or memory 112 implement memory 814.

[0113] In at least one embodiment, one or more APIs 810 are APIs for parallel computing. In at least one embodiment, one or more APIs 810 are any other APIs further described herein. In at least one embodiment, one or more APIs 810 are provided by one or more drivers 804 and / or one or more runtimes 804. In at least one embodiment, one or more APIs 810 are provided by CUDA user-mode drivers. In at least one embodiment, one or more APIs 810 are provided by the CUDA runtime. In at least one embodiment, one or more drivers 804 are data values ​​and software instructions that, if executed, perform and / or otherwise facilitate the operation of one or more functions 812 of one or more APIs 810 during the loading and execution of one or more portions of at least one software program 802. In at least one embodiment, one or more runtimes 804 are data values ​​and / or software instructions that, if executed, perform or otherwise facilitate the operation of one or more functions 812 of one or more APIs 810 during the execution of at least one software program 802. In at least one embodiment, one or more specific software programs 802 utilize one or more APIs 810 implemented and / or otherwise provided by one or more drivers 804 and / or one or more runtimes 804 to perform combinatorial arithmetic operations by one or more specific software programs during execution by one or more PPUs (such as GPUs).

[0114] In at least one embodiment, one or more software programs 802 utilize one or more APIs 810 provided by one or more drivers 804 and / or one or more runtimes 804 to perform combinatorial arithmetic operations for one or more PPUs (such as GPUs). In at least one embodiment, as described above, one or more APIs 810 provide combinatorial arithmetic operations via one or more drivers 804 and / or one or more runtimes 804. In at least one embodiment, one or more software programs 802 utilize one or more APIs 810 provided by one or more drivers 804 and / or one or more runtimes 804 to allocate or otherwise reserve one or more blocks of memory 814 for one or more PPUs (such as GPUs). In at least one embodiment, one or more software programs 802 utilize one or more APIs 810 provided by one or more drivers 804 and / or one or more runtimes 804 to allocate or otherwise reserve blocks of memory 814.

[0115] In at least one embodiment, to improve the availability and / or performance of one or more specific programs of one or more software programs 802, one or more portions of the specific software program will be accelerated by one or more PPUs (such as GPUs). In at least one embodiment, one or more functions 812 receive one or more input parameters, which indicate one or more inputs to one or more neural networks and / or other data to be utilized by one or more neural networks, such as one or more hyperparameters of one or more neural networks. In at least one embodiment, one or more input parameters include one or more inputs and / or other data. In at least one embodiment, one or more input parameters include one or more pointers to one or more storage locations storing one or more inputs and / or other data.

[0116] In at least one embodiment, system 800 includes at least one processor (e.g., Figure 9 The system 800 includes one or more processors 908 (as shown), which include one or more circuitry for executing one or more software programs to combine two or more of the one or more APIs 810 into a single API. In at least one embodiment, the system 800 includes at least one processor (e.g., Figure 9The system 800 includes one or more processors 908 (as shown), which use one or more APIs 810 to generate, use, and / or train one or more hierarchical FMs (such as tree FMs 200), perform inference processes 310, perform inference processes 330, perform training processes 400, perform methods 500, perform training methods 600, perform training methods 700, and / or otherwise perform the operations described herein. In at least one embodiment, the system 800 includes at least one processor (e.g., Figure 9 The processor 908 shown uses one or more APIs 810 to execute... Figures 1-7 The image shown and / or about Figures 1-7 One or more of the operations described, such as Figures 1-7 Or one or more processes shown in their portions. In at least one embodiment, system 800 includes at least one processor (e.g., Figure 9 One or more processors 908 (as shown) are used to execute one or more functions 812, such as combination Figures 1-7 The described functions. In at least one embodiment, one or more APIs 810 will be combined with Figures 9-12 The hardware described is used to execute it.

[0117] Figure 9An example of a system 900 according to at least one embodiment is illustrated. The system 900 may include software and hardware for generating at least one latent representation of two or more input data streams using multiple machine learning processes arranged hierarchically (e.g., included in FM nodes), performing one or more tasks based at least in part on the at least one latent representation, or otherwise performing any of the operations described herein. For example, the software and hardware of system 900 may generate, use, and / or train one or more hierarchical FMs (such as tree FM 200), perform inference process 310, perform inference process 330, perform training process 400, perform method 500, perform training method 600, perform training method 700, and / or otherwise perform the operations described herein. System 900 may include a storage device 902 and one or more processors 908. Storage device 902 may include, for example, memory, cache, or other storage devices further described herein. Storage device 902 may be separate from one or more processors 908, or storage device 902 may be included in one or more processors 908 (e.g., in storage device 912). In at least one embodiment, software program 904 and / or software library (or instructions) 906 may be stored in memory, cache, or other storage devices and provided to one or more processors 908 to cause one or more circuits of one or more processors 908 to perform the operations described herein. In at least one embodiment, software program 904 and / or software library (or instructions) 906 may be integrated into one or more circuits of one or more processors 908. Software program 904, which can be used to perform any of the operations described herein, may be stored on storage device 902.

[0118] In at least one embodiment, software program 904 may include one or more software modules. The software modules may implement FM function 116 (e.g., by storing instructions for at least partially implementing FM function 116), generate, use, and / or train one or more hierarchical FMs (such as tree FM 200), perform inference process 310, perform inference process 330, perform training process 400, perform method 500, perform training method 600, perform training method 700, and / or otherwise perform the operations described herein. In at least one embodiment, the software module implements a computer-implemented method comprising: generating a sequence of latent vectors by aligning and merging at least two latent representations obtained at least partially based on two or more input data streams; generating at least one latent representation using at least one neural network at least partially based on the sequence of latent vectors; performing one or more tasks at least partially based on the at least one latent representation; and / or otherwise performing the operations described herein.

[0119] In at least one embodiment, as used in any implementation described herein, unless explicitly stated otherwise from the context or explicitly to the contrary, "module" refers to any combination of software logic, firmware logic, hardware logic, and / or circuitry configured to provide the functionality described herein. In at least one embodiment, software may be embodied as a software package, code, and / or instruction set or instructions, and as used in any implementation described herein, "hardware" may include, for example, hardwired circuitry systems, programmable circuitry systems, state machine circuitry systems, fixed-function circuitry systems, execution unit circuitry systems, and / or firmware storing instructions executed by programmable circuitry, either individually or in any combination. In at least one embodiment, modules may be collectively or individually embodied as circuitry systems forming part of a larger system, such as an integrated circuit (IC), a system-on-a-chip (SoC), etc. In at least one embodiment, a module performs one or more processes in conjunction with any suitable processing unit and / or combination of processing units (such as one or more CPUs, GPUs, GPGPUs, PPUs, and / or variants thereof, including those further described herein).

[0120] In at least one embodiment, software program 904 may include a collection of software code, commands, instructions, or other text sequences for instructing a computing device to perform one or more computational operations and / or to invoke one or more other instruction sets (such as one or more APIs or one or more API functions or instruction set architecture (ISA) level instructions) to be executed or otherwise operated. Instructions (e.g., hardware instructions) or microcode may relate to ISA level instructions, which may include native ISA instructions or non-native ISA commands. Software program 904 and / or software library (or instruction) 906 (e.g., one or more modules) may be distributed among multiple processors that communicate via buses, networks, writes to shared memory, and / or any suitable communication procedures as described herein.

[0121] In at least one embodiment, system 900 may include one or more software libraries 906, which may, for example, provide one or more API and / or ISA instructions. In at least one embodiment, one or more API and / or ISA instructions may be used to generate, use, and / or train one or more hierarchical FMs (such as tree FM 200), perform inference process 310, perform inference process 330, perform training process 400, perform method 500, perform training method 600, perform training method 700, and / or otherwise perform the operations described herein. In at least one embodiment, one or more software libraries 906 may be included in a driver and / or runtime. In at least one embodiment, software library 906 (e.g., including one or more API and / or ISA instructions) may include a set of software instructions that, if executed or otherwise operated, cause one or more processors 908 to perform one or more computational operations, such as any of the operations described herein. In at least one embodiment, one or more API and / or ISA instructions may be provided distributedly or otherwise as part of one or more software libraries 906, runtimes, drivers, and / or any other grouping of software and / or executable code further described herein. In at least one embodiment, in response to a call to software program 904, one or more API and / or ISA instructions may perform one or more computational operations.

[0122] One or more processors 908 may include any number of processors and any suitable processing units, and / or combinations of processing units, such as, but not limited to, a central processing unit (“CPU”), a graphics processing unit (“GPU”), or other processors (including accelerators, field-programmable gate arrays (FPGAs), graphics processors, parallel processors, GPGPUs, DPUs, and / or variations thereof, including those further described herein), including any processors described herein, such as, but not limited to, those described herein. Figures 9-12The processors in the memory. In at least one embodiment, one or more processors 908 may retrieve or extract instructions (e.g., one or more API and / or ISA instructions) from memory 902 using, for example, instruction retrieval 916 (e.g., for the instruction retrieval phase). Instructions may include instructions for generating, using, and / or training one or more hierarchical FMs (such as tree FM 200), executing inference process 310, executing inference process 330, executing training process 400, executing method 500, executing training method 600, executing training method 700, and / or otherwise performing the operations described herein. In at least one embodiment, one or more processors 908 may include memory 912 and instruction queue 910 for storing and queuing instructions retrieved from memory 902. In at least one embodiment, the retrieved instructions may be decoded by decoder 918 to determine what operation one or more processors 908 should perform (e.g., in the instruction decoding phase). In at least one embodiment, one or more processors 908 may fetch additional operands (data) that can be used for instructions, and the operands may be stored in, for example, registers or storage devices 912. In at least one embodiment, micro-operations 920 may perform operations on the data stored in one or more registers or storage devices 912. For example, each step of an instruction fetched by one or more processors 908 may be broken down during execution so that one or more processors 908 may execute the instruction step-by-step through a series of micro-operations 920. In at least one embodiment, a program counter (PC) 914 may hold the address of the next instruction and may be updated to point to the next instruction to be executed by one or more processors 908.

[0123] In at least one embodiment, one or more processors 908 may execute instructions (e.g., during execution phase). For example, one or more processors 908 may perform operations specified by the instructions, such as arithmetic operations, logical operations, or data transfers. In at least one embodiment, one or more computing units 922 may execute instructions to implement any of the operations described herein. In at least one embodiment, one or more computing units may include one or more ALUs 924 (Arithmetic Logic Units) that can be used to perform arithmetic and logical operations. In at least one embodiment, one or more computing units may include one or more FPUs (Floating Point Units) 926 that can be used to perform floating-point calculations. In at least one embodiment, other circuitry 928 may be used to perform other operations, such as vector and / or scalar operations. In at least one embodiment, one or more accelerators 930 may include one or more matrix multiplication accelerators, one or more parallel processing units (PPUs) (such as GPUs), or any other accelerators or processors further described herein. In at least one embodiment, software program 904 may utilize one or more API and / or ISA instructions to perform various computational operations (such as matrix multiplication, arithmetic operations) or any other computational operations further described herein via one or more accelerators 930. In at least one embodiment, one or more computational operations using one or more accelerators 930 may include at least one or more sets of computational operations that will be accelerated by being at least partially executed via one or more accelerators 930, including generating, using, and / or training one or more hierarchical FMs (such as tree FM 200), performing inference process 310, performing inference process 330, performing training process 400, performing method 500, performing training method 600, performing training method 700, and / or otherwise performing the operations described herein.

[0124] In at least one embodiment, system 900 can be used to execute one or more instructions that include functions or operations, such as combining Figures 1-12 The functions or operations described herein. In at least one embodiment, system 900 includes one or more processors (including one or more circuits) for using multiple neural networks arranged in a hierarchical manner to: generate at least one latent representation of two or more input data streams having different modalities; generate one or more hierarchical FMs (such as tree FM 200); use one or more hierarchical FMs (such as tree FM 200); train one or more hierarchical FMs (such as tree FM 200); perform inference process 310; perform inference process 330; perform training process 400; perform method 500; perform training method 600; perform training method 700; and / or otherwise perform the operations described herein. In at least one embodiment, system 900 is included in Figures 1-12 The illustrated system and / or otherwise includes Figures 1-12 The illustrated system enables one or more circuits to use multiple neural networks arranged in a hierarchical manner to: generate at least one latent representation of two or more input data streams with different modalities; generate one or more hierarchical FMs (such as tree FM 200); use one or more hierarchical FMs (such as tree FM 200); train one or more hierarchical FMs (such as tree FM 200); perform inference process 310; perform inference process 330; perform training process 400; perform method 500; perform training method 600; perform training method 700; and / or otherwise perform the operations described herein. In at least one embodiment, system 900 includes Figures 1-12 The illustrated hardware is used to: generate at least one latent representation of two or more input data streams with different modalities, generate one or more hierarchical FMs (such as tree FM 200), use one or more hierarchical FMs (such as tree FM 200), train one or more hierarchical FMs (such as tree FM 200), perform inference process 310, perform inference process 330, perform training process 400, perform method 500, perform training method 600, perform training method 700, and / or otherwise perform the operations described herein.

[0125] logic Figure 10A Logic 1015 is illustrated, and as described elsewhere herein, it can be used in one or more devices to perform operations such as those discussed herein, according to at least one embodiment. In at least one embodiment, logic 1015 is used to perform inference and / or training operations associated with one or more embodiments. In at least one embodiment, logic 1015 is inference and / or training logic. The following is in conjunction with... Figure 10A and / or Figure 10B Detailed information about logic 1015 is provided. In at least one embodiment, logic refers to any combination of software logic, hardware logic, and / or firmware logic to provide the functionality or operation described herein, wherein the logic may be embodied collectively or individually as circuitry forming part of a larger system, such as an integrated circuit (IC), a system-on-a-chip (SoC), or one or more processors (e.g., CPU, GPU).

[0126] In at least one embodiment, logic 1015 may include, but is not limited to, code and / or data storage 1001 for storing forward and / or output weights and / or input / output data, and / or other parameters for configuring neurons or layers of a neural network trained and / or used for inference in one or more embodiments. In at least one embodiment, logic 1015 may include or be coupled to code and / or data storage 1001 for storing graph code or other software to control timing and / or sequence, wherein weight and / or other parameter information is loaded to configure logic including integer and / or floating-point units (collectively, arithmetic logic units (ALUs)). In at least one embodiment, code (such as graph code) loads weight or other parameter information into the processor ALU based on the architecture of the neural network to which the code corresponds. In at least one embodiment, code and / or data storage 1001 stores weight parameters and / or input / output data of each layer of a neural network trained or used in one or more embodiments during forward propagation of input / output data and / or weight parameters using aspects of one or more embodiments. In at least one embodiment, any portion of the code and / or data storage 1001 may be included within other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory.

[0127] In at least one embodiment, any portion of the code and / or data storage 1001 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data storage 1001 may be a cache memory, dynamic random access memory (“DRAM”), static random access memory (“SRAM”), non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the choice of whether the code and / or data storage 1001 is internal or external to the processor, for example, or including DRAM, SRAM, flash memory, or some other storage type, may depend on the on-chip versus off-chip available storage, the latency requirements of the training and / or inference functions being performed, the batch size of the data used in the inference and / or training of the neural network, or some combination of these factors.

[0128] In at least one embodiment, logic 1015 may include, but is not limited to, code and / or data storage 1005 for storing backpropagation and / or output weights and / or input / output data corresponding to neurons or layers of a neural network trained and / or used for inference in one or more aspects of the embodiments. In at least one embodiment, during training and / or inference using one or more aspects of the embodiments, code and / or data storage 1005 stores weight parameters and / or input / output data of each layer of the neural network trained or used in one or more embodiments during backpropagation of input / output data and / or weight parameters. In at least one embodiment, logic 1015 may include or be coupled to code and / or data storage 1005 for storing graph code or other software to control timing and / or sequence, wherein weight and / or other parameter information is loaded to configure logic including integer and / or floating-point units (collectively, arithmetic logic units (ALUs)).

[0129] In at least one embodiment, code (such as graph code) causes the architecture of the neural network corresponding to that code to load weights or other parameter information into the processor ALU. In at least one embodiment, any portion of the code and / or data storage 1005 may be included together with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory. In at least one embodiment, any portion of the code and / or data storage 1005 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data storage 1005 may be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the choice of whether the code and / or data storage 1005 is internal or external to the processor, for example, including DRAM, SRAM, flash memory, or some other type of storage, may depend on the available on-chip versus off-chip storage, the latency requirements of the training and / or inference functions being performed, the batch size of the data used in the inference and / or training of the neural network, or some combination of these factors.

[0130] In at least one embodiment, code and / or data storage 1001 and code and / or data storage 1005 may be separate storage structures. In at least one embodiment, code and / or data storage 1001 and code and / or data storage 1005 may be the same storage structure. In at least one embodiment, code and / or data storage 1001 and code and / or data storage 1005 may be partially combined and partially separated. In at least one embodiment, any portion of code and / or data storage 1001 and code and / or data storage 1005 may be included together with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory.

[0131] In at least one embodiment, logic 1015 may include, but is not limited to, one or more arithmetic logic units (“ALUs”) 1010 (including integer and / or floating-point units) for performing logical and / or mathematical operations at least in part based on training and / or inference code (e.g., graph code) or as instructed thereto, the results of which may produce activations (e.g., output values ​​from layers or neurons within a neural network) stored in activation storage 1020, which are functions of input / output and / or weight parameter data stored in code and / or data storage 1001 and / or code and / or data storage 1005. In at least one embodiment, activations stored in activation storage 1020 are generated based on linear algebra and / or matrix-based mathematics performed by ALU 1010 in response to execution instructions or other code, wherein weight values ​​stored in code and / or data storage 1005 and / or data storage 1001 are used as operands, and other values, such as bias values, gradient information, momentum values, or other parameters or hyperparameters, may be stored in code and / or data storage 1005 or code and / or data storage 1001 or other on-chip or off-chip storage.

[0132] In at least one embodiment, one or more processors or other hardware logic devices or circuits include one or more ALUs 1010, while in another embodiment, one or more ALUs 1010 may be external to the processor or other hardware logic device or the circuitry that uses them (e.g., a coprocessor). In at least one embodiment, ALUs 1010 may be included within an execution unit of a processor, or otherwise included in an ALU bank accessible by the execution unit of the processor, which may be within the same processor or distributed among different processors of different types (e.g., a central processing unit, a graphics processing unit, a fixed-function unit, etc.). In at least one embodiment, data storage 1001, code and / or data storage 1005, and activation storage 1020 may share a processor or other hardware logic device or circuitry, while in another embodiment, they may be in different processors or other hardware logic devices or circuitry, or some combination of the same and different processors or other hardware logic devices or circuitry. In at least one embodiment, any portion of activation storage 1020 may be included together with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory. Furthermore, inference and / or training code may be stored together with other code accessible to the processor or other hardware logic or circuitry, and may be retrieved and / or processed using the processor’s fetch, decode, schedule, execute, exit, and / or other logic circuitry.

[0133] In at least one embodiment, the active memory 1020 may be a cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other memory. In at least one embodiment, the active memory 1020 may be entirely or partially located within or outside one or more processors or other logic circuits. In at least one embodiment, the choice of whether the active memory 1020 is internal to or external to the processor, for example, or including DRAM, SRAM, flash memory, or certain other memory types, may depend on the available on-chip versus off-chip memory, the latency requirements for performing training and / or inference functions, the batch size of the data used in inference and / or training the neural network, or some combination of these factors.

[0134] In at least one embodiment, Figure 10A The logic 1015 shown can be used in conjunction with an application-specific integrated circuit (“ASIC”), such as a TensorFlow® processing unit from Google, an inference processing unit (IPU) from Graphcore™, or a Nervana® processor from Intel (e.g., “Lake Crest”). In at least one embodiment, Figure 10AThe logic 1015 shown can be used in conjunction with central processing unit (“CPU”) hardware, graphics processing unit (“GPU”) hardware, or other hardware such as field programmable gate array (“FPGA”).

[0135] Figure 10B A logic 1015 according to at least one embodiment is illustrated. In at least one embodiment, the logic 1015 is inference and / or training logic. In at least one embodiment, the logic 1015 may include, but is not limited to, hardware logic, wherein computational resources, along with weight values ​​or other information corresponding to one or more layers of neurons within a neural network, are used dedicatedly or otherwise exclusively. In at least one embodiment, Figure 10B The logic 1015 shown can be used in conjunction with an application-specific integrated circuit (ASIC), such as a TensorFlow® processing unit from Google, an inference processing unit (IPU) from Graphcore™, or a Nervana® processor from Intel (e.g., “Lake Crest”). In at least one embodiment, Figure 10B The logic 1015 shown can be used in conjunction with central processing unit (CPU) hardware, graphics processing unit (GPU) hardware, or other hardware (e.g., field-programmable gate array (FPGA)). In at least one embodiment, logic 1015 includes, but is not limited to, code and / or data storage 1001 and code and / or data storage 1005, which can be used to store code (e.g., graph code), weight values, and / or other information, including bias values, gradient information, momentum values, and / or other parameter or hyperparameter information. Figure 10B In at least one embodiment shown, each of code and / or data storage 1001 and code and / or data storage 1005 is associated with dedicated computing resources (e.g., computing hardware 1002 and computing hardware 1006), respectively. In at least one embodiment, each of computing hardware 1002 and computing hardware 1006 includes one or more ALUs that perform mathematical functions (e.g., linear algebraic functions) on the information stored in code and / or data storage 1001 and code and / or data storage 1005, respectively, with the results stored in active storage 1020.

[0136] In at least one embodiment, each of the code and / or data storage 1001, 1005 and the corresponding computing hardware 1002 and 1006 corresponds to a different layer of the neural network, such that an activation obtained from a storage / computation pair 1001 / 1002 of the code and / or data storage 1001 and computing hardware 1002 is provided as input to the next storage / computation pair 1005 / 1006 of the code and / or data storage 1005 and computing hardware 1006, in order to reflect the conceptual organization of the neural network. In at least one embodiment, each storage / computation pair 1001 / 1002 and 1005 / 1006 may correspond to more than one neural network layer. In at least one embodiment, additional storage / computation pairs (not shown) may be included in logic 1015 following or in parallel with the storage / computation pairs 1001 / 1002 and 1005 / 1006.

[0137] Logic 1015 can generate one or more hierarchical FMs (such as tree FM 200), use one or more hierarchical FMs (such as tree FM 200), train one or more hierarchical FMs (such as tree FM 200), perform inference process 310, perform inference process 330, perform training process 400, perform method 500, perform training method 600, perform training method 700, and / or otherwise perform the operations described herein.

[0138] Data Center Figure 11 An example data center 1100 that can be used with at least one embodiment is shown. In at least one embodiment, the data center 1100 includes a data center infrastructure layer 1110, a framework layer 1120, a software layer 1130, and an application layer 1140.

[0139] In at least one embodiment, such as Figure 11As shown, the data center infrastructure layer 1110 may include a resource coordinator 1112, grouped computing resources 1114, and node computing resources (“nodes CR”) 1116(1)-1116(N), where “N” represents a positive integer (which may be an integer “N” different from the integers used in other diagrams). In at least one embodiment, nodes CR 1116(1)-1116(N) may include, but are not limited to, any number of central processing units (“CPUs”) or other processors (including accelerators, field-programmable gate arrays (FPGAs), graphics processors, etc.), memory storage devices 1118(1)-1118(N) (e.g., dynamic read-only memory, solid-state storage, or disk drives), network input / output (“NW I / O”) devices, network switches, virtual machines (“VMs”), power modules, and cooling modules, etc. In at least one embodiment, one or more nodes CR 1116(1)-1116(N) may be servers having one or more of the aforementioned computing resources.

[0140] In at least one embodiment, the grouped computing resources 1114 may include individual groups of node CRs housed within one or more racks (not shown), or a plurality of racks housed within data centers (also not shown) in various geographical locations. In at least one embodiment, the individual groups of node CRs within the grouped computing resources 1114 may include computing, networking, memory, or storage resources that can be configured or allocated to support groups of one or more workloads. In at least one embodiment, several node CRs, including CPUs or processors, may be grouped within one or more racks to provide computing resources to support one or more workloads. In at least one embodiment, one or more racks may also include any number of power modules, cooling modules, and network switches in any combination.

[0141] In at least one embodiment, resource coordinator 1112 may be configured or otherwise control one or more nodes CR 1116(1)-1116(N) and / or grouped computing resources 1114. In at least one embodiment, resource coordinator 1112 may include a Software Design Infrastructure (“SDI”) management entity for data center 1100. In at least one embodiment, resource coordinator 1112 may include hardware, software, or some combination thereof.

[0142] In at least one embodiment, such as Figure 11As shown, framework layer 1120 includes job scheduler 1122, configuration manager 1124, resource manager 1126, and distributed file system 1128. In at least one embodiment, framework layer 1120 may include a framework of software 1132 supporting software layer 1130 and / or one or more applications 1142 supporting application layer 1140. In at least one embodiment, software 1132 or application 1142 may respectively include web-based service software or applications, such as service software or applications provided by Amazon Web Services, Google Cloud, and Microsoft Azure. In at least one embodiment, framework layer 1120 may be, but is not limited to, a type of free and open-source software web application framework, such as Apache Spark™ (hereinafter referred to as "Spark") which can leverage distributed file system 1128 for large-scale data processing (e.g., "big data"). In at least one embodiment, job scheduler 1122 may include Spark drivers for facilitating the scheduling of workloads supported by the various layers of data center 1100. In at least one embodiment, configuration manager 1124 may be able to configure different layers, such as software layer 1130 and framework layer 1120 including Spark and distributed file system 1128 for supporting large-scale data processing. In at least one embodiment, resource manager 1126 may be able to manage clustered or grouped computing resources mapped to or allocated to support distributed file system 1128 and job scheduler 1122. In at least one embodiment, clustered or grouped computing resources may include grouped computing resources 1114 at data center infrastructure layer 1110. In at least one embodiment, resource manager 1126 may coordinate with resource coordinator 1112 to manage these mapped or allocated computing resources.

[0143] In at least one embodiment, the software 1132 included in software layer 1130 may include software used by at least portions of nodes CR1116(1)-1116(N), grouped computing resources 1114, and / or the distributed file system 1128 of framework layer 1120. In at least one embodiment, one or more types of software may include, but are not limited to, Internet web page search software, email virus scanning software, database software, and streaming video content software.

[0144] In at least one embodiment, one or more applications 1142 included in application layer 1140 may include one or more types of applications used by at least portions of nodes CR 1116(1)-1116(N), grouped computing resources 1114, and / or the distributed file system 1128 of framework layer 1120. In at least one embodiment, one or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing, applications, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), or other machine learning applications used in conjunction with one or more embodiments.

[0145] In at least one embodiment, any of the configuration manager 1124, resource manager 1126, and resource coordinator 1112 can perform any number and type of self-modification actions based on any amount and type of data acquired in any technically feasible manner. In at least one embodiment, self-modification actions can mitigate potentially poor configuration decisions by data center operators of data center 1100 and can prevent underutilization and / or poor performance of the data center.

[0146] In at least one embodiment, data center 1100 may include tools, services, software, or other resources for training one or more machine learning models or using one or more machine learning models to predict or infer information according to one or more embodiments described herein. For example, in at least one embodiment, a machine learning model can be trained by calculating weight parameters based on a neural network architecture using the software and computing resources described above with respect to data center 1100. In at least one embodiment, information can be inferred or predicted using trained machine learning models corresponding to one or more neural networks using the resources described above with respect to data center 1100 by using weight parameters calculated through one or more training techniques described herein.

[0147] In at least one embodiment, the data center may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, or other hardware to utilize the aforementioned resources to perform training and / or inference. Furthermore, one or more of the aforementioned software and / or hardware resources may be configured as a service to allow a user to train or perform information inference, such as image recognition, speech recognition, or other artificial intelligence services.

[0148] The logic 1015 is used to perform inference and / or training operations associated with one or more embodiments. This document combines... Figure 10A and / or Figure 10BDetails regarding logic 1015 are provided. In at least one embodiment, logic 1015 may be used in data center 1100 for inference or prediction operations based at least in part on weight parameters computed using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0149] Data center 1100 may generate one or more hierarchical FMs (such as tree FM 200), use one or more hierarchical FMs (such as tree FM 200), train one or more hierarchical FMs (such as tree FM 200), perform inference process 310, perform inference process 330, perform training process 400, perform method 500, perform training method 600, perform training method 700, and / or otherwise perform the operations described herein.

[0150] Computing System Figure 12 This is a block diagram illustrating an exemplary computer system according to at least one embodiment. The exemplary computer system may be a system with interconnected devices and components, a system-on-a-chip (SoC), or some combination thereof formed with a processor, which may include an execution unit for executing instructions. In at least one embodiment, according to this disclosure, such as in the embodiments described herein, computer system 1200 may include, but is not limited to, components such as processor 1202 for employing execution units (including logic) to execute algorithms for process data. In at least one embodiment, computer system 1200 may include a processor, such as the PENTIUM® processor family, Xeon™, Itanium®, XScale™ and / or StrongARM™, Intel® Core™ or Intel® Nervana™ microprocessors available from Intel Corporation of Santa Clara, California, although other systems (including PCs, engineering workstations, set-top boxes, etc.) with other microprocessors may also be used. In at least one embodiment, the computer system 1200 may execute a version of the Windows operating system available from Microsoft Corporation of Redmond, Wash., although other operating systems (such as UNIX and Linux), embedded software, and / or graphical user interfaces may also be used.

[0151] The embodiments can be used in other devices, such as handheld devices and embedded applications. Some examples of handheld devices include cellular phones, Internet Protocol (IP) devices, digital cameras, personal digital assistants (“PDAs”), and handheld PCs. In at least one embodiment, the embedded application may include a microcontroller, a digital signal processor (“DSP”), a system-on-a-chip (SoC), a network computer (“NetPC”), a set-top box, a network hub, a wide area network (“WAN”) switch, or any other system capable of executing one or more instructions according to at least one embodiment.

[0152] In at least one embodiment, the computer system 1200 may include, but is not limited to, a processor 1202, which may include, but is not limited to, one or more execution units 1208 for performing machine learning model training and / or inference according to the techniques described herein. In at least one embodiment, the computer system 1200 is a single-processor desktop or server system, but in another embodiment, the computer system 1200 may be a multiprocessor system. In at least one embodiment, the processor 1202 may include, but is not limited to, for example, a Complex Instruction Set Computer (“CISC”) microprocessor, a Reduced Instruction Set Computing (“RISC”) microprocessor, a Very Long Instruction Word (“VLIW”) microprocessor, a processor implementing instruction set combination, or any other processor device, such as a digital signal processor. In at least one embodiment, the processor 1202 may be coupled to a processor bus 1210, which can transmit data signals between the processor 1202 and other components in the computer system 1200.

[0153] In at least one embodiment, processor 1202 may include, but is not limited to, a Level 1 (“L1”) internal cache memory (“cache”) 1204. In at least one embodiment, processor 1202 may have a single internal cache or multiple levels of internal caches. In at least one embodiment, the cache memory may reside external to processor 1202. Depending on specific implementation and requirements, other embodiments may also include a combination of internal and external caches. In at least one embodiment, register file 1206 may store different types of data in various registers, including but not limited to integer registers, floating-point registers, status registers, and instruction pointer registers.

[0154] In at least one embodiment, an execution unit 1208, including but not limited to logic for performing integer and floating-point operations, is also located in the processor 1202. In at least one embodiment, the processor 1202 may also include a microcode (“ucode”) read-only memory (“ROM”) storing microcode of certain macro instructions. In at least one embodiment, the execution unit 1208 may include logic for processing a packaged instruction set 1209. In at least one embodiment, by including the packaged instruction set 1209 in the instruction set of the general-purpose processor and the associated circuitry to be executed, packaged data in the processor 1202 can be used to perform operations used by numerous multimedia applications. In at least one embodiment, numerous multimedia applications can be accelerated and executed more efficiently by using the full width of the processor’s data bus to perform operations on the packaged data, eliminating the need to transfer smaller data units on the processor’s data bus to perform one or more operations on one data element at a time.

[0155] In at least one embodiment, execution unit 1208 may also be used in a microcontroller, embedded processor, graphics device, DSP, and other types of logic circuitry. In at least one embodiment, computer system 1200 may include, but is not limited to, memory 1220. In at least one embodiment, memory 1220 may be a dynamic random access memory (“DRAM”) device, a static random access memory (“SRAM”) device, a flash memory device, or other memory device. In at least one embodiment, memory 1220 may store one or more instructions 1219 and / or data 1221 represented by data signals executable by processor 1202.

[0156] In at least one embodiment, the system logic chip may be coupled to processor bus 1210 and memory 1220. In at least one embodiment, the system logic chip may include, but is not limited to, a memory controller hub (“MCH”) 1216, and processor 1202 may communicate with MCH 1216 via processor bus 1210. In at least one embodiment, MCH 1216 may provide a high-bandwidth memory path 1218 to memory 1220 for instruction and data storage, as well as for storage of graphics commands, data, and textures. In at least one embodiment, MCH 1216 may direct data signals between processor 1202, memory 1220, and other components in computer system 1200, and bridge data signals between processor bus 1210, memory 1220, and system I / O interface 1222. In at least one embodiment, the system logic chip may provide a graphics port for coupling to a graphics controller. In at least one embodiment, MCH 1216 can be coupled to memory 1220 via high-bandwidth memory path 1218, and graphics / video card 1212 can be coupled to MCH 1216 via Accelerated Graphics Port (“AGP”) interconnect 1214.

[0157] In at least one embodiment, the computer system 1200 may use the system I / O interface 1222 as a proprietary hub interface bus to couple the MCH 1216 to the I / O controller hub (“ICH”) 1230. In at least one embodiment, the ICH 1230 may provide direct connectivity to certain I / O devices via a local I / O bus. In at least one embodiment, the local I / O bus may include, but is not limited to, a high-speed I / O bus for connecting peripheral devices to the memory 1220, chipset, and processor 1202. Examples may include, but are not limited to, an audio controller 1229, a firmware hub (“Flash BIOS”) 1228, a wireless transceiver 1226, a data storage 1224, a conventional I / O controller 1223 including a user input and keyboard interface 1225, a serial expansion port 1227 (such as a Universal Serial Bus (“USB”) port), and a network controller 1234. In at least one embodiment, the data storage 1224 may include a hard disk drive, a floppy disk drive, a CD-ROM device, a flash memory device, or other mass storage device.

[0158] In at least one embodiment, Figure 12 A system including interconnected hardware devices or "chips" is shown, while in other embodiments, Figure 12 An exemplary SoC can be shown. In at least one embodiment, Figure 12The devices shown can be interconnected using proprietary interconnects, standardized interconnects (e.g., PCIe), or some combination thereof. In at least one embodiment, one or more components of the computer system 1200 are interconnected using a Compute Fast Link (CXL) interconnect.

[0159] The logic 1015 is used to perform inference and / or training operations associated with one or more embodiments. This document combines... Figure 10A and / or Figure 10B Details regarding logic 1015 are provided. In at least one embodiment, logic 1015 may be used in computer system 1200 for performing inference or prediction operations based at least in part on weight parameters computed using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0160] One or more processors 105 may include processor 1202, which may generate one or more hierarchical FMs (such as tree FM 200), use one or more hierarchical FMs (such as tree FM 200), train one or more hierarchical FMs (such as tree FM 200), perform inference process 310, perform inference process 330, perform training process 400, perform method 500, perform training method 600, perform training method 700, and / or otherwise perform the operations described herein. One or more instructions 1219 may include one or more instructions 114. Memory 1220 and / or data storage device 1224 may include memory 110 and storage device 112.

[0161] At least one embodiment of this disclosure may be described in accordance with the following terms: Clause 1. A processor comprising: one or more circuits, said one or more circuits being configured to: generate at least one latent representation of two or more input data streams using a plurality of neural networks arranged in a hierarchical manner, wherein said two or more input data streams have different modalities.

[0162] 2. The processor as described in Clause 1, wherein the plurality of neural networks comprises: a first set of neural networks for generating a first set of latent representations using the two or more input data streams; and a second set of neural networks for generating a second set of latent representations using the first set of latent representations, the first set of latent representations being for corresponding to a first data resolution, and the second set of latent representations being for corresponding to a second data resolution, wherein the second data resolution is lower than the first data resolution.

[0163] 3. The processor as described in Clause 2, wherein the first data resolution represents different devices in a data center, and the second data resolution represents a combination of two or more devices.

[0164] 4. The processor as described in any one of Clauses 1 to 3, wherein the plurality of neural networks are configured to: obtain a plurality of first latent representations at least in part based on the two or more input data streams; align and merge at least two of the plurality of first latent representations to generate a plurality of second latent representations; and align and merge at least two of the plurality of second latent representations to generate the at least one latent representation.

[0165] 5. The processor as described in Clause 4, wherein at least two of the plurality of first potential representations correspond to the same device.

[0166] 6. The processor as described in Clause 4 or Clause 5, wherein different portions of the at least two of the plurality of second potential representations correspond to different devices.

[0167] 7. The processor as described in any of Clauses 1 to 6, wherein the one or more circuits are at least configured to: generate one or more predictions based at least in part on patterns identified from the at least one latent representation.

[0168] 8. The processor as described in any of Clauses 1 to 7, wherein the two or more input data streams include at least one of multimodal data, multivariate data, multicomponent data, or multisource data.

[0169] 9. The processor as described in any of Clauses 1 to 8, wherein at least one of the plurality of neural networks is trained using self-supervised training.

[0170] 10. A computer-implemented method comprising: generating a sequence of latent vectors by aligning and merging at least two latent representations, wherein the at least two latent representations are obtained at least partially based on two or more input data streams; generating at least one latent representation using at least one neural network based at least partially based on the sequence of latent vectors; and performing one or more tasks based at least partially based on the at least one latent representation.

[0171] 11. The computer-implemented method as described in Clause 10, wherein the two or more input data streams transmit information relating to one or more devices in a data center.

[0172] 12. A computer-implemented method as described in Clause 10 or Clause 11, wherein the one or more tasks include: generating one or more predictions based at least in part on patterns identified using the at least one latent representation.

[0173] 13. The computer-implemented method as described in any of Clauses 10 to 12, wherein the two or more input data streams include one or more of multimodal data, multivariate data, multicomponent data, or multisource data.

[0174] 14. The computer-implemented method as described in any one of Clauses 10 to 13, further comprising: training the at least one neural network using self-supervision.

[0175] 15. A system comprising: one or more processors, the one or more processors being configured to: generate at least one latent representation of two or more input data streams using a plurality of machine learning processes arranged in a hierarchical manner; and to perform one or more tasks based at least in part on the at least one latent representation.

[0176] 16. The system of claim 15, wherein the plurality of machine learning processes are at least used to: obtain a plurality of latent representations based at least in part on the two or more input data streams; and to align and merge at least two of the plurality of latent representations to generate the at least one latent representation.

[0177] 17. The system as described in Clause 15 or Clause 16, wherein the one or more tasks include at least one of: detecting one or more anomalies, predicting one or more future events, making one or more decisions, identifying at least one pattern in the two or more input data streams, classifying information, or performing regression.

[0178] 18. The system as described in any of Clauses 15 to 17, wherein the two or more input data streams transmit information relating to one or more devices in a data center.

[0179] 19. The system as described in Clause 18, wherein the two or more input data streams include at least one of multimodal data, multivariate data, multicomponent data, or multisource data.

[0180] 20. The system as described in any one of Clauses 15 to 19, wherein the plurality of machine learning processes are a first plurality of machine learning processes defining a first hierarchical structure, a second plurality of machine learning processes defining a second hierarchical structure, and a third plurality of machine learning processes defining a third hierarchical structure, the third hierarchical structure being configured to receive the outputs of the first hierarchical structure and the second hierarchical structure as input, the first hierarchical structure, the second hierarchical structure, and the third hierarchical structure collectively constituting at least a portion of a combined structure, the combined structure being trained using at least one of supervised learning or unsupervised learning.

[0181] In at least one embodiment, a single semiconductor platform may refer to a unique, single semiconductor-based integrated circuit or chip. In at least one embodiment, a multi-chip module with increased connectivity can be used, simulating on-chip operations and representing a substantial improvement over conventional implementations utilizing a central processing unit (“CPU”) and bus. In at least one embodiment, various modules may also be placed individually or in various combinations of semiconductor platforms, depending on user requirements.

[0182] In at least one embodiment, a computer program in the form of machine-readable executable code or computer control logic algorithms is stored in main memory and / or secondary storage, such as those described herein. According to at least one embodiment, if executed by one or more processors, the computer program enables at least one system described herein to perform various functions. In at least one embodiment, memory, storage, and / or any other storage are possible examples of computer-readable media. In at least one embodiment, secondary storage can refer to any suitable storage device or system, such as hard disk drives and / or removable storage drives, representing floppy disk drives, magnetic tape drives, optical disk drives, digital versatile disc (“DVD”) drives, recording devices, Universal Serial Bus (“USB”) flash memory, etc. In at least one embodiment, the architecture and / or functionality of the various preceding figures are implemented within the context of the CPU described herein, the parallel processing system described herein, an integrated circuit capable of having at least a portion of the capabilities of two CPUs, a parallel processing system, a chipset (e.g., a set of integrated circuits designed to operate as units performing related functions and sold), and / or any suitable combination of one or more integrated circuits.

[0183] In at least one embodiment, the architecture and / or functionality of the various preceding figures are implemented within the context of a general-purpose computer system, a circuit board system, a game console system dedicated to entertainment purposes, a special-purpose system, etc. In at least one embodiment, the computer system described herein can take the form of a desktop computer, laptop computer, tablet computer, server, supercomputer, smartphone (e.g., wireless, handheld device), personal digital assistant (“PDA”), digital camera, vehicle, head-mounted display, handheld electronic device, mobile phone device, television, workstation, game console, embedded system, and / or any other type of logic. In at least one embodiment, the computer system includes or relates to any device shown in the figures and / or described herein.

[0184] In at least one embodiment, the parallel processing system includes, but is not limited to, multiple parallel processing units (“PPUs”) and associated memory. In at least one embodiment, the PPUs are connected to a host processor or other peripheral device via interconnects and switches or multiplexers. In at least one embodiment, the parallel processing system distributes computational tasks across parallelizable PPUs, for example, as part of a computational task distribution across multiple graphics processing units (“GPUs”) thread blocks. In at least one embodiment, memory is shared and accessed (e.g., for read and / or write access) among some or all of the PPUs; however, such shared memory may incur performance penalties relative to using local memory and registers residing on the PPU. In at least one embodiment, the operation of the PPUs is synchronized using commands such as `syncthreads()`, where all threads in a block (e.g., executing across multiple PPUs) arrive at a certain point of code execution before proceeding.

[0185] In at least one embodiment, one or more techniques described herein utilize the oneAPI programming model. In at least one embodiment, the oneAPI programming model refers to a programming model for interacting with various computing accelerator architectures. In at least one embodiment, oneAPI refers to an application programming interface (API) designed for interacting with various computing accelerator architectures. In at least one embodiment, the oneAPI programming model utilizes the DPC++ programming language. In at least one embodiment, the DPC++ programming language refers to a high-level language used to achieve data-parallel programming productivity. In at least one embodiment, the DPC++ programming language is at least partially based on the C and / or C++ programming languages. In at least one embodiment, the oneAPI programming model is a programming model such as those developed by Intel Corporation of Santa Clara, California.

[0186] In at least one embodiment, oneAPI and / or the oneAPI programming model are used to interact with various accelerators, GPUs, processors, and / or their variations and architectures. In at least one embodiment, oneAPI includes a set of libraries that implement various functionalities. In at least one embodiment, oneAPI includes at least the oneAPI DPC++ library, the oneAPI Math Kernel library, the oneAPI Data Analysis library, the oneAPI Deep Neural Network library, the oneAPI Collective Communication library, the oneAPI Thread Building Block library, the oneAPI Video Processing library, and / or their variations.

[0187] In at least one embodiment, the oneAPI DPC++ library, also known as oneDPL, is a library that implements algorithms and functions to accelerate DPC++ kernel programming. In at least one embodiment, oneDPL implements one or more Standard Template Library (STL) functions. In at least one embodiment, oneDPL implements one or more parallel STL functions. In at least one embodiment, oneDPL provides a set of library classes and functions, such as parallel algorithms, iterators, function object classes, range-based APIs, and / or variations thereof. In at least one embodiment, oneDPL implements one or more classes and / or functions from the C++ Standard Library. In at least one embodiment, oneDPL implements one or more random number generator functions.

[0188] In at least one embodiment, the oneAPI math kernel library, also known as oneMKL, is a library that implements various optimized and parallelized routines for various mathematical functions and / or operations. In at least one embodiment, oneMKL implements one or more Basic Linear Algebra Subroutines (BLAS) and / or Linear Algebra Packages (LAPACK) dense linear algebra routines. In at least one embodiment, oneMKL implements one or more sparse BLAS linear algebra routines. In at least one embodiment, oneMKL implements one or more random number generators (RNGs). In at least one embodiment, oneMKL implements one or more vector mathematics (VM) routines for performing mathematical operations on vectors. In at least one embodiment, oneMKL implements one or more Fast Fourier Transform (FFT) functions.

[0189] In at least one embodiment, the oneAPI data analytics library, also known as oneDAL, is a library for implementing various data analytics applications and distributed computing. In at least one embodiment, oneDAL implements various algorithms for preprocessing, transformation, analysis, modeling, verification, and decision-making in batch, online, and distributed computing processing modes. In at least one embodiment, oneDAL implements various C++ and / or Java APIs and various connectors to one or more data sources. In at least one embodiment, oneDAL implements the DPC++ API extension to the traditional C++ interface, enabling GPUs to be used for various algorithms.

[0190] In at least one embodiment, the oneAPI deep neural network library, also known as oneDNN, is a library that implements various deep learning functions. In at least one embodiment, oneDNN implements various neural networks, machine learning and deep learning functions, algorithms and / or variations thereof.

[0191] In at least one embodiment, the oneAPI collective communication library, also known as oneCCL, is a library that implements various applications for deep learning and machine learning workloads. In at least one embodiment, oneCCL is built on lower-level communication middleware such as Message Passing Interface (MPI) and libfabrics. In at least one embodiment, oneCCL enables a set of deep learning-specific optimizations, such as priority, persistent operations, out-of-order execution, and / or variations thereof. In at least one embodiment, oneCCL implements various CPU and GPU functionalities.

[0192] In at least one embodiment, the oneAPI thread building block library, also known as oneTBB, is a library for implementing various parallelized processes for various applications. In at least one embodiment, oneTBB is used for host-based task-based shared parallel programming. In at least one embodiment, oneTBB implements general-purpose parallel algorithms. In at least one embodiment, oneTBB implements concurrent containers. In at least one embodiment, oneTBB implements a scalable memory allocator. In at least one embodiment, oneTBB implements a work-stealing task scheduler. In at least one embodiment, oneTBB implements low-level synchronization primitives. In at least one embodiment, oneTBB is compiler-independent and can be used with various processors, such as GPUs, PPUs, CPUs, and / or variations thereof.

[0193] In at least one embodiment, the oneAPI video processing library, also known as oneVPL, is a library for accelerating video processing in one or more applications. In at least one embodiment, oneVPL implements various video decoding, encoding, and processing functions. In at least one embodiment, oneVPL implements various functions for media pipelines on CPUs, GPUs, and other accelerators. In at least one embodiment, oneVPL implements device discovery and selection in media hubs and video analytics workloads. In at least one embodiment, oneVPL implements API primitives for zero-copy buffer sharing.

[0194] In at least one embodiment, the oneAPI programming model utilizes the DPC++ programming language. In at least one embodiment, the DPC++ programming language is a programming language that includes, but is not limited to, a functionally similar version of a CUDA mechanism for defining device code and distinguishing between device code and host code. In at least one embodiment, the DPC++ programming language may include a subset of the functionality of a CUDA programming language. In at least one embodiment, one or more CUDA programming model operations are performed using the oneAPI programming model with the DPC++ programming language.

[0195] In at least one embodiment, any application programming interface (API) described herein is compiled by a compiler, interpreter, or other software tool into one or more instructions, operations, or other signals. In at least one embodiment, compilation includes generating one or more machine-executable instructions, operations, or other signals from source code. In at least one embodiment, an API compiled into one or more instructions, operations, or other signals, when executed, causes one or more processors (such as a graphics processor, graphics core, parallel processor, CPU, or any other logic circuitry further described herein) to perform one or more computational operations.

[0196] It should be noted that while the example embodiments described herein may relate to the CUDA programming model, the techniques described herein can be used with any suitable programming model, such as HIP, oneAPI, and / or variations thereof.

[0197] Other variations are within the spirit of this disclosure. Therefore, although the disclosed technology is readily adaptable to various modifications and alternative constructions, certain embodiments thereof are illustrated in the accompanying drawings and have been described in detail above. However, it should be understood that the disclosure is not intended to be limited to one or more specific forms disclosed, but rather, it is intended to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of this disclosure as defined in the appended claims.

[0198] Unless otherwise stated or obviously contradicted by the context, the terms “a,” “an,” and “the,” and similar pronouns, used in the context of describing the disclosed embodiments (particularly in the context of the appended claims), should be interpreted as encompassing both singular and plural forms, rather than as definitions of the terms. Unless otherwise stated, the terms “comprising,” “having,” “including,” and “containing” should be interpreted as open-ended terms (meaning “including, but not limited to”). The term “connection” (wherein it is not modified, it refers to a physical connection) should be interpreted as partially or wholly contained, attached to, or connected together, even with some intervening elements. Unless otherwise indicated herein, references to numerical ranges herein are intended only as a way of abbreviating each individual value falling within that range, and each individual value is incorporated into the specification as if it were separately described herein. In at least one embodiment, unless otherwise indicated or contradicted by the context, the use of the terms “set” (e.g., “item set”) or “subset” should be interpreted as a non-empty set comprising one or more members. Furthermore, unless otherwise indicated or contradicted by the context, the term “subset” of the corresponding set does not necessarily mean an appropriate subset of the corresponding set, but rather that the subset and the corresponding set can be equal.

[0199] Unless otherwise explicitly stated or clearly contradicted by the context, connective phrases such as “at least one of A, B, and C” or “at least one of A, B, and C” are understood in the context to generally refer to items, terms, etc., which can be A or B or C, or any non-empty subset of the set A, B, and C. For example, in an illustrative example of a set with three members, the connective phrases “at least one of A, B, and C” and “at least one of A, B, and C” refer to any of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Therefore, such connective language is generally not intended to imply that some embodiments require the presence of each of at least one of A, at least one of B, and at least one of C. Additionally, unless otherwise stated or contradicted by the context, the term “multiple” indicates a plural state (e.g., “multiple items” indicates multiple items). In at least one embodiment, the number of items in the multiple items is at least two, but may be more if explicitly indicated or indicated by the context. Furthermore, unless otherwise stated or clearly understood from the context, the phrase “based on” means “at least partially based on” rather than “based on only”.

[0200] Unless otherwise indicated herein or clearly contradicted by the context, the operations of the processes described herein may be performed in any suitable order. In at least one embodiment, processes such as those described herein (or variations thereof and / or combinations thereof) are executed under the control of one or more computer systems configured with executable instructions and are implemented as code (e.g., executable instructions, one or more computer programs, or one or more application programs) executed jointly by hardware or a combination thereof on one or more processors. In at least one embodiment, the code is stored on a computer-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. In at least one embodiment, the computer-readable storage medium is a non-transitory computer-readable storage medium that excludes transient signals (e.g., propagating transient electrical or electromagnetic transmissions) but includes non-transitory data storage circuitry (e.g., buffers, caches, and queues) within a transceiver that includes transient signals. In at least one embodiment, code (e.g., executable code or source code) is stored on a set of one or more non-transitory computer-readable storage media (or other memory for storing executable instructions) on which executable instructions are stored, which, when executed by one or more processors of a computer system (i.e., as a result of execution), cause the computer system to perform the operations described herein. In at least one embodiment, the set of non-transitory computer-readable storage media comprises a plurality of non-transitory computer-readable storage media, and one or more of the various non-transitory storage media lack the complete code, instead the plurality of non-transitory computer-readable storage media collectively store the complete code. In at least one embodiment, the executable instructions are executed such that different instructions are executed by different processors; for example, the non-transitory computer-readable storage media store the instructions, and the main central processing unit (“CPU”) executes some instructions while the graphics processing unit (“GPU”) executes other instructions. In at least one embodiment, different components of the computer system have separate processors, and the different processors execute different subsets of the instructions.

[0201] In at least one embodiment, an arithmetic logic unit is a set of combinational logic circuits that takes one or more inputs to produce a result. In at least one embodiment, a processor uses arithmetic logic units to implement mathematical operations, such as addition, subtraction, or multiplication. In at least one embodiment, arithmetic logic units are used to implement logical operations, such as logical AND / OR or XOR. In at least one embodiment, arithmetic logic units are stateless and made of physical switching elements, such as semiconductor transistors arranged to form logic gates. In at least one embodiment, arithmetic logic units may operate internally as stateful logic circuits with an associated clock. In at least one embodiment, arithmetic logic units may be configured as asynchronous logic circuits whose internal state is not maintained in an associated set of registers. In at least one embodiment, a processor uses arithmetic logic units to combine operands stored in one or more registers of the processor and produce an output that can be stored by the processor in another register or memory location.

[0202] In at least one embodiment, as a result of processing instructions retrieved by the processor, the processor presents one or more inputs or operands to the arithmetic logic unit (ALU), causing the ALU to produce a result at least partially based on instruction code provided to the ALU of the inputs. In at least one embodiment, the instruction code provided by the processor to the ALU is at least partially based on instructions executed by the processor. In at least one embodiment, combinational logic in the ALU processes the inputs and produces an output placed on a bus within the processor. In at least one embodiment, the processor selects a destination register, memory location, output device, or output storage location on the output bus to clock the processor such that the result produced by the ALU is sent to the desired location.

[0203] Within the scope of this application, the term Arithmetic Logic Unit or ALU is used to refer to any computational logic circuit that processes operands to produce results. For example, in this document, the term ALU may refer to a floating-point unit, a DSP, a tensor core, a shader core, a coprocessor, or a CPU.

[0204] In at least one embodiment, one or more components of the disclosed system and / or processor may communicate with one or more CPUs, ASICs, GPUs, FPGAs, or other hardware, circuitry, or integrated circuit components, including, for example, amplifiers or upsamplers for amplifying images, image fusioners or image fusioner components for fusing, mixing, or adding images together, samplers for sampling images (e.g., as part of a DSP), neural network circuitry configured to perform amplifiers to amplify images (e.g., from low-resolution images to high-resolution images), or other hardware for modifying or generating images, frames, or videos to adjust their resolution, size, or pixels; one or more components of the disclosed system and / or processor may use the components described in this disclosure to perform methods, operations, or instructions for generating or modifying images.

[0205] Therefore, in at least one embodiment, the computer system is configured to implement one or more services that perform the operations of the processes described herein, either individually or collectively, and such a computer system is configured with suitable hardware and / or software to enable the performance of the operations. Furthermore, the computer system implementing at least one embodiment of this disclosure is a single device, and in another embodiment it is a distributed computer system comprising multiple devices operating differently, such that the distributed computer system performs the operations described herein, and that a single device does not perform all the operations.

[0206] The use of any and all examples or exemplary language (e.g., “such as”) provided herein is intended only to better illustrate embodiments of this disclosure and does not impose a limitation on the scope of the disclosure unless otherwise required. No language in the specification should be construed as indicating that any unclaimed element is essential to the practice of the disclosure.

[0207] All references cited in this article, including publications, patent applications and patents, are incorporated herein by reference to the same extent that each reference is individually and specifically indicated as incorporated herein by reference and the entire contents of which are described herein.

[0208] The terms “coupled” and “connected”, and their derivatives, may be used in the specification and claims. It should be understood that these terms may not be intended to be synonyms with each other. Rather, in certain examples, “connected” or “coupled” may be used to indicate that two or more elements are in direct or indirect physical or electrical contact with each other. “Coupled” may also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other.

[0209] Unless otherwise expressly stated, it will be understood that throughout this specification, terms such as “processing,” “calculation,” “operation,” “determine,” etc., refer to the actions and / or processes of a computer or computing system or similar electronic computing device that manipulate and / or convert data represented as physical quantities (e.g., electronic quantities) in the registers and / or memory of the computing system into other data similarly represented as physical quantities in the memory, registers, or other such information storage, transmission, or display devices of the computing system.

[0210] Similarly, the term "processor" can refer to any device or part of a device that processes electronic data from registers and / or memory and converts that electronic data into other electronic data that can be stored in registers and / or memory. As a non-limiting example, a "processor" can be a CPU or a GPU. A "computing platform" can include one or more processors. As used herein, a "software" process can include, for example, software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Likewise, each process can refer to multiple processes that execute instructions sequentially or in parallel, continuously or intermittently. In at least one embodiment, the terms "system" and "method" are used interchangeably herein, provided that a system can embody one or more methods, and a method can be considered a system.

[0211] In this document, reference may be made to obtaining, acquiring, receiving, or inputting analog or digital data into a subsystem, computer system, or computer-implemented machine. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog and digital data can be accomplished in various ways, such as by receiving data as a parameter to a function call or a call to an application programming interface. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog or digital data can be accomplished by transmitting data via a serial or parallel interface. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog or digital data can be accomplished by transmitting data from a providing entity to an acquiring entity via a computer network. In at least one embodiment, reference may also be made to providing, outputting, transmitting, sending, or presenting analog or digital data. In various examples, the process of providing, outputting, transmitting, sending, or presenting analog or digital data can be implemented by transmitting data as an input or output parameter to a function call, an application programming interface, or an inter-process communication mechanism.

[0212] While this document describes example implementations of the described technologies, other architectures can be used to implement the described functionality and are intended to fall within the scope of this disclosure. Furthermore, although specific assignments of responsibilities have been defined above for descriptive purposes, various functions and responsibilities may be assigned and divided in different ways depending on the circumstances.

[0213] Furthermore, although the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter claimed in the appended claims is not necessarily limited to the specific features or actions described. Rather, specific features and actions are disclosed as exemplary forms for implementing the claims.

Claims

1. A processor, the processor comprising: One or more circuits, said one or more circuits being used for at least: Multiple neural networks arranged in a hierarchical manner are used to generate at least one latent representation of two or more input data streams, wherein the two or more input data streams have different modalities.

2. The processor of claim 1, wherein the plurality of neural networks comprises: A first set of neural networks is used to generate a first set of latent representations using the two or more input data streams; And a second set of neural networks, used to generate a second set of latent representations using the first set of latent representations. The first set of potential representations is used to correspond to the first data resolution, and The second set of potential representations is used to correspond to a second data resolution, wherein the second data resolution is lower than the first data resolution.

3. The processor of claim 2, wherein the first data resolution represents different devices in a data center, and the second data resolution represents a combination of two or more devices.

4. The processor of claim 1, wherein the plurality of neural networks are at least used for: Multiple first latent representations are obtained, at least in part, based on the two or more input data streams; Align and merge at least two of the plurality of first latent representations to generate a plurality of second latent representations; and At least two of the plurality of second potential representations are aligned and merged to generate the at least one potential representation.

5. The processor of claim 4, wherein at least two of the plurality of first potential representations correspond to the same device.

6. The processor of claim 4, wherein different portions of the at least two second potential representations of the plurality of second potential representations correspond to different devices.

7. The processor of claim 1, wherein the one or more circuits are at least used for: One or more predictions are generated, at least in part, based on patterns identified from the at least one latent representation.

8. The processor of claim 1, wherein the two or more input data streams include at least one of multimodal data, multivariate data, multicomponent data, or multisource data.

9. The processor of claim 1, wherein at least one of the plurality of neural networks is trained using self-supervised training.

10. A computer-implemented method, the computer-implemented method comprising: A sequence of latent vectors is generated by aligning and merging at least two latent representations, wherein the at least two latent representations are obtained at least in part based on two or more input data streams; At least one latent representation is generated using at least one neural network, based at least in part on the latent vector sequence; and One or more tasks are performed, at least in part, based on the at least one potential representation.

11. The computer-implemented method of claim 10, wherein the two or more input data streams transmit information related to one or more devices in a data center.

12. The computer-implemented method of claim 10, wherein the one or more tasks include: One or more predictions are generated, at least in part, based on patterns identified using the at least one latent representation.

13. The computer-implemented method of claim 10, wherein the two or more input data streams include one or more of multimodal data, multivariate data, multicomponent data, or multisource data.

14. The computer-implemented method of claim 10, further comprising: The at least one neural network is trained using self-supervised training.

15. A system comprising: One or more processors, said one or more processors being used for at least: Multiple machine learning processes arranged in a hierarchical manner are used to generate at least one latent representation of two or more input data streams; and One or more tasks are performed, at least in part, based on the at least one potential representation.

16. The system of claim 15, wherein the plurality of machine learning processes are used for at least: Multiple latent representations are obtained, at least in part, based on the two or more input data streams; and At least two of the plurality of potential representations are aligned and merged to generate the at least one potential representation.

17. The system of claim 15, wherein the one or more tasks include at least one of: detecting one or more anomalies, predicting one or more future events, making one or more decisions, identifying at least one pattern in the two or more input data streams, classifying information, or performing regression.

18. The system of claim 15, wherein the two or more input data streams transmit information related to one or more devices in the data center.

19. The system of claim 18, wherein the two or more input data streams include at least one of multimodal data, multivariate data, multicomponent data, or multisource data.

20. The system of claim 15, wherein the plurality of machine learning processes are a first plurality of machine learning processes defining a first hierarchical structure, a second plurality of machine learning processes defining a second hierarchical structure, and a third plurality of machine learning processes defining a third hierarchical structure, the third hierarchical structure being configured to receive the outputs of the first hierarchical structure and the second hierarchical structure as input, the first hierarchical structure, the second hierarchical structure, and the third hierarchical structure together constituting at least a portion of a combined structure, the combined structure being trained using at least one of supervised learning or unsupervised learning.

Images