Information processing system, information processing method, and program

Abstract

This information processing system comprises at least one first memory, at least one first processor connected to the at least one first memory, at least one second memory, and at least one second processor connected to the at least one second memory. The first processor generates intermediate data by inputting input information into a machine learning model. The second processor uses at least a portion of the intermediate data generated by the first processor to generate output information of the machine learning model. The ratio of the data transfer rate of the first memory to the operation speed of the first processor is less than the ratio of the data transfer rate of the second memory to the operation speed of the second processor.

Description

Information Processing System, Information Processing Method, and Program

[0001] The present disclosure relates to an information processing system, an information processing method, and a program.

[0002] Machine learning models such as large language models (LLMs) are known. To efficiently execute the inference process of large language models, there is a technique for generating intermediate data called a key value cache.

[0003] Insu Jang, "LLM Inference: Continuous Batching and PagedAttention", [online], [Searched on August 29, 2024], Internet <URL: https: / / insujang.github.io / 2024-01-07 / llm-inference-continuous-batching-and-pagedattention / >

[0004] An object of the present disclosure is to provide a technique for quickly generating output information of a machine learning model.

[0005] An information processing system according to an aspect of the present disclosure includes at least one first memory, at least one first processor connected to the at least one first memory, at least one second memory, and at least one second processor connected to the at least one second memory. The at least one first processor generates intermediate data by inputting input information into a machine learning model. The at least one second processor generates output information of the machine learning model using at least a part of the intermediate data generated by the at least one first processor. The ratio of the data transfer speed of the at least one first memory to the operation speed of the at least one first processor is lower than the ratio of the data transfer speed of the at least one second memory to the operation speed of the at least one second processor.

[0006] Figure 1 is a diagram illustrating an example of the inference process of a machine learning model. Figure 2 is a block diagram showing the first example of the overall configuration of an information processing system. Figure 3 is a block diagram showing the second example of the overall configuration of an information processing system. Figure 4 is a block diagram showing the third example of the overall configuration of an information processing system. Figure 5 is a flowchart showing an example of the inference process of a machine learning model. Figure 6 is a block diagram showing an example of the computer hardware configuration.

[0007] Hereinafter, embodiments of this disclosure will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0008] [First Embodiment] The first embodiment of the present disclosure is an example of an information processing system that performs a predetermined task based on a machine learning model. In this embodiment, the machine learning model may include a neural network having an attention mechanism. The neural network having an attention mechanism may be, for example, a neural network also called a Transformer.

[0009] Machine learning models are not limited to Transformers. For example, a machine learning model may be a state-space model (SSM). A machine learning model may consist of multiple machine learning models. For example, a machine learning model may include Mamba, Jamba, etc. Any machine learning model that generates intermediate data in the inference process is acceptable.

[0010] The inference process of a transformer, an example of a machine learning model, will be explained with reference to Figure 1. Figure 1 is a diagram illustrating an example of the inference process of a machine learning model.

[0011] As shown in Figure 1, the inference process of a transformer can be broadly divided into a prefill process and a decode process. The prefill process generates a key-value cache based on input information called a prompt. The decode process generates output information using the key-value cache generated in the prefill process. The prefill process is an example of the first process. The decode process is an example of the second process.

[0012] In the prefill process, the input prompt is divided into tokens, and each token is input to the transformer one by one from the beginning. The transformer performs a forward process for each input token. In the forward process, the transformer calculates a query vector, key vector, and value vector for each token. The transformer outputs one token for each input token.

[0013] The transformer caches an array of key vectors and value vectors calculated for each token. This cached array of key vectors and value vectors is called the key-value cache. A key-value cache is generated for each layer that makes up the transformer. The key-value cache for each layer is calculated sequentially, starting from the layer closest to the input layer and moving towards the output layer. In the example in Figure 1, the transformer includes N hidden layers, and key-value caches are generated sequentially from the first layer to the Nth layer.

[0014] A token is a processing unit used by machine learning models to process electronic data, and the amount of data in one token can vary depending on the design of the machine learning model. For example, a token may represent a single Japanese character or a single English word, but depending on its frequency of occurrence, one character may be represented by two tokens, or two or more characters may be represented by one token.

[0015] A cache, for example, is a technique that copies recently read or frequently used data from a slow storage device to a high-speed storage device. The high-speed storage device that copies the data, or the copied data itself, is sometimes also called a cache. In other words, a key-value cache may include the process by which a machine learning model stores key vectors and value vectors, may include the stored key vectors and value vectors, or may include the storage area in which the key vectors and value vectors are stored.

[0016] In the decoding process, the token last output in the prefill process is input to the transformer. The transformer uses the key-value cache generated in the prefill process to perform a forward operation on the input token. The token output by the forward operation is input to the transformer again. The decoding process generates output information for the prompt by repeatedly performing a forward operation on the token output in the previous forward operation. In other words, the decoding process generates output information for the prompt by performing an autoregressive forward operation.

[0017] The prefill process has low computation order dependency, allowing for parallel forward processing for each input token. On the other hand, the decode process has high computation order dependency, requiring sequential forward processing and resulting in a larger amount of data being read from and written to memory. Therefore, executing the decode process requires more processor memory bandwidth per unit of computation compared to executing the prefill process. In other words, when executing both the prefill and decode processes, the prefill process is computationally limited, while the decode process is memory-limited.

[0018] Note that the prefill process is an example of the first process, and the decode process is an example of the second process. However, the first process may be one in which the degree of computational rate limitation is relatively greater than that of the second process, and the second process may be one in which the degree of memory rate limitation is relatively greater than that of the first process.

[0019] In conventional technology, the inference process of a machine learning model is executed on the same processor. This includes using multiple processors of the same type. When the prefill and decode processes are executed on the same processor, memory bandwidth is wasted during the prefill process, and computational performance is wasted during the decode process. In other words, executing the inference process of a machine learning model on the same processor prevents effective utilization of the processor's performance. Therefore, there is room for improvement in the efficiency of the machine learning model inference process.

[0020] This embodiment provides a technique for rapidly generating output information for a machine learning model. In this embodiment, the information processing system comprises two or more processors with different ratios of data transfer rate to computation speed. The first processor, which has a relatively high ratio of data transfer rate to computation speed (i.e., high computational performance), inputs input information to a machine learning model to generate intermediate data and transmits at least a portion of the intermediate data to the second processor. The second processor, which has a relatively low ratio of data transfer rate to computation speed (i.e., a large memory bandwidth), uses at least a portion of the intermediate data received from the first processor to generate output information for the machine learning model.

[0021] The generation of intermediate data is computationally limited. Therefore, using a processor with high computational performance allows for the rapid generation of intermediate data. The generation of output information is memory-limited. Therefore, using a processor with a large memory bandwidth allows for the rapid generation of output information. In one aspect, according to this embodiment, the performance of the processor can be effectively utilized through the inference processing of the machine learning model, thus enabling the rapid generation of output information for the machine learning model.

[0022] <Overall Configuration of the Information Processing System> The overall configuration of the information processing system according to this embodiment will be explained with reference to Figures 2 to 4.

[0023] Figure 2 is a block diagram showing a first example of the overall configuration of an information processing system. As shown in Figure 2, the information processing system 1000 may include one or more server devices 10. The information processing system 1000 may include multiple server devices 10.

[0024] The server device 10 may include one or more processing units 100 and one or more processing units 200. The server device 10 may include multiple processing units 100. The server device 10 may include multiple processing units 200. The number of processing units 100 and the number of processing units 200 in the server device 10 may be the same or different.

[0025] If the server device 10 includes multiple processing units 100, the multiple processing units 100 may form a cluster. The multiple processing units 100 constituting the cluster may perform parallel processing. The multiple processing units 100 may combine the parallel processed data and transmit it to the processing unit 200.

[0026] The number of processing units 200 provided by the server device 10 may be greater than the number of processing units 100 provided by the server device 10. For example, the server device 10 may include one processing unit 100 and multiple processing units 200. If the server device 10 includes multiple processing units 200, the multiple processing units 200 may form a cluster. The multiple processing units 200 that make up the cluster may perform parallel processing. If the server device 10 includes multiple processing units 200, the processing unit 100 may transmit data to each of the multiple processing units 200.

[0027] The processing unit 100 may comprise one or more processors 110 and one or more memories 120. The processing unit 100 may comprise multiple processors 110 and multiple memories 120. The processors 110 and memories 120 may be electrically connected. The processors 110 and memories 120 may be connected to each other in a manner that enables data communication.

[0028] The processing unit 200 may include one or more processors 210 and one or more memories 220. The processing unit 200 may include multiple processors 210 and multiple memories 220. The processors 210 and memories 220 may be electrically connected. The processors 210 and memories 220 may be connected to each other so as to enable data communication.

[0029] The processing unit 100 and the processing unit 200 may be connected to bus B of the server device 10. The processing unit 100 and the processing unit 200 may be configured to communicate data with each other via bus B. Bus B may be a dedicated transmission path (interconnect) that enables data communication between the processing unit 100 and the processing unit 200. Bus B may also be the system bus provided by the server device 10.

[0030] The processing unit 100 and the processing unit 200 may include processing circuits that can perform specific calculations at high speed. The processing unit 100 and the processing unit 200 may also include processing circuits that can perform general-purpose calculations. Specific calculations may include, for example, multiply-accumulate operations, which are often used in training neural networks.

[0031] The processing units 100 and 200 may, for example, be a GPU package (e.g., a GPU board or GPU card) equipped with one or more graphics processing units (GPUs). The processing units 100 and 200 may, for example, be a CPU package (e.g., a CPU board or CPU card) equipped with one or more central processing units (CPUs). The processing units 100 and 200 may also be equipped with one or more various accelerators.

[0032] The processing unit 100 and the processing unit 200 may be of the same type or of different types. For example, the processing unit 100 may be equipped with a GPU package or an accelerator, and the processing unit 200 may be equipped with a CPU package. For example, the processing unit 100 may be equipped with a first type of GPU package, and the processing unit 200 may be equipped with a second type of GPU package. For example, the processing unit 100 may be equipped with a first type of accelerator, and the processing unit 200 may be equipped with a second type of accelerator.

[0033] Memory 120 may be a storage device that stores data used in calculations performed by processor 110 and data obtained from calculations performed by processor 110. Memory 220 may be a storage device that stores data used in calculations performed by processor 210 and data obtained from calculations performed by processor 210. Memory 120 and memory 220 may, for example, be DRAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory). Memory 120 and memory 220 may, for example, be a three-dimensional memory structure with multiple stacked units.

[0034] Memory 120 and memory 220 may store parameters of the machine learning model. Memory 120 and memory 220 may store at least some of the parameters of the machine learning model. Memory 120 and memory 220 may store all of the parameters of the machine learning model. The parameters stored in memory 120 and the parameters stored in memory 220 may be the same or different. Memory 120 may store parameters required for processing performed by processor 110. Memory 220 may store parameters required for processing performed by processor 210.

[0035] The processing unit 100 has a lower B / F ratio (Bytes / FLOPS) compared to the processing unit 200. Conversely, the processing unit 200 has a higher B / F ratio compared to the processing unit 100. The B / F ratio is the ratio of memory bandwidth to computational performance. Memory bandwidth is one example of an indicator of memory data transfer speed. Data transfer speed may, for example, be the amount of data that the processor can read and write to memory per unit time. Computational performance is one example of an indicator of computation speed. Computational performance may, for example, be the number of floating-point operations that the processor can perform per unit time. Computational performance may vary depending on the precision of the floating-point number resulting from the calculation.

[0036] In other words, the ratio of the data transfer rate of the memory 120 to the processing speed of the processor 110 (data transfer rate / processing speed) is lower than the ratio of the data transfer rate of the memory 220 to the processing speed of the processor 210. Conversely, the ratio of the data transfer rate of the memory 220 to the processing speed of the processor 210 is higher than the ratio of the data transfer rate of the memory 120 to the processing speed of the processor 110.

[0037] The difference between the B / F ratio of the processing unit 100 and the B / F ratio of the processing unit 200 can be arbitrary. For example, the difference between the B / F ratio of the processing unit 100 and the B / F ratio of the processing unit 200 may be as much as 10 times. For example, the B / F ratio of the processing unit 100 may be about 1:100 (=0.001), and the B / F ratio of the processing unit 200 may be about 1:100 (=0.01).

[0038] The memory data transfer speed and processor calculation speed may be theoretical performance or performance when performing a predetermined process. The theoretical performance may be the catalog values ​​for the memory and processor. The predetermined process may include processing related to a machine learning model. As an example, the predetermined process may include inference processing of a machine learning model, or one or more operations included in the inference processing.

[0039] The ratio of data transfer rate to computational performance may be a value evaluated over the entire process, or a value evaluated per unit time when the process is executed. In other words, the ratio of data transfer rate to computational performance may be the ratio of the total amount of data transferred over the entire process related to the machine learning model to the total number of computations performed over the entire process related to the machine learning model. Alternatively, the ratio of data transfer rate to computational performance may be the ratio of the amount of data transferred per unit time when the process related to the machine learning model is executed to the number of computations performed per unit time when the process related to the machine learning model is executed.

[0040] The ratio of data transfer rate to computational performance may be a value evaluated according to the combination of the number of processors and the number of memory units used when executing the process. As a first example, the ratio of data transfer rate to computational performance may be the performance when executing the process using one memory unit and one processor, and may also be the theoretical performance in this case. As a second example, the ratio of data transfer rate to computational performance may be the performance when executing the process using one memory unit and two or more processors, and may also be the theoretical performance in this case. As a third example, the ratio of data transfer rate to computational performance may be the performance when executing the process using two or more memory units and one processor, and may also be the theoretical performance in this case. As a fourth example, the ratio of data transfer rate to computational performance may be the performance when executing the process using two or more memory units and two or more processors, and may also be the theoretical performance in this case.

[0041] Figure 3 is a block diagram showing a second example of the overall configuration of an information processing system. As shown in Figure 3, the information processing system 1000 may include one or more server devices 10 and one or more server devices 20. The information processing system 1000 may include multiple server devices 10. The information processing system 1000 may include multiple server devices 20. The number of server devices 10 and the number of server devices 20 included in the information processing system 1000 may be the same or different.

[0042] The server device 10 may include one or more processing units 100 and a network interface 130. The server device 10 may include multiple processing units 100. The processing units 100 may be configured in the same manner as in the first example.

[0043] The server device 20 may include one or more processing units 200 and a network interface 230. The server device 20 may include multiple processing units 200. The processing units 200 may be configured in the same manner as in the first example.

[0044] The network interface 130 and the network interface 230 may be connected to the communication network N. The processing device 100 and the processing device 200 may be configured to be capable of data communication with each other via the communication network N. The communication network N may include a dedicated communication path capable of data communication between the network interface 130 and the network interface 230. The communication network N may be, for example, a communication network such as a LAN (Local Area Network) or the Internet.

[0045] When the information processing system 1000 includes a plurality of server devices 10, a cluster may be configured by the plurality of server devices 10. The plurality of server devices 10 constituting the cluster may execute parallel processing. The plurality of server devices 10 may combine the data processed in parallel and transmit it to the server device 20.

[0046] The number of server devices 20 may be greater than the number of server devices 10. As an example, the information processing system 1000 may include one server device 10 and a plurality of server devices 20. When the information processing system 1000 includes a plurality of server devices 20, a cluster may be configured by the plurality of server devices 20. The plurality of server devices 20 constituting the cluster may execute parallel processing. When the information processing system 1000 includes a plurality of server devices 20, the server device 10 may transmit data to each of the plurality of server devices 20.

[0047] The processing device 100 and the processing device 200 may be configured in the same manner as in the first example. That is, the processing device 100 included in the server device 10 has a lower B / F ratio (Bytes / FLOPS) compared to the processing device 200 included in the server device 20. Conversely, the processing device 200 included in the server device 20 has a higher B / F ratio compared to the processing device 100 included in the server device 10.

[0048] In other words, the ratio of the data transfer speed of the memory 120 to the operation speed of the processor 110 (data transfer speed / operation speed) is lower than the ratio of the data transfer speed of the memory 220 to the operation speed of the processor 210. Conversely, the ratio of the data transfer speed of the memory 220 to the operation speed of the processor 210 is higher than the ratio of the data transfer speed of the memory 120 to the operation speed of the processor 110.

[0049] FIG. 4 is a block diagram showing a third example of the overall configuration of the information processing system. As shown in FIG. 4, the information processing system 1000 may include one or more server devices 10. The information processing system 1000 may include a plurality of server devices 10.

[0050] The server device 10 may include one or more processing devices 100. The server device 10 may include a plurality of processing devices 100.

[0051] The processing device 100 may include one or more processors 110, one or more processors 210, one or more memories 120, and one or more memories 220. The processing device 100 may include a plurality of processors 110. The processing device 100 may include a plurality of memories 120. The processing device 100 may include a plurality of processors 210. The processing device 100 may include a plurality of memories 220.

[0052] The processor 110 and the memory 120 may be electrically connected. The processor 110 and the memory 120 may be connected so as to be capable of data communication with each other. The processor 210 and the memory 220 may be electrically connected. The processor 210 and the memory 220 may be connected so as to be capable of data communication with each other.

[0053] The processor 110 and the processor 210 may be connected to a circuit C formed in the processing device 100. The processor 110 and the processor 210 may be configured to be capable of data communication with each other via the circuit C. The circuit C may include, for example, electronic components or wirings formed on the substrate of the processing device 100.

[0054] If the processing unit 100 includes multiple processors 110, the multiple processors 110 may perform parallel processing. The multiple processors 110 may combine the parallel processed data and transmit it to the processor 210.

[0055] The number of processors 210 may be greater than the number of processors 110. For example, the processing unit 100 may include one processor 110 and multiple processors 210. If the processing unit 100 includes multiple processors 210, the multiple processors 210 may perform parallel processing. If the processing unit 100 includes multiple processors 210, processor 110 may send data to each of the multiple processors 210.

[0056] The processors 110, 210 and the memories 120, 220 may be configured in the same manner as in the first example. That is, the ratio of the data transfer rate of the memory 120 to the processing speed of the processor 110 (data transfer rate / processing speed) is lower than the ratio of the data transfer rate of the memory 220 to the processing speed of the processor 210. Conversely, the ratio of the data transfer rate of the memory 220 to the processing speed of the processor 210 is higher than the ratio of the data transfer rate of the memory 120 to the processing speed of the processor 110.

[0057] Note that the overall configuration of the information processing system 1000 shown in Figures 2 to 4 is just one example, and various system configurations are possible depending on the application and purpose. The information processing system 1000 may consist of one or more devices. The devices included in the information processing system 1000 may be a system composed of multiple devices. Each function included in the information processing system 1000 may be implemented by any device that constitutes the system. Each component included in the information processing system 1000 may be included in any device that constitutes the system.

[0058] <Inference Processing Flow> The inference processing of the machine learning model executed by the information processing system 1000 will be explained with reference to Figure 5. Figure 5 is a flowchart showing an example of the machine learning model inference processing. Here, we will explain an example in which the information processing system 1000 shown in Figure 2 performs the machine learning model inference processing.

[0059] The information processing system 1000 stores the parameters of the machine learning model in memory 120 and memory 220 before executing the inference process of the machine learning model. The parameters of the machine learning model stored in memory 120 may be read from an auxiliary storage device built into the server device 10 or an external storage device connected to the server device 10. The parameters of the machine learning model stored in memory 220 may be read from an auxiliary storage device built into the server device 20 or an external storage device connected to the server device 20.

[0060] In step S1, the processing unit 100 receives input information for the machine learning model. The input information may, for example, be a natural language sentence called a prompt. The input information is not limited to text data. The input information may include at least one of audio data or image data. The image data may include still images and videos, and videos may include audio data. The input information may also be multimodal data including at least two of text data, audio data, or image data. The input information may be entered by a user of the information processing system 1000. The input information may be generated by the information processing system 1000 or an external information processing device or information processing system. Note that "external" means not included in the information processing system 1000.

[0061] For example, the processing unit 100 may receive input information received by the server device 10 from another device. The other device may be a terminal device operated by a user, or an external information processing device or information processing system that generated the input information. For example, the processing unit 100 may receive input information input to the server device 10.

[0062] In step S2, the processing unit 100 generates intermediate data based on the input information received in step S1. The processing unit 100 may also generate intermediate data by inputting the input information into a machine learning model. If the server device 10 includes multiple processing units 100, the processing units 100 may generate intermediate data by parallel processing. The processing unit 100 stores the intermediate data generated by the machine learning model in the memory 120.

[0063] The processing unit 100 may generate a key-value cache, which is an example of intermediate data, by inputting a prompt, which is an example of input information, into a transformer, which is an example of a machine learning model. The processing unit 100 may also generate a state in a state-space model, which is an example of intermediate data, by inputting a prompt, which is an example of input information, into a state-space model, which is an example of a machine learning model. The state in the state-space model may be represented by one or more vectors or tensors.

[0064] As an example, the processing unit 100 may perform a transformer prefill process. As another example, the processing unit 100 may calculate an embedded representation, which is another example of intermediate data, based on the image data contained in the input information.

[0065] The transformer performs a forward process for each input token. In the forward process, the transformer calculates a query vector, key vector, and value vector for each token. The transformer caches the array of key vectors and value vectors for each token. This generates a key-value cache. The transformer generates the key-value cache sequentially, starting from the input layer and moving towards the output layer.

[0066] The processing unit 100 may read at least some of the parameters of the machine learning model stored in the memory 120. The processing unit 100 may generate intermediate data based on at least some of the parameters read from the memory 120. The processing unit 100 may read at least some of the parameters of the machine learning model from the memory 120 each time it performs forward processing on an input token. The processing unit 100 may read at least some of the parameters of the machine learning model related to an input token from the memory 120.

[0067] In step S3, the processing unit 100 transmits the intermediate data generated in step S3 to the processing unit 200. The processing unit 100 may read the intermediate data stored in the memory 120 and transmit it to the processing unit 200 via the bus B. The processing unit 100 may transmit at least a portion of the intermediate data to the processing unit 200.

[0068] The processing unit 100 may directly transmit the intermediate data to the processing unit 200 (memory 220). The processing unit 100 may indirectly transmit the intermediate data to the processing unit 200 (memory 220). As an example of the transmission process, the processing unit 100 may store the intermediate data in a storage device and transmit information that allows it to identify the intermediate data stored in the storage device to the processing unit 200. The storage device that stores the intermediate data may be memory 120 or another memory. The processing unit 200 may use the information that allows it to identify the intermediate data to retrieve the intermediate data stored in the storage device. The processing unit 100 may also transmit the intermediate data to memory 220 via another device. In other words, the processing unit 100 can make the intermediate data available to the processing unit 200 by any means necessary.

[0069] If the server device 10 includes multiple processing units 100, the processing unit 100 may combine multiple intermediate data stored in multiple memories 120 and transmit the combined intermediate data to the processing unit 200. If the server device 10 includes multiple processing units 200, the processing unit 100 may transmit the intermediate data to each of the multiple processing units 200.

[0070] The processing unit 100 may start transmitting at least a portion of the generated intermediate data to the processing unit 200 before the generation of the intermediate data is completed. The processing unit 100 may transmit the key-value caches generated by the transformer layer by layer to the processing unit 200 layer by layer. For example, the processing unit 100 may start transmitting the generated key-value cache each time it generates a key-value cache for one layer, before the generation of key-value caches for all layers is completed. For another example, the processing unit 100 may start transmitting the key-value caches for all layers after the generation of key-value caches for all layers is completed.

[0071] In step S4, the processing unit 200 receives intermediate data from the processing unit 100. The processing unit 200 stores the intermediate data received from the processing unit 100 in the memory 220.

[0072] The information processing system 1000 may repeatedly execute the processes from step S2 to step S4. For example, the processing unit 100 may repeatedly generate a key-value cache for each layer of the transformer, and each time the generation of the key-value cache for one layer is completed, it may send the generated key-value cache to the processing unit 200, for the total number of transformers.

[0073] In step S5, the processing unit 100 generates output information for the input information received in step S1. The processing unit 100 may also obtain output information output by the machine learning model after the generation of intermediate data is complete. For example, in the prefill process of the transformer, the processing unit 100 may obtain the last token output by forward processing on the last token of the prompt. The processing unit 100 transmits the generated output information to the processing unit 200.

[0074] The output information generated during the prefill process is not limited to text data (tokens). The output information may include at least one of audio data or image data. Image data may include still images and videos, and videos may include audio data. The output information may also be multimodal data including at least two of text data, audio data, or image data.

[0075] In step S6, the processing unit 200 receives output information from the processing unit 100. The processing unit 200 may store the output information received from the processing unit 100 in the memory 220. The processing unit 200 may output the output information received from the processing unit 100. For example, the processing unit 100 may transmit the output information to another device that has received input information from the server device 10. For example, the processing unit 100 may output the output information to the display device of the server device 10.

[0076] In step S7, the processing unit 200 generates output information following the output information received in step S6. The processing unit 200 may generate output information based on the intermediate data received in step S4. The processing unit 200 may read the intermediate data stored in the memory 220 and generate output information based on the read intermediate data. The processing unit 200 may generate output information based on at least a portion of the intermediate data. If the server device 10 comprises a plurality of processing units 200, the processing units 200 may generate output information by parallel processing.

[0077] The processing unit 200 may, for example, perform the decoding process of the transformer. The processing unit 200 may, for example, input the last token generated by the prefill process into the transformer.

[0078] The transformer reads the key-value cache stored in memory 220 and performs forward processing on the input token. If the transformer generated an embedded representation of an image in step S2, it may read the embedded representation stored in memory 220 and perform forward processing using the embedded representation. The transformer uses the key-value cache to generate tokens following the input token and a new key-value cache. The transformer outputs the tokens generated by the forward processing on the input token.

[0079] The output information generated during the decoding process is not limited to text data (tokens). The output information may include at least one of audio data or image data. Image data may include still images and videos, and videos may include audio data. The output information may also be multimodal data containing at least two of text data, audio data, or image data.

[0080] The processing unit 200 acquires the token output by the transformer as output information. The processing unit 200 stores the new key-value cache calculated by the transformer in the memory 220. The processing unit 200 may store the key-value cache in the memory 220 until the inference process is completed.

[0081] As another example, the processing unit 200 may perform a state-space model decoding process. As another example, the processing unit 200 may input the last token generated by the prefill process into the state-space model. The state-space model reads the state of the state-space model stored in memory 220 and performs forward processing on the input token. The state-space model generates a token following the input token and updates the state of the state-space model. The processing unit 200 obtains the token output by the state-space model as output information. The processing unit 200 overwrites the state stored in memory 220 with the updated state of the state-space model.

[0082] As another example, the processing unit 200 may perform a decoding process for a machine learning model that combines transformers and a state-space model. As an example, this type of machine learning model may have some layers composed of transformers and other layers composed of a state-space model. The processing unit 200 may input the output information generated by the prefill process into the machine learning model. In the layers composed of transformers, the machine learning model performs forward processing using the key-value cache stored in memory 220, and in the layers composed of a state-space model, it performs forward processing using the state of the state-space model stored in memory 220. In the layers composed of transformers, the machine learning model stores the new key-value cache computed by the transformers in memory 220, and in the layers composed of a state-space model, it updates the state of the state-space model stored in memory 220.

[0083] The processing unit 200 may read at least some of the parameters of the machine learning model stored in the memory 220. The processing unit 200 may generate output information based on at least some of the parameters read from the memory 220. The processing unit 200 may read at least some of the parameters of the machine learning model from the memory 220 each time it performs forward processing on an input token. The processing unit 200 may read some of the parameters of the machine learning model related to the input token from the memory 220.

[0084] In step S8, the processing unit 200 outputs the output information generated in step S7. The processing unit 200 may store the generated output information in the memory 220. The processing unit 100 may, for example, transmit the output information to another device that received the input information from the server device 10. The processing unit 100 may, for example, output the output information to the display device of the server device 10.

[0085] The processing unit 200 repeatedly executes steps S7 and S8 until the generation of output information for the machine learning model is complete. In the second and subsequent executions of step S7, the processing unit 200 may generate the next output information by inputting the output information generated in the previous step S7 into the machine learning model. In other words, the processing unit 200 may autoregressively generate output information for input information.

[0086] The processing unit 200 may perform a predetermined process using the input information received in step S1. The processing unit 200 may receive the input information from the processing unit 100 along with intermediate data. The processing unit 200 may store the input information received from the processing unit 100 in the memory 220. The predetermined process may be a forward process or a process other than a forward process.

[0087] [Other Embodiments] In the above embodiment, a configuration was described in which the information processing system 1000 performs inference processing of a machine learning model. The inference processing of the machine learning model may also be performed for training processing of the machine learning model.

[0088] The information processing system 1000 may perform inference processing as internal processing during the training process of a machine learning model. For example, the information processing system 1000 may perform inference processing of a machine learning model and perform training processing to update the parameters of the machine learning model based on the inference results. Specifically, the information processing system 1000 may perform training processing that repeatedly adjusts the parameters of the machine learning model so that the error between the inference result when training data is input to the machine learning model and the correct data for said training data becomes small.

[0089] <Summary> As is clear from the above description, an information processing system 1000 according to one embodiment of the present disclosure comprises at least one first memory, at least one first processor connected to at least one first memory, at least one second memory, and at least one second processor connected to at least one second memory. The at least one first processor generates intermediate data by inputting input information to a machine learning model and transmits at least a portion of the intermediate data to at least one second memory. The at least one second processor generates output information of the machine learning model using at least a portion of the intermediate data transmitted by the at least one first processor to the at least one second memory. The ratio of the data transfer rate of the at least one first memory to the calculation speed of the at least one first processor is lower than the ratio of the data transfer rate of the at least one second memory to the calculation speed of the at least one second processor.

[0090] At least one first memory may store at least some of the parameters of the machine learning model. At least one second memory may store at least some of the parameters of the machine learning model.

[0091] At least one first processor may generate intermediate data by performing a prefill process using a machine learning model. At least one second processor may generate output information by performing a decode process using a machine learning model.

[0092] The machine learning model may include a neural network with an attention mechanism. At least a portion of the intermediate data may include at least a portion of the key vectors and value vectors used in the attention mechanism.

[0093] At least one first processor may generate intermediate data by performing a first process that is computationally limiting using a machine learning model. At least one second processor may generate output information by performing a second process that is memory-limiting using a machine learning model. The first process may be a prefill process using a machine learning model. The second process may be a decode process using a machine learning model.

[0094] The machine learning model may include a state-space model. At least a portion of the intermediate data may include at least a portion of the states in the state-space model.

[0095] At least one second processor may repeatedly execute the second process using output information generated in a previously executed second process. At least one first processor may start transmitting at least a portion of the intermediate data before the generation of the intermediate data is complete. At least one first processor may start transmitting at least a portion of the intermediate data after the generation of the intermediate data is complete.

[0096] The ratio of data transfer rate to computation speed may also be the ratio of total data transfer volume to total number of computations when performing the entire process related to the machine learning model. Alternatively, the ratio of data transfer rate to computation speed may also be the ratio of data transfer volume per unit time to number of computations per unit time when performing the process related to the machine learning model.

[0097] The information processing system 1000 may have a plurality of second memories and a plurality of second processors. At least one first processor may transmit at least a portion of the intermediate data to each of the plurality of second memories. The plurality of second processors may form a cluster.

[0098] The information processing system 1000 may have a plurality of first memories and a plurality of first processors. The plurality of first processors may combine at least a portion of the intermediate data generated by each of the plurality of first processors and transmit it to at least one second memory.

[0099] The information processing system 1000 may perform inference processing for a machine learning model. The information processing system 1000 may perform inference processing for training a machine learning model.

[0100] The information processing system 1000 may have a first processing unit having at least one first memory and at least one first processor connected to at least one first memory. The information processing system 1000 may also have a second processing unit having at least one second memory and at least one second processor connected to at least one second memory.

[0101] As a result, according to one embodiment of this disclosure, a first processor with high computational performance generates intermediate data which is computationally limited, and a second processor with a large memory bandwidth generates output information which is memory-limited, thereby enabling high-speed computation of output information for machine learning models.

[0102] [Hardware Configuration] In the embodiments described above, some or all of the devices (server devices 10, 20) may be composed of hardware, or they may be composed of information processing by software (programs) executed by a CPU (Central Processing Unit), GPU (Graphics Processing Unit), etc. If the information processing is composed of software, the software that realizes at least some of the functions of each device in the embodiments described above may be stored in a non-temporary storage medium (non-temporary computer-readable medium) such as a CD-ROM (Compact Disc-Read Only Memory) or USB (Universal Serial Bus) memory, and the information processing of the software may be executed by having a computer read it. Alternatively, the software may be downloaded via a communication network. Furthermore, all or part of the processing of the software may be implemented in a circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array), so that the information processing by the software is executed by hardware.

[0103] The storage medium for the software may be a removable medium such as an optical disc, or a fixed storage medium such as a hard disk or memory. Furthermore, the storage medium may be located inside the computer (main memory, auxiliary storage, etc.) or outside the computer.

[0104] Figure 6 is a block diagram showing an example of the hardware configuration of each device (server devices 10, 20) in the embodiment described above. Each device may be implemented as a computer 7 comprising, for example, a processor 101, a main memory 102 (memory), an auxiliary memory 103 (memory), a network interface 104, and a device interface 105, which are connected via a bus 106.

[0105] The computer 7 in Figure 6 has one of each component, but it may have multiple identical components. Also, although Figure 6 shows one computer 7, the software may be installed on multiple computers, and each of these multiple computers may execute the same or different parts of the software's processing. In this case, it may be a distributed computing configuration in which each computer communicates via a network interface 104 or the like to execute processing. In other words, each device (server device 10, 20) in the above-described embodiment may be configured as a system that realizes its function by having one or more computers execute instructions stored in one or more storage devices. Alternatively, it may be configured so that information transmitted from a terminal is processed by one or more computers located on the cloud, and the processing results are transmitted to the terminal.

[0106] The various calculations performed by each device (server devices 10, 20) in the embodiments described above may be executed in parallel using one or more processors, or using multiple computers via a network. Alternatively, the various calculations may be distributed to multiple processing cores within a processor and executed in parallel. Furthermore, some or all of the processing and means of this disclosure may be implemented by at least one of a processor and a storage device located on a cloud that can communicate with computer 7 via a network. Thus, each device in the embodiments described above may be in the form of parallel computing using one or more computers.

[0107] The processor 101 may be at least an electronic circuit (processing circuit, processing circuitry, CPU, GPU, FPGA, ASIC, etc.) that performs either control or calculations of a computer. The processor 101 may also be a general-purpose processor, a dedicated processing circuit designed to perform specific calculations, or a semiconductor device including both a general-purpose processor and a dedicated processing circuit. Furthermore, the processor 101 may include optical circuits or quantum computing-based calculation functions.

[0108] The processor 101 may perform calculations based on data and software input from various devices within the computer 7, and may output calculation results and control signals to these devices. The processor 101 may also control the various components of the computer 7 by executing the computer 7's OS (Operating System) or applications.

[0109] Each of the devices (server devices 10, 20) in the embodiments described above may be implemented by one or more processors 101. Here, processor 101 may refer to one or more electronic circuits arranged on one chip, or one or more electronic circuits arranged on two or more chips or two or more devices. When multiple electronic circuits are used, each electronic circuit may communicate by wire or wireless.

[0110] The main memory 102 may store instructions executed by the processor 101 and various data, and the information stored in the main memory 102 may be read by the processor 101. The auxiliary storage device 103 is a storage device other than the main memory 102. These storage devices refer to any electronic component capable of storing electronic information, and may be semiconductor memory. The semiconductor memory may be either volatile memory or non-volatile memory. In each of the devices (server devices 10, 20) in the above-described embodiment, the storage device for storing various data may be implemented by the main memory 102 or the auxiliary storage device 103, or by the built-in memory of the processor 101. For example, each storage unit in the above-described embodiment may be implemented by the main memory 102 or the auxiliary storage device 103.

[0111] In the embodiments described above, if each device (server device 10, 20) consists of at least one storage device (memory) and at least one processor connected to (coupled with) this at least one storage device, then at least one processor may be connected to one storage device. Alternatively, at least one storage device may be connected to one processor. Furthermore, the configuration may include at least one processor among a plurality of processors being connected to at least one storage device among a plurality of storage devices. This configuration may also be realized by storage devices and processors included in a plurality of computers. Moreover, the configuration may include a storage device integrated with a processor (for example, a cache memory including an L1 cache and an L2 cache).

[0112] The network interface 104 is an interface for connecting to the communication network 8 wirelessly or via a wired connection. The network interface 104 can be any appropriate interface, such as one conforming to existing communication standards. Information may be exchanged between the computer 7 and an external device 9A connected via the communication network 8 through the network interface 104. The communication network 8 may be a WAN (Wide Area Network), LAN (Local Area Network), PAN (Personal Area Network), or a combination thereof, as long as information is exchanged between the computer 7 and the external device 9A. An example of a WAN is the Internet; an example of a LAN is IEEE 802.11 or Ethernet (registered trademark); and an example of a PAN is Bluetooth (registered trademark) or NFC (Near Field Communication).

[0113] The device interface 105 is an interface such as USB that connects directly to the external device 9B.

[0114] External device 9A is a device connected to computer 7 via a network. External device 9B is a device directly connected to computer 7.

[0115] External device 9A or external device 9B may, for example, be an input device. The input device may be a camera, microphone, motion capture device, various sensors, keyboard, mouse, touch panel, etc., and provides the acquired information to the computer 7. Alternatively, it may be a device equipped with an input unit, memory, and processor, such as a personal computer, tablet terminal, or smartphone.

[0116] Furthermore, the external device 9A or external device 9B may, for example, be an output device. The output device may be a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) panel, or it may be a speaker that outputs sound, etc. It may also be a device equipped with an output unit, memory, and processor, such as a personal computer, tablet terminal, or smartphone.

[0117] Furthermore, external devices 9A and 9B may be storage devices (memory). For example, external device 9A may be network storage, and external device 9B may be storage such as an HDD.

[0118] Furthermore, the external device 9A or external device 9B may be a device that has some of the functions of the components of each device (server devices 10, 20) in the embodiments described above. In other words, the computer 7 may transmit some or all of the processing results to the external device 9A or external device 9B, or may receive some or all of the processing results from the external device 9A or external device 9B.

[0119] Where the expression "at least one of a, b, and c" or "at least one of a, b, or c" (including similar expressions) is used in this specification (including the claims), it includes any of a, b, c, ab, ac, bc, or abc. It also includes multiple instances of any element, such as aa, abb, aabbcc, etc. Furthermore, it includes adding other elements other than the enumerated elements (a, b, and c), such as abcd having d.

[0120] In this specification (including the claims), when expressions such as "using data as input / based on data / according to / in accordance with data" (including similar expressions) are used, unless otherwise specified, this includes using the data itself or using data that has been processed in some way (e.g., data with added noise, normalized data, features extracted from the data, intermediate representations of the data, etc.). Furthermore, when it is stated that some result is obtained "using data as input / based on data / according to / in accordance with data" (including similar expressions), unless otherwise specified, this includes cases where the result is obtained based solely on the data in question or where the result is also influenced by other data, factors, conditions, and / or states other than the data in question. Furthermore, when it is stated that "data is output" (including similar expressions), unless otherwise specified, this includes cases where the data itself is used as output or where data that has been processed in some way (e.g., data with added noise, normalized data, features extracted from the data, intermediate representations of various types of data, etc.) is used as output.

[0121] In this specification (including the claims), the terms “connected” and “coupled” are intended to be non-restrictive terms that include any direct connection / coupling, indirect connection / coupling, electrical connection / coupling, communicative connection / coupling, operational connection / coupling, or physical connection / coupling. The terms should be interpreted as appropriate in the context in which they are used, but any form of connection / coupling that is not intentionally or naturally excluded should be interpreted non-restrictively as being included in the terms.

[0122] In this specification (including the claims), when the expression "A configured to B" is used, it may include that the physical structure of element A has a configuration capable of performing operation B, and that the permanent or temporary setting / configuration of element A is configured to actually perform operation B. For example, if element A is a general-purpose processor, it is sufficient that the processor has a hardware configuration capable of performing operation B, and that it is configured to actually perform operation B by the setting of a permanent or temporary program (instruction). Furthermore, if element A is a dedicated processor, dedicated arithmetic circuit, etc., it is sufficient that the circuit structure of the processor is implemented to actually perform operation B, regardless of whether control instructions and data are actually attached.

[0123] Wherever terms meaning "comprising" or "having" are used in this specification (including the claims), they are intended to be open-ended terms, including cases where the subject matter of such terms is not the object of the term. Where the object of such terms meaning "comprising" or "having" is an expression that does not specify a quantity or suggests a singular number (an expression with the article a or an), such expression should be interpreted as not being limited to a specific number.

[0124] In this specification (including the claims), even if expressions such as "one or more" or "at least one" are used in some places, and expressions that do not specify a quantity or suggest singularity (expressions using the articles a or an) are used in other places, the latter expressions are not intended to mean "one." In general, expressions that do not specify a quantity or suggest singularity (expressions using the articles a or an) should not necessarily be interpreted as not being limited to a specific number.

[0125] In this specification, if a particular configuration of an embodiment is described as providing a specific advantage or result, it should be understood, unless otherwise stated, that the same advantage or result can also be obtained from one or more other embodiments having that configuration. However, it should be understood that the presence or absence of such advantage or result generally depends on various factors, conditions, and / or states, and that the configuration does not necessarily guarantee that the advantage or result can be obtained. The advantage or result can only be obtained from the configuration described in the embodiment when various factors, conditions, and / or states are met, and the advantage or result cannot necessarily be obtained in the claimed invention that defines that configuration or a similar configuration.

[0126] In this specification (including the claims), when multiple hardware components perform a predetermined process, each component may cooperate to perform the predetermined process, or some components may perform all of the predetermined process. Alternatively, some components may perform part of the predetermined process, while other components perform the remainder. In this specification (including the claims), when expressions such as "one or more hardware components perform process A, and the one or more hardware components perform process B" (including similar expressions) are used, the hardware component performing process A and the hardware component performing process B may be the same or different. In other words, it is sufficient that the hardware component performing process A and the hardware component performing process B are included in the one or more hardware components. Hardware may include electronic circuits, devices containing electronic circuits, etc.

[0127] In this specification (including the claims), when multiple storage devices (memories) store data, each of the multiple storage devices may store only a portion of the data or the entire data. Furthermore, a configuration in which some of the multiple storage devices store data is also included.

[0128] In this specification (including the claims), terms such as "first," "second," etc., are used merely as a way of distinguishing between two or more elements and are not necessarily intended to impose technical meanings such as temporal, spatial, order, or quantity on the subject. Therefore, for example, references to a first element and a second element do not necessarily mean that only two elements can be employed therein, that the first element must precede the second element, or that the first element must exist for the second element to exist.

[0129] While embodiments of this disclosure have been described in detail above, this disclosure is not limited to the individual embodiments described above. Various additions, modifications, substitutions, and partial deletions are possible, provided that they do not depart from the conceptual idea and spirit of the present invention derived from the claims and their equivalents. For example, where numerical values ​​or mathematical formulas are used in the description of the embodiments described above, these are provided for illustrative purposes only and do not limit the scope of this disclosure. Similarly, the sequence of operations shown in the embodiments is also illustrative and does not limit the scope of this disclosure.

[0130] Furthermore, the following forms are possible for disclosure technology.

[0131] (Note 1) An information processing system comprising: at least one first memory; at least one first processor connected to the at least one first memory; at least one second memory; and at least one second processor connected to the at least one second memory, wherein the at least one first processor generates intermediate data by inputting input information to a machine learning model; transmits at least a portion of the intermediate data to the at least one second memory; the at least one second processor generates output information of the machine learning model using at least a portion of the intermediate data transmitted by the at least one first processor to the at least one second memory; and the ratio of the data transfer rate of the at least one first memory to the processing speed of the at least one first processor is lower than the ratio of the data transfer rate of the at least one second memory to the processing speed of the at least one second processor.

[0132] (Note 2) The information processing system according to Note 1, wherein the at least one first memory stores at least some parameters of the machine learning model, and the at least one second memory stores at least some parameters of the machine learning model.

[0133] (Note 3) The information processing system according to Note 1 or 2, wherein the at least one first processor generates the intermediate data by performing a prefill process using the machine learning model, and the at least one second processor generates the output information by performing a decode process using the machine learning model.

[0134] (Note 4) The machine learning model is the information processing system described in Note 3, which includes a neural network having an attention mechanism.

[0135] (Note 5) The information processing system according to Note 4, wherein at least a portion of the intermediate data includes at least a portion of the key vector and value vector used in the attention mechanism.

[0136] (Note 6) The information processing system according to any one of Notes 1 to 5, wherein the at least one first processor generates the intermediate data by executing a first process that is computationally limited using the machine learning model, and the at least one second processor generates the output information by a second process that is memory-limited using the machine learning model.

[0137] (Note 7) The information processing system as described in Note 6, wherein the first process is a prefill process using the machine learning model, and the second process is a decode process using the machine learning model.

[0138] (Note 8) The machine learning model is the information processing system described in Note 6, which includes a state-space model.

[0139] (Note 9) The information processing system according to Note 8, wherein at least a portion of the intermediate data includes at least a portion of the states in the state-space model.

[0140] (Note 10) The information processing system according to any one of Notes 6 to 9, wherein the at least one second processor repeatedly executes the second process using the output information generated in the second process that was executed in the past.

[0141] (Note 11) The information processing system according to any one of Notes 1 to 10, wherein the at least one first processor starts transmitting at least a portion of the intermediate data before the generation of the intermediate data is completed.

[0142] (Note 12) The information processing system according to any one of Notes 1 to 11, wherein the at least one first processor starts transmitting at least a portion of the intermediate data after the generation of the intermediate data is completed.

[0143] (Note 13) The data transfer rate to the calculation rate is the ratio of the total amount of data transferred to the total number of calculations when the entire process relating to the machine learning model is executed, as described in any of Notes 1 to 12.

[0144] (Note 14) The data transfer rate to the calculation rate is the ratio of the amount of data transferred per unit time to the number of calculations per unit time when processing the machine learning model is performed, as described in any of Notes 1 to 13.

[0145] (Note 15) An information processing system according to any one of Notes 1 to 14, comprising a plurality of second memories and a plurality of second processors, wherein at least one first processor transmits at least a portion of the intermediate data to each of the plurality of second memories.

[0146] (Note 16) The information processing system described in Note 15, wherein the plurality of second processors constitute a cluster.

[0147] (Note 17) An information processing system according to any one of Notes 1 to 16, comprising a plurality of first memories and a plurality of first processors, wherein the plurality of first processors combine at least a portion of the intermediate data generated by each of the plurality of first processors and transmit it to the at least one second memory.

[0148] (Note 18) The information processing system is an information processing system according to any one of Notes 1 to 17 that performs inference processing of the machine learning model.

[0149] (Note 19) The information processing system is the information processing system described in Note 18, which performs the inference process for training the machine learning model.

[0150] (Note 20) An information processing system according to any one of Notes 1 to 19, comprising: a first processing unit having at least one first memory and at least one first processor connected to the at least one first memory; and a second processing unit having at least one second memory and at least one second processor connected to the at least one second memory.

[0151] (Note 21) An information processing method comprising: at least one first processor connected to at least one first memory; inputting input information into a machine learning model to generate intermediate data; transmitting at least a portion of the intermediate data to at least one second memory; at least one second processor connected to the at least one second memory to generate output information of the machine learning model using at least a portion of the intermediate data transmitted by the at least one first processor to the at least one second memory; and the ratio of the data transfer rate of the at least one first memory to the processing speed of the at least one first processor being lower than the ratio of the data transfer rate of the at least one second memory to the processing speed of the at least one second processor.

[0152] (Note 22) A program wherein at least one first processor connected to at least one first memory is configured to perform a process of inputting input information into a machine learning model to generate intermediate data, and sending at least a portion of the intermediate data to at least one second memory, and at least one second processor connected to at least one second memory is configured to perform a process of generating output information for the machine learning model using at least a portion of the intermediate data sent by the at least one first processor to the at least one second memory, and the ratio of the data transfer rate of the at least one first memory to the calculation speed of the at least one first processor is lower than the ratio of the data transfer rate of the at least one second memory to the calculation speed of the at least one second processor.

[0153] This application claims priority to Japanese Patent Application No. 2024-173485, filed with the Japan Patent Office on 2 October 2024, which is incorporated herein by reference to its entire contents.

[0154] 10, 20: Server device 100, 200: Processing unit 110, 210: Processor 120, 220: Memory 130, 230: Network interface 1000: Information processing system

Claims

1. An information processing system comprising: at least one first memory; at least one first processor connected to the at least one first memory; at least one second memory; and at least one second processor connected to the at least one second memory, wherein the at least one first processor generates intermediate data by inputting input information to a machine learning model; the at least one second processor generates output information for the machine learning model using at least a portion of the intermediate data generated by the at least one first processor; and the ratio of the data transfer rate of the at least one first memory to the processing speed of the at least one first processor is lower than the ratio of the data transfer rate of the at least one second memory to the processing speed of the at least one second processor.

2. The information processing system according to claim 1, wherein the at least one first memory stores at least some parameters of the machine learning model, and the at least one second memory stores at least some parameters of the machine learning model.

3. The information processing system according to claim 1 or 2, wherein the at least one first processor generates the intermediate data by performing a prefill process using the machine learning model, and the at least one second processor generates the output information by performing a decode process using the machine learning model.

4. The information processing system according to claim 3, wherein the machine learning model includes a neural network having an attention mechanism.

5. The information processing system according to claim 4, wherein at least a portion of the intermediate data includes at least a portion of the key vector and value vector used in the attention mechanism.

6. The information processing system according to any one of claims 1 to 5, wherein the at least one first processor generates the intermediate data by performing a first process using the machine learning model, and the at least one second processor generates the output information by performing a second process using the machine learning model.

7. The information processing system according to claim 6, wherein the first process is a computation-limited process, and the second process is a memory-limited process.

8. The information processing system according to claim 6 or 7, wherein the first process is a prefill process and the second process is a decode process.

9. The information processing system according to claim 6 or 7, wherein the machine learning model includes a state-space model.

10. The information processing system according to claim 9, wherein at least a portion of the intermediate data includes at least a portion of the states in the state-space model.

11. The information processing system according to any one of claims 6 to 10, wherein the at least one second processor repeatedly executes the second process using the output information generated in the second process that was executed in the past.

12. The data transfer rate to the calculation rate is the ratio of the total amount of data transferred to the total number of calculations when the entire process relating to the machine learning model is executed, according to any one of claims 1 to 11.

13. The data transfer rate to the calculation rate is the ratio of the amount of data transferred per unit time to the number of calculations per unit time when processing the machine learning model is performed, according to any one of claims 1 to 11.

14. An information processing system according to any one of claims 1 to 13, comprising a plurality of second memories and a plurality of second processors, wherein each of the plurality of second processors generates output information of the machine learning model using at least a portion of the intermediate data generated by the at least one first processor.

15. The information processing system according to claim 14, wherein the plurality of second processors constitute a cluster.

16. An information processing system according to any one of claims 1 to 15, comprising a plurality of first memories and a plurality of first processors, wherein the plurality of first processors combine at least a portion of the intermediate data generated by each of the plurality of first processors and transmit it to the at least one second memory.

17. The information processing system according to any one of claims 1 to 16, wherein the information processing system performs inference processing of the machine learning model.

18. The information processing system according to claim 17, wherein the information processing system performs the inference process for training the machine learning model.

19. An information processing system according to any one of claims 1 to 18, comprising: a first processing unit having at least one first memory and at least one first processor connected to the at least one first memory; and a second processing unit having at least one second memory and at least one second processor connected to the at least one second memory.

20. The information processing system according to claim 19, wherein the first processing device starts transmitting at least a portion of the intermediate data to the second processing device before the generation of the intermediate data is completed.

21. The information processing system according to claim 19, wherein the first processing device starts transmitting at least a portion of the intermediate data to the second processing device after the generation of the intermediate data is completed.

22. The information processing system according to any one of claims 19 to 21, wherein the first processing device is a first GPU package, and the second processing device is a second GPU package.

23. The information processing system according to any one of claims 19 to 21, wherein the first processing device is a first type of accelerator, and the second processing device is a second type of accelerator.

24. The information processing system according to any one of claims 19 to 23, wherein the first processing unit and the second processing unit are connected by an interconnect.

25. An information processing system according to any one of claims 19 to 24, comprising a plurality of the first processing devices and a plurality of the second processing devices.

26. The information processing system according to claim 25, wherein the number of the plurality of first processing devices is the same as the number of the plurality of second processing devices.

27. An information processing system according to any one of claims 1 to 26, comprising a cluster.

28. An information processing method comprising: at least one first processor connected to at least one first memory generates intermediate data by inputting input information to a machine learning model; at least one second processor connected to at least one second memory generates output information of the machine learning model using at least a portion of the intermediate data generated by the at least one first processor; and the ratio of the data transfer rate of the at least one first memory to the calculation speed of the at least one first processor is lower than the ratio of the data transfer rate of the at least one second memory to the calculation speed of the at least one second processor.

29. A program that causes at least one first processor connected to at least one first memory to execute a process that generates intermediate data by inputting input information into a machine learning model, and causes at least one second processor connected to at least one second memory to execute a process that generates output information of the machine learning model using at least a portion of the intermediate data generated by the at least one first processor, wherein the ratio of the data transfer rate of the at least one first memory to the calculation speed of the at least one first processor is lower than the ratio of the data transfer rate of the at least one second memory to the calculation speed of the at least one second processor.

Images