Information processing system

Abstract

This technology provides a way to quickly generate output information for machine learning models. [Solution] The information processing system includes at least one first memory, at least one first processor connected to at least one first memory, at least one second memory, and at least one second processor connected to at least one second memory. The first processor generates intermediate data by inputting input information into a machine learning model. The second processor generates output information for the machine learning model using at least a portion of the intermediate data generated by the first processor. The ratio of the data transfer rate of the first memory to the calculation speed of the first processor is lower than the ratio of the data transfer rate of the second memory to the calculation speed of the second processor.

Description

Technical Field

[0001] The present disclosure relates to an information processing system, an information processing method, and a program.

Background Art

[0002] Machine learning models such as large language models (LLMs) are known. In order to efficiently execute the inference process of large language models, there is a technique for generating intermediate data called a key value cache.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] An object of the present disclosure is to provide a technique for quickly generating output information of a machine learning model.

Means for Solving the Problems

[0005] An information processing system according to one aspect of the present disclosure includes at least one first memory, at least one first processor connected to at least one first memory, at least one second memory, and at least one second processor connected to at least one second memory, wherein the at least one first processor generates intermediate data by inputting input information to a machine learning model, the at least one second processor generates output information for the machine learning model using at least a portion of the intermediate data generated by the at least one first processor, and the ratio of the data transfer rate of the at least one first memory to the processing speed of the at least one first processor is lower than the ratio of the data transfer rate of the at least one second memory to the processing speed of the at least one second processor. [Brief explanation of the drawing]

[0006] [Figure 1] Figure 1 is a diagram illustrating an example of the inference process of a machine learning model. [Figure 2] Figure 2 is a block diagram showing the first example of the overall configuration of an information processing system. [Figure 3] Figure 3 is a block diagram showing a second example of the overall configuration of an information processing system. [Figure 4] Figure 4 is a block diagram showing a third example of the overall configuration of an information processing system. [Figure 5] Figure 5 is a flowchart showing an example of the inference process of a machine learning model. [Figure 6] Figure 6 is a block diagram showing an example of a computer hardware configuration. [Modes for carrying out the invention]

[0007] Hereinafter, embodiments of this disclosure will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0008] [First Embodiment] A first embodiment of this disclosure is an example of an information processing system that performs a predetermined task based on a machine learning model. In this embodiment, the machine learning model may include a neural network having an attention mechanism. The neural network having an attention mechanism may be, for example, a neural network also known as a Transformer.

[0009] Machine learning models are not limited to transformers. For example, a machine learning model may be a state-space model (SSM). A machine learning model may consist of multiple machine learning models. For example, a machine learning model may include Mamba, Jamba, etc. Any machine learning model that generates intermediate data in the inference process is acceptable.

[0010] The inference process of a transformer, an example of a machine learning model, will be explained with reference to Figure 1. Figure 1 is a diagram illustrating an example of the inference process of a machine learning model.

[0011] As shown in Figure 1, the inference process of a transformer can be broadly divided into a prefill process and a decode process. The prefill process generates a key-value cache based on input information called a prompt. The decode process generates output information using the key-value cache generated in the prefill process. The prefill process is an example of the first process. The decode process is an example of the second process.

[0012] In the prefill process, the input prompt is divided into tokens, and each token is input to the transformer one by one from the beginning. The transformer performs a forward process for each input token. In the forward process, the transformer calculates a query vector, key vector, and value vector for each token. The transformer outputs one token for each input token.

[0013] The transformer caches an array of key vectors and value vectors calculated for each token. This cached array of key vectors and value vectors is called the key-value cache. A key-value cache is generated for each layer that makes up the transformer. The key-value cache for each layer is calculated sequentially, starting from the layer closest to the input layer and moving towards the output layer. In the example in Figure 1, the transformer includes N hidden layers, and key-value caches are generated sequentially from the first layer to the Nth layer.

[0014] A token is a processing unit used by machine learning models to process electronic data, and the amount of data in one token can vary depending on the design of the machine learning model. For example, a token may represent a single Japanese character or a single English word, but depending on its frequency of occurrence, one character may be represented by two tokens, or two or more characters may be represented by one token.

[0015] A cache, for example, is a technique that copies recently read or frequently used data from a slow storage device to a high-speed storage device. The high-speed storage device that copies the data, or the copied data itself, is sometimes also called a cache. In other words, a key-value cache may include the process by which a machine learning model stores key vectors and value vectors, may include the stored key vectors and value vectors, or may include the storage area in which the key vectors and value vectors are stored.

[0016] In the decoding process, the token finally output in the prefill process is input to the transformer. The transformer uses the key-value cache generated in the prefill process to perform forward processing on the input token. The token output by the forward processing is input to the transformer again. The decoding process generates output information for the prompt by repeatedly performing forward processing on the token output in the previous forward processing. That is, the decoding process generates output information for the prompt by performing forward processing autoregressively.

[0017] Since the prefill process has low computational order dependence, forward processing can be performed in parallel for each of the input tokens. On the other hand, since the decoding process has high computational order dependence, forward processing must be performed sequentially, increasing the amount of data read and written to memory. Therefore, the execution of the decoding process requires a higher memory bandwidth of the processor per amount of computation compared to the execution of the prefill process. That is, when executing the prefill process and the decoding process, it can be said that the prefill process is computation bottlenecked, and the decoding process is memory bottlenecked.

[0018] Note that the prefill process is an example of the first process, and the decoding process is an example of the second process. However, the first process may be a process with a relatively larger degree of computation bottleneck than the second process, or the second process may be a process with a relatively larger degree of memory bottleneck than the first process.

[0019] In the prior art, the inference processing of the machine learning model is executed on the same processor. Note that the same processor includes using a plurality of processors of the same type. When the prefill process and the decoding process are executed on the same processor, there is excess memory bandwidth in the prefill process and excess computing performance in the decoding process. In other words, when the inference processing of the machine learning model is executed on the same processor, the performance of the processor cannot be effectively utilized. Therefore, there is room for improving the efficiency of the inference processing of the machine learning model.

[0020] This embodiment provides a technique for quickly generating output information of a machine learning model. In this embodiment, an information processing system includes two or more processors having different ratios of data transfer speed to arithmetic speed. The first processor with a relatively high ratio of data transfer speed to arithmetic speed (i.e., high arithmetic performance) inputs input information into the machine learning model to generate intermediate data, and transmits at least a part of the intermediate data to the second processor. The second processor with a relatively low ratio of data transfer speed to arithmetic speed (i.e., large memory bandwidth) uses at least a part of the intermediate data received from the first processor to generate output information of the machine learning model.

[0021] The generation of intermediate data becomes the arithmetic bottleneck. Therefore, when using a processor with high arithmetic performance, intermediate data can be generated quickly. The generation of output information becomes the memory bottleneck. Therefore, when using a processor with a large memory bandwidth, output information can be generated quickly. In one aspect, according to this embodiment, since the performance of the processor can be effectively utilized through the inference process of the machine learning model, the output information of the machine learning model can be generated quickly.

[0022] <Overall configuration of the information processing system> The overall configuration of the information processing system according to this embodiment will be described with reference to FIGS. 2 to 4.

[0023] FIG.  2 is a block diagram showing a first example of the overall configuration of the information processing system. As shown in FIG. 2, the information processing system 1000 may include one or more server devices 10. The information processing system 1000 may include a plurality of server devices 10.

[0024] The server device 10 may include one or more processing devices 100 and one or more processing devices 200. The server device 10 may include a plurality of processing devices 100. The server device 10 may include a plurality of processing devices 200. The number of processing devices 100 and the number of processing devices 200 included in the server device 10 may be the same or different.

[0025] If the server device 10 includes multiple processing units 100, the multiple processing units 100 may form a cluster. The multiple processing units 100 constituting the cluster may perform parallel processing. The multiple processing units 100 may combine the parallel processed data and transmit it to the processing unit 200.

[0026] The number of processing units 200 provided by the server device 10 may be greater than the number of processing units 100 provided by the server device 10. For example, the server device 10 may include one processing unit 100 and multiple processing units 200. If the server device 10 includes multiple processing units 200, the multiple processing units 200 may form a cluster. The multiple processing units 200 forming the cluster may perform parallel processing. If the server device 10 includes multiple processing units 200, the processing unit 100 may send data to each of the multiple processing units 200.

[0027] The processing unit 100 may include one or more processors 110 and one or more memories 120. The processing unit 100 may include multiple processors 110 and multiple memories 120. The processors 110 and memories 120 may be electrically connected. The processors 110 and memories 120 may be connected to each other in a way that enables data communication.

[0028] The processing unit 200 may comprise one or more processors 210 and one or more memories 220. The processing unit 200 may comprise multiple processors 210 and multiple memories 220. The processors 210 and memories 220 may be electrically connected. The processors 210 and memories 220 may be connected to each other in a manner that enables data communication.

[0029] The processing unit 100 and the processing unit 200 may be connected to bus B of the server device 10. The processing unit 100 and the processing unit 200 may be configured to communicate data with each other via bus B. Bus B may be a dedicated transmission path (interconnect) that enables data communication between the processing unit 100 and the processing unit 200. Bus B may also be the system bus provided by the server device 10.

[0030] The processing units 100 and 200 may include processing circuits capable of performing specific calculations at high speed. The processing units 100 and 200 may also include processing circuits capable of performing general-purpose calculations. Specific calculations may include, for example, multiply-accumulate operations, which are frequently used in training neural networks.

[0031] The processing units 100 and 200 may, for example, be a GPU package (e.g., a GPU board or GPU card) equipped with one or more graphics processing units (GPUs). The processing units 100 and 200 may, for example, be a CPU package (e.g., a CPU board or CPU card) equipped with one or more central processing units (CPUs). The processing units 100 and 200 may also be equipped with one or more various accelerators.

[0032] The processing unit 100 and the processing unit 200 may be of the same type or different types. For example, the processing unit 100 may be equipped with a GPU package or an accelerator, and the processing unit 200 may be equipped with a CPU package. For example, the processing unit 100 may be equipped with a first type of GPU package, and the processing unit 200 may be equipped with a second type of GPU package. For example, the processing unit 100 may be equipped with a first type of accelerator, and the processing unit 200 may be equipped with a second type of accelerator.

[0033] Memory 120 may be a storage device that stores data used in calculations performed by processor 110 and data obtained from calculations performed by processor 110. Memory 220 may be a storage device that stores data used in calculations performed by processor 210 and data obtained from calculations performed by processor 210. Memory 120 and memory 220 may, for example, be DRAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory). Memory 120 and memory 220 may, for example, be a three-dimensional memory structure with multiple stacked units.

[0034] Memory 120 and memory 220 may store parameters of the machine learning model. Memory 120 and memory 220 may store at least some of the parameters of the machine learning model. Memory 120 and memory 220 may store all of the parameters of the machine learning model. The parameters stored in memory 120 and the parameters stored in memory 220 may be the same or different. Memory 120 may store parameters required for processing performed by processor 110. Memory 220 may store parameters required for processing performed by processor 210.

[0035] Processing unit 100 has a lower B / F ratio (Bytes / FLOPS) compared to processing unit 200. Conversely, processing unit 200 has a higher B / F ratio compared to processing unit 100. The B / F ratio is the ratio of memory bandwidth to computational performance. Memory bandwidth is one example of an indicator of memory data transfer speed. Data transfer speed may, for example, be the amount of data that the processor can read and write to memory per unit time. Computational performance is one example of an indicator of computation speed. Computational performance may, for example, be the number of floating-point operations that the processor can perform per unit time. Computational performance may vary depending on the precision of the floating-point number resulting from the calculation.

[0036] In other words, the ratio of the data transfer rate of memory 120 to the processing speed of processor 110 (data transfer rate / processing speed) is lower than the ratio of the data transfer rate of memory 220 to the processing speed of processor 210. Conversely, the ratio of the data transfer rate of memory 220 to the processing speed of processor 210 is higher than the ratio of the data transfer rate of memory 120 to the processing speed of processor 110.

[0037] The difference between the B / F ratio of processing unit 100 and the B / F ratio of processing unit 200 can be arbitrary. For example, the difference between the B / F ratio of processing unit 100 and the B / F ratio of processing unit 200 may be as much as 10 times. For example, the B / F ratio of processing unit 100 may be around 1:100 (=0.001), and the B / F ratio of processing unit 200 may be around 1:100 (=0.01).

[0038] The memory data transfer speed and processor calculation speed may be theoretical performance or performance when performing a predetermined process. The theoretical performance may be the catalog values ​​for the memory and processor. The predetermined process may include processing related to a machine learning model. As an example, the predetermined process may include inference processing of a machine learning model, or one or more operations included in the inference processing.

[0039] The ratio of data transfer rate to computational performance may be a value evaluated over the entire process, or a value evaluated per unit time when the process is executed. In other words, the ratio of data transfer rate to computational performance may be the ratio of the total amount of data transferred over the entire process related to the machine learning model to the total number of computations performed over the entire process related to the machine learning model. Alternatively, the ratio of data transfer rate to computational performance may be the ratio of the amount of data transferred per unit time when the process related to the machine learning model is executed to the number of computations performed per unit time when the process related to the machine learning model is executed.

[0040] The ratio of data transfer rate to computational performance may be a value evaluated according to the combination of the number of processors and the number of memory units used when executing the process. As a first example, the ratio of data transfer rate to computational performance may be the performance when executing the process using one memory unit and one processor, and may also be the theoretical performance in this case. As a second example, the ratio of data transfer rate to computational performance may be the performance when executing the process using one memory unit and two or more processors, and may also be the theoretical performance in this case. As a third example, the ratio of data transfer rate to computational performance may be the performance when executing the process using two or more memory units and one processor, and may also be the theoretical performance in this case. As a fourth example, the ratio of data transfer rate to computational performance may be the performance when executing the process using two or more memory units and two or more processors, and may also be the theoretical performance in this case.

[0041] Figure 3 is a block diagram showing a second example of the overall configuration of an information processing system. As shown in Figure 3, the information processing system 1000 may include one or more server devices 10 and one or more server devices 20. The information processing system 1000 may include multiple server devices 10. The information processing system 1000 may include multiple server devices 20. The number of server devices 10 and the number of server devices 20 included in the information processing system 1000 may be the same or different.

[0042] The server device 10 may include one or more processing units 100 and a network interface 130. The server device 10 may include multiple processing units 100. The processing units 100 may be configured in the same manner as in the first example.

[0043] The server device 20 may include one or more processing units 200 and a network interface 230. The server device 20 may include multiple processing units 200. The processing units 200 may be configured in the same manner as in the first example.

[0044] Network interfaces 130 and 230 may be connected to a communication network N. Processing units 100 and 200 may be configured to communicate data with each other via the communication network N. The communication network N may include a dedicated communication path that enables data communication between network interface 130 and network interface 230. The communication network N may be, for example, a LAN (Local Area Network) or the Internet.

[0045] If the information processing system 1000 includes multiple server devices 10, the multiple server devices 10 may form a cluster. The multiple server devices 10 forming the cluster may perform parallel processing. The multiple server devices 10 may combine the parallel processed data and transmit it to the server device 20.

[0046] The number of server devices 20 may be greater than the number of server devices 10. For example, the information processing system 1000 may include one server device 10 and multiple server devices 20. If the information processing system 1000 includes multiple server devices 20, the multiple server devices 20 may form a cluster. The multiple server devices 20 forming the cluster may perform parallel processing. If the information processing system 1000 includes multiple server devices 20, server device 10 may send data to each of the multiple server devices 20.

[0047] The processing units 100 and 200 may be configured in the same manner as in the first example. That is, the processing unit 100 provided by server device 10 has a lower B / F ratio (Bytes / FLOPS) compared to the processing unit 200 provided by server device 20. Conversely, the processing unit 200 provided by server device 20 has a higher B / F ratio compared to the processing unit 100 provided by server device 10.

[0048] In other words, the ratio of the data transfer rate of memory 120 to the processing speed of processor 110 (data transfer rate / processing speed) is lower than the ratio of the data transfer rate of memory 220 to the processing speed of processor 210. Conversely, the ratio of the data transfer rate of memory 220 to the processing speed of processor 210 is higher than the ratio of the data transfer rate of memory 120 to the processing speed of processor 110.

[0049] Figure 4 is a block diagram showing a third example of the overall configuration of an information processing system. As shown in Figure 4, the information processing system 1000 may include one or more server devices 10. The information processing system 1000 may include multiple server devices 10.

[0050] The server device 10 may include one or more processing units 100. The server device 10 may include multiple processing units 100.

[0051] The processing unit 100 may include one or more processors 110, one or more processors 210, one or more memory 120, and one or more memory 220. The processing unit 100 may include multiple processors 110. The processing unit 100 may include multiple memory 120. The processing unit 100 may include multiple processors 210. The processing unit 100 may include multiple memory 220.

[0052] The processor 110 and memory 120 may be electrically connected. The processor 110 and memory 120 may be connected to each other in a way that enables data communication. The processor 210 and memory 220 may be electrically connected. The processor 210 and memory 220 may be connected to each other in a way that enables data communication.

[0053] Processor 110 and processor 210 may be connected to a circuit C formed within the processing unit 100. Processor 110 and processor 210 may be configured to communicate data with each other via the circuit C. The circuit C may include, for example, electronic components or wiring formed on the substrate of the processing unit 100.

[0054] If the processing unit 100 includes multiple processors 110, the multiple processors 110 may perform parallel processing. The multiple processors 110 may combine the parallel processed data and send it to the processor 210.

[0055] The number of processors 210 may be greater than the number of processors 110. For example, the processing unit 100 may include one processor 110 and multiple processors 210. If the processing unit 100 includes multiple processors 210, the multiple processors 210 may perform parallel processing. If the processing unit 100 includes multiple processors 210, processor 110 may send data to each of the multiple processors 210.

[0056] Processors 110, 210 and memories 120, 220 may be configured in the same manner as in the first example. That is, the ratio of the data transfer rate of memory 120 to the processing speed of processor 110 (data transfer rate / processing speed) is lower than the ratio of the data transfer rate of memory 220 to the processing speed of processor 210. Conversely, the ratio of the data transfer rate of memory 220 to the processing speed of processor 210 is higher than the ratio of the data transfer rate of memory 120 to the processing speed of processor 110.

[0057] Note that the overall configuration of the information processing system 1000 shown in Figures 2 to 4 is just one example, and various system configurations are possible depending on the application and purpose. The information processing system 1000 may consist of one or more devices. The devices included in the information processing system 1000 may be a system composed of multiple devices. Each function included in the information processing system 1000 may be implemented by any device that constitutes the system. Each component included in the information processing system 1000 may be included in any device that constitutes the system.

[0058] <Inference Processing Flow> The inference process of the machine learning model executed by the information processing system 1000 will be explained with reference to Figure 5. Figure 5 is a flowchart showing an example of the machine learning model inference process. Here, we will explain an example in which the information processing system 1000 shown in Figure 2 performs the machine learning model inference process.

[0059] The information processing system 1000 stores the parameters of the machine learning model in memory 120 and memory 220 before executing the inference process of the machine learning model. The parameters of the machine learning model stored in memory 120 may be read from an auxiliary storage device built into the server device 10 or an external storage device connected to the server device 10. The parameters of the machine learning model stored in memory 220 may be read from an auxiliary storage device built into the server device 20 or an external storage device connected to the server device 20.

[0060] In step S1, the processing unit 100 receives input information for the machine learning model. The input information may, for example, be a natural language sentence called a prompt. The input information is not limited to text data. The input information may include at least one of audio data or image data. The image data may include still images and videos, and videos may include audio data. The input information may also be multimodal data including at least two of text data, audio data, or image data. The input information may be entered by a user of the information processing system 1000. The input information may be generated by the information processing system 1000 or an external information processing unit or information processing system. Note that "external" means not included in the information processing system 1000.

[0061] For example, the processing unit 100 may receive input information received by the server device 10 from another device. The other device may be a terminal device operated by a user, or an external information processing device or information processing system that generated the input information. For example, the processing unit 100 may receive input information entered into the server device 10.

[0062] In step S2, the processing unit 100 generates intermediate data based on the input information received in step S1. The processing unit 100 may also generate intermediate data by inputting the input information into a machine learning model. If the server device 10 comprises multiple processing units 100, the processing units 100 may generate intermediate data by parallel processing. The processing unit 100 stores the intermediate data generated by the machine learning model in the memory 120.

[0063] The processing unit 100 may generate a key-value cache, which is an example of intermediate data, by inputting a prompt, which is an example of input information, into a transformer, which is an example of a machine learning model. The processing unit 100 may also generate a state in a state-space model, which is an example of intermediate data, by inputting a prompt, which is an example of input information, into a state-space model, which is an example of a machine learning model. The state in the state-space model may be represented by one or more vectors or tensors.

[0064] As an example, the processing unit 100 may perform a transformer prefill process. As another example, the processing unit 100 may calculate an embedded representation, which is another example of intermediate data, based on the image data contained in the input information.

[0065] The transformer performs a forward process for each input token. In the forward process, the transformer calculates a query vector, key vector, and value vector for each token. The transformer caches the array of key vectors and value vectors for each token. This generates a key-value cache. The transformer generates the key-value cache sequentially, starting from the input layer and moving towards the output layer.

[0066] The processing unit 100 may read at least some of the parameters of the machine learning model stored in memory 120. The processing unit 100 may generate intermediate data based on at least some of the parameters read from memory 120. The processing unit 100 may read at least some of the parameters of the machine learning model from memory 120 each time it performs forward processing on an input token. The processing unit 100 may read at least some of the parameters of the machine learning model related to an input token from memory 120.

[0067] In step S3, the processing unit 100 transmits the intermediate data generated in step S3 to the processing unit 200. The processing unit 100 may read the intermediate data stored in the memory 120 and transmit it to the processing unit 200 via the bus B. The processing unit 100 may transmit at least a portion of the intermediate data to the processing unit 200.

[0068] The processing unit 100 may directly transmit the intermediate data to the processing unit 200 (memory 220). The processing unit 100 may also indirectly transmit the intermediate data to the processing unit 200 (memory 220). As an example of the transmission process, the processing unit 100 may store the intermediate data in a storage device and transmit information that allows it to identify the intermediate data stored in the storage device to the processing unit 200. The storage device that stores the intermediate data may be memory 120 or another memory. The processing unit 200 may use the information that allows it to identify the intermediate data to retrieve the intermediate data stored in the storage device. The processing unit 100 may also transmit the intermediate data to memory 220 via another device. In other words, the processing unit 100 can make the intermediate data available to the processing unit 200 by any means necessary.

[0069] If the server device 10 includes multiple processing units 100, the processing unit 100 may combine multiple intermediate data stored in multiple memories 120 and transmit the combined intermediate data to the processing unit 200. If the server device 10 includes multiple processing units 200, the processing unit 100 may transmit the intermediate data to each of the multiple processing units 200.

[0070] The processing unit 100 may start sending at least some of the generated intermediate data to the processing unit 200 before the generation of the intermediate data is complete. The processing unit 100 may send the key-value caches generated by the transformer layer by layer to the processing unit 200 layer by layer. For example, the processing unit 100 may start sending the generated key-value cache each time it generates a key-value cache for one layer, before the generation of key-value caches for all layers is complete. For another example, the processing unit 100 may start sending the key-value caches for all layers after the generation of key-value caches for all layers is complete.

[0071] In step S4, the processing unit 200 receives intermediate data from the processing unit 100. The processing unit 200 stores the intermediate data received from the processing unit 100 in the memory 220.

[0072] The information processing system 1000 may repeatedly execute the processes from step S2 to step S4. For example, the processing unit 100 may repeatedly generate a key-value cache for each layer of the transformer, and each time the generation of the key-value cache for one layer is completed, send the generated key-value cache to the processing unit 200, for the total number of transformers.

[0073] In step S5, the processing unit 100 generates output information for the input information received in step S1. The processing unit 100 may also obtain output information output by the machine learning model after the generation of intermediate data is complete. For example, in the prefill process of the transformer, the processing unit 100 may obtain the last token output by forward processing on the last token of the prompt. The processing unit 100 sends the generated output information to the processing unit 200.

[0074] The output information generated during the prefill process is not limited to text data (tokens). The output information may include at least one of audio data or image data. Image data may include still images and videos, and videos may include audio data. The output information may also be multimodal data containing at least two of text data, audio data, or image data.

[0075] In step S6, the processing unit 200 receives output information from the processing unit 100. The processing unit 200 may store the output information received from the processing unit 100 in the memory 220. The processing unit 200 may output the output information received from the processing unit 100. For example, the processing unit 100 may transmit the output information to another device that has received input information from the server device 10. For example, the processing unit 100 may output the output information to the display device of the server device 10.

[0076] In step S7, the processing unit 200 generates output information following the output information received in step S6. The processing unit 200 may generate output information based on the intermediate data received in step S4. The processing unit 200 may read the intermediate data stored in the memory 220 and generate output information based on the read intermediate data. The processing unit 200 may generate output information based on at least a portion of the intermediate data. If the server device 10 comprises multiple processing units 200, the processing units 200 may generate output information by parallel processing.

[0077] The processing unit 200 may, for example, perform the decoding process of the transformer. The processing unit 200 may, for example, input the last token generated by the prefill process into the transformer.

[0078] The transformer reads the key-value cache stored in memory 220 and performs forward processing on the input token. If the transformer generated an image embedding representation in step S2, it may read the embedding representation stored in memory 220 and perform forward processing using the embedding representation. The transformer uses the key-value cache to generate tokens following the input token and a new key-value cache. The transformer outputs the tokens generated by the forward processing on the input token.

[0079] The output information generated during the decoding process is not limited to text data (tokens). The output information may include at least one of audio data or image data. Image data may include still images and videos, and videos may include audio data. The output information may also be multimodal data containing at least two of text data, audio data, or image data.

[0080] The processing unit 200 acquires the token output by the transformer as output information. The processing unit 200 stores the new key-value cache calculated by the transformer in memory 220. The processing unit 200 may store the key-value cache in memory 220 until the inference process is completed.

[0081] As another example, the processing unit 200 may perform a decoding process of the state-space model. As another example, the processing unit 200 may input the last token generated by the prefill process into the state-space model. The state-space model reads the state of the state-space model stored in memory 220 and performs a forward process on the input token. The state-space model generates a token following the input token and updates the state of the state-space model. The processing unit 200 obtains the token output by the state-space model as output information. The processing unit 200 overwrites the state stored in memory 220 with the updated state of the state-space model.

[0082] As another example, the processing unit 200 may perform a decoding process for a machine learning model that combines transformers and a state-space model. This type of machine learning model may, for example, have some layers composed of transformers and other layers composed of a state-space model. The processing unit 200 may input the output information generated by the prefill process into the machine learning model. In the layers composed of transformers, the machine learning model performs forward processing using the key-value cache stored in memory 220, and in the layers composed of a state-space model, it performs forward processing using the state of the state-space model stored in memory 220. In the layers composed of transformers, the machine learning model stores the new key-value cache computed by the transformers in memory 220, and in the layers composed of a state-space model, it updates the state of the state-space model stored in memory 220.

[0083] The processing unit 200 may read at least some of the parameters of the machine learning model stored in memory 220. The processing unit 200 may generate output information based on at least some of the parameters read from memory 220. The processing unit 200 may read at least some of the parameters of the machine learning model from memory 220 each time it performs forward processing on an input token. The processing unit 200 may read some of the parameters of the machine learning model related to the input token from memory 220.

[0084] In step S8, the processing unit 200 outputs the output information generated in step S7. The processing unit 200 may store the generated output information in the memory 220. The processing unit 100 may, for example, transmit the output information to another device that received the input information from the server device 10. The processing unit 100 may, for example, output the output information to the display device of the server device 10.

[0085] The processing unit 200 repeatedly executes steps S7 and S8 until the generation of output information for the machine learning model is complete. In the second and subsequent executions of step S7, the processing unit 200 may generate the next output information by inputting the output information generated in the previous step S7 into the machine learning model. In other words, the processing unit 200 may autoregressively generate output information for input information.

[0086] The processing unit 200 may perform a predetermined process using the input information received in step S1. The processing unit 200 may receive the input information from the processing unit 100 along with intermediate data. The processing unit 200 may store the input information received from the processing unit 100 in the memory 220. The predetermined process may be a forward process or a process other than a forward process.

[0087] [Other embodiments] In the above embodiment, a configuration was described in which the information processing system 1000 performs inference processing of a machine learning model. The inference processing of the machine learning model may also be performed for training processing of the machine learning model.

[0088] The information processing system 1000 may perform inference processing as an internal process during the training process of a machine learning model. For example, the information processing system 1000 may perform inference processing of a machine learning model and then perform training processing to update the parameters of the machine learning model based on the inference results. Specifically, the information processing system 1000 may perform training processing that repeatedly adjusts the parameters of the machine learning model so that the error between the inference result when training data is input to the machine learning model and the correct data for that training data becomes small.

[0089] <Summary> As is clear from the above description, an information processing system 1000 according to one embodiment of the present disclosure comprises at least one first memory, at least one first processor connected to the at least one first memory, at least one second memory, and at least one second processor connected to the at least one second memory. The at least one first processor generates intermediate data by inputting input information to a machine learning model and transmits at least a portion of the intermediate data to at least one second memory. The at least one second processor generates output information for the machine learning model using at least a portion of the intermediate data transmitted by the at least one first processor to the at least one second memory. The ratio of the data transfer rate of the at least one first memory to the processing speed of the at least one first processor is lower than the ratio of the data transfer rate of the at least one second memory to the processing speed of the at least one second processor.

[0090] At least one first memory may store at least some of the parameters of the machine learning model. At least one second memory may store at least some of the parameters of the machine learning model.

[0091] At least one first processor may generate intermediate data by performing a prefill process using a machine learning model. At least one second processor may generate output information by performing a decode process using a machine learning model.

[0092] The machine learning model may include a neural network with an attention mechanism. At least a portion of the intermediate data may include at least a portion of the key vectors and value vectors used in the attention mechanism.

[0093] At least one first processor may generate intermediate data by performing a first process that is computationally limiting using a machine learning model. At least one second processor may generate output information by performing a second process that is memory-limiting using a machine learning model. The first process may be a prefill process using a machine learning model. The second process may be a decode process using a machine learning model.

[0094] The machine learning model may include a state-space model. At least a portion of the intermediate data may include at least a portion of the states in the state-space model.

[0095] At least one second processor may repeatedly execute the second process using output information generated in a previously executed second process. At least one first processor may start transmitting at least a portion of the intermediate data before the generation of the intermediate data is complete. At least one first processor may start transmitting at least a portion of the intermediate data after the generation of the intermediate data is complete.

[0096] The ratio of data transfer rate to computation speed may also be the ratio of total data transfer volume to total number of computations when performing the entire process related to the machine learning model. Alternatively, the ratio of data transfer rate to computation speed may also be the ratio of data transfer volume per unit time to number of computations per unit time when performing the process related to the machine learning model.

[0097] The information processing system 1000 may have a plurality of second memories and a plurality of second processors. At least one first processor may transmit at least a portion of the intermediate data to each of the plurality of second memories. The plurality of second processors may form a cluster.

[0098] The information processing system 1000 may have a plurality of first memories and a plurality of first processors. The plurality of first processors may combine at least a portion of the intermediate data generated by each of the plurality of first processors and transmit it to at least one second memory.

[0099] The information processing system 1000 may perform inference processing for a machine learning model. The information processing system 1000 may perform inference processing for training a machine learning model.

[0100] The information processing system 1000 may have a first processing unit having at least one first memory and at least one first processor connected to at least one first memory. The information processing system 1000 may also have a second processing unit having at least one second memory and at least one second processor connected to at least one second memory.

[0101] As a result, according to one embodiment of this disclosure, a first processor with high computational performance generates intermediate data which is computationally limited, and a second processor with a large memory bandwidth generates output information which is memory-limited, thereby enabling high-speed computation of output information for machine learning models.

[0102] [Hardware configuration] In the embodiments described above, some or all of the devices (server devices 10, 20) may be composed of hardware, or they may be composed of information processing by software (programs) executed by a CPU (Central Processing Unit), GPU (Graphics Processing Unit), etc. If they are composed of information processing by software, the software that realizes at least some of the functions of each device in the embodiments described above may be stored on a non-temporary storage medium (non-temporary computer-readable medium) such as a CD-ROM (Compact Disc-Read Only Memory) or USB (Universal Serial Bus) memory, and the information processing by the software may be executed by loading it into a computer. Alternatively, the software may be downloaded via a communication network. Furthermore, all or part of the processing by the software may be implemented in a circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array), so that the information processing by the software is executed by hardware.

[0103] The storage medium for the software may be a removable medium such as an optical disc, or a fixed storage medium such as a hard disk or memory. Furthermore, the storage medium may be located inside the computer (main memory, auxiliary storage, etc.) or outside the computer.

[0104] Figure 6 is a block diagram showing an example of the hardware configuration of each device (server devices 10, 20) in the embodiment described above. Each device may be implemented as a computer 7, for example, comprising a processor 101, main memory 102 (memory), auxiliary memory 103 (memory), network interface 104, and device interface 105, which are connected via a bus 106.

[0105] The computer 7 in Figure 6 has one of each component, but it may have multiple identical components. Also, although Figure 6 shows one computer 7, the software may be installed on multiple computers, and each of these computers may execute the same or different parts of the software's processing. In this case, it may be a distributed computing configuration in which each computer communicates via a network interface 104 or the like to execute processing. In other words, each device (server device 10, 20) in the above-described embodiment may be configured as a system that realizes its function by having one or more computers execute instructions stored in one or more storage devices. Alternatively, it may be configured so that information transmitted from a terminal is processed by one or more computers located on the cloud, and the processing results are transmitted to the terminal.

[0106] The various calculations performed by each device (server devices 10, 20) in the embodiments described above may be executed in parallel using one or more processors, or using multiple computers via a network. Alternatively, the various calculations may be distributed to multiple processing cores within a processor and executed in parallel. Furthermore, some or all of the processing and means of this disclosure may be implemented by at least one of a processor and a storage device located on a cloud that can communicate with computer 7 via a network. Thus, each device in the embodiments described above may be in the form of parallel computing using one or more computers.

[0107] The processor 101 may be an electronic circuit (processing circuit, processing circuitry, CPU, GPU, FPGA, ASIC, etc.) that performs either control or calculations for a computer. The processor 101 may also be a general-purpose processor, a dedicated processing circuit designed to perform specific calculations, or a semiconductor device including both a general-purpose processor and a dedicated processing circuit. Furthermore, the processor 101 may include optical circuits or quantum computing-based calculation functions.

[0108] The processor 101 may perform calculations based on data and software input from various devices within the computer 7, and may output calculation results and control signals to these devices. The processor 101 may also control the various components of the computer 7 by executing the computer 7's OS (Operating System) or applications.

[0109] Each of the devices (server devices 10, 20) in the embodiments described above may be implemented by one or more processors 101. Here, processor 101 may refer to one or more electronic circuits arranged on one chip, or one or more electronic circuits arranged on two or more chips or two or more devices. When multiple electronic circuits are used, each electronic circuit may communicate by wire or wireless.

[0110] The main memory 102 may store instructions executed by the processor 101 and various data, and the information stored in the main memory 102 may be read by the processor 101. The auxiliary memory 103 is a memory device other than the main memory 102. These memory devices refer to any electronic component capable of storing electronic information, and may be semiconductor memory. The semiconductor memory may be either volatile memory or non-volatile memory. In each of the devices (server devices 10, 20) in the above-described embodiment, the memory device for storing various data may be implemented by the main memory 102 or the auxiliary memory 103, or by the built-in memory of the processor 101. For example, each storage unit in the above-described embodiment may be implemented by the main memory 102 or the auxiliary memory 103.

[0111] In the embodiments described above, if each device (server device 10, 20) consists of at least one storage device (memory) and at least one processor connected to (coupled with) this at least one storage device, then at least one processor may be connected to one storage device. Also, at least one storage device may be connected to one processor. Furthermore, the configuration may include at least one processor among a plurality of processors being connected to at least one storage device among a plurality of storage devices. This configuration may also be realized by storage devices and processors included in a plurality of computers. Moreover, the configuration may include a storage device integrated with a processor (for example, a cache memory including an L1 cache and an L2 cache).

[0112] The network interface 104 is an interface for connecting to the communication network 8 wirelessly or via a wired connection. The network interface 104 can be any appropriate interface, such as one conforming to existing communication standards. Information may be exchanged between the computer 7 and the external device 9A connected via the communication network 8 through the network interface 104. The communication network 8 may be a WAN (Wide Area Network), LAN (Local Area Network), PAN (Personal Area Network), or a combination thereof, as long as it allows for information exchange between the computer 7 and the external device 9A. An example of a WAN is the Internet; an example of a LAN is IEEE 802.11 or Ethernet®; and an example of a PAN is Bluetooth® or NFC (Near Field Communication).

[0113] The device interface 105 is an interface such as USB that connects directly to the external device 9B.

[0114] External device 9A is a device connected to computer 7 via a network. External device 9B is a device directly connected to computer 7.

[0115] External device 9A or external device 9B may, for example, be an input device. The input device may be a camera, microphone, motion capture device, various sensors, keyboard, mouse, touch panel, etc., and provides the acquired information to the computer 7. Alternatively, it may be a device equipped with an input unit, memory, and processor, such as a personal computer, tablet terminal, or smartphone.

[0116] Furthermore, external device 9A or external device 9B may, for example, be an output device. The output device may be a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) panel, or a speaker that outputs sound, etc. It may also be a device equipped with an output unit, memory, and a processor, such as a personal computer, tablet terminal, or smartphone.

[0117] Furthermore, external devices 9A and 9B may be storage devices (memory). For example, external device 9A may be network storage, and external device 9B may be storage such as an HDD.

[0118] Furthermore, the external device 9A or external device 9B may be a device that has some of the functions of the components of each device (server devices 10, 20) in the embodiments described above. In other words, the computer 7 may transmit some or all of the processing results to the external device 9A or external device 9B, or may receive some or all of the processing results from the external device 9A or external device 9B.

[0119] In this specification (including the claims), when the expression "at least one of a, b, and c" or "at least one of a, b, or c" (including similar expressions) is used, it includes any of a, b, c, ab, ac, bc, or abc. Furthermore, any element may have multiple instances, such as aa, abb, aabbcc, etc. In addition, it is also possible to add other elements other than the enumerated elements (a, b, and c), such as abcd which has d.

[0120] In this specification (including the claims), when expressions such as "using data as input / based on data / according to / in accordance with data" (including similar expressions) are used, unless otherwise specified, this includes using the data itself or using data that has been processed in some way (e.g., data with added noise, normalized data, features extracted from the data, intermediate representations of the data, etc.). Furthermore, when it is stated that some result is obtained "using data as input / based on data / according to / in accordance with data" (including similar expressions), unless otherwise specified, this includes cases where the result is obtained based solely on the data in question or where the result is influenced by other data, factors, conditions, and / or states other than the data in question. Furthermore, when it is stated that "data is output" (including similar expressions), unless otherwise specified, this includes cases where the data itself is used as output or where data that has been processed in some way (e.g., data with added noise, normalized data, features extracted from the data, intermediate representations of various types of data, etc.) is used as output.

[0121] In this specification (including the claims), the terms “connected” and “coupled” are intended to be non-restrictive terms that include any direct connection / coupling, indirect connection / coupling, electrical connection / coupling, communicative connection / coupling, operational connection / coupling, physical connection / coupling, etc. The terms should be interpreted as appropriate in the context in which they are used, but any form of connection / coupling that is not intentionally or naturally excluded should be interpreted non-restrictively as being included in the terms.

[0122] In this specification (including the claims), when the expression "A configured to B" is used, it may include that the physical structure of element A has a configuration capable of performing operation B, and that the permanent or temporary setting / configuration of element A is configured to actually perform operation B. For example, if element A is a general-purpose processor, it is sufficient that the processor has a hardware configuration capable of performing operation B, and that it is configured to actually perform operation B by the setting of a permanent or temporary program (instruction). Furthermore, if element A is a dedicated processor, dedicated arithmetic circuit, etc., it is sufficient that the circuit structure of the processor is implemented to actually perform operation B, regardless of whether control instructions and data are actually attached.

[0123] Wherever terms meaning "comprising" or "possessing" (e.g., "comprising / including," "having," etc.) are used herein, they are intended to be open-ended terms, including cases where the subject matter of such terms is not the object of the term. Where the object of such terms meaning "comprising" or "possessing" is an expression that does not specify a quantity or suggests a singular number (an expression with the article "a" or "an"), such expression should be interpreted as not being limited to a specific number.

[0124] In this specification (including the claims), even if expressions such as "one or more" or "at least one" are used in some places, and expressions that do not specify a quantity or suggest a singularity (expressions using the articles a or an) are used in other places, the latter expressions are not intended to mean "one." In general, expressions that do not specify a quantity or suggest a singularity (expressions using the articles a or an) should not necessarily be interpreted as not being limited to a specific number.

[0125] In this specification, if a particular configuration of an embodiment is described as having a specific advantage or result, it should be understood, unless otherwise stated, that the same advantage or result can also be obtained from one or more other embodiments having that configuration. However, it should be understood that the presence or absence of such advantage or result generally depends on various factors, conditions, and / or states, and that the configuration does not necessarily guarantee that the advantage or result can be obtained. The advantage or result can only be obtained from the configuration described in the embodiment when various factors, conditions, and / or states are met, and the advantage or result cannot necessarily be obtained in the invention claimed to define that configuration or a similar configuration.

[0126] In this specification (including the claims), when multiple hardware components perform a predetermined process, each component may cooperate to perform the predetermined process, or some components may perform all of the predetermined process. Alternatively, some components may perform part of the predetermined process, while other components perform the remainder. In this specification (including the claims), when expressions such as "one or more hardware components perform process A, and the one or more hardware components perform process B" (including similar expressions) are used, the hardware component performing process A and the hardware component performing process B may be the same or different. In other words, it is sufficient that the hardware component performing process A and the hardware component performing process B are included in the one or more hardware components. Hardware may include electronic circuits, devices containing electronic circuits, etc.

[0127] In this specification (including the claims), when multiple memory devices store data, each of the multiple memory devices may store only a portion of the data or the entire data. Furthermore, a configuration in which some of the multiple memory devices store data is also included.

[0128] In this specification (including the claims), terms such as “first,” “second,” etc., are used merely as a way of distinguishing between two or more elements and are not necessarily intended to impose technical meanings such as temporal, spatial, order, or quantity on the subject. Therefore, for example, references to a first element and a second element do not necessarily mean that only two elements can be employed therein, that the first element must precede the second element, or that the first element must exist for the second element to exist.

[0129] While embodiments of this disclosure have been described in detail above, this disclosure is not limited to the individual embodiments described above. Various additions, modifications, substitutions, and partial deletions are possible, provided that they do not depart from the conceptual idea and spirit of the present invention derived from the claims and their equivalents. For example, where numerical values ​​or mathematical formulas are used in the description of the embodiments described above, these are provided for illustrative purposes only and do not limit the scope of this disclosure. Similarly, the sequence of operations shown in the embodiments is also illustrative and does not limit the scope of this disclosure.

[0130] Furthermore, the following forms are possible for disclosure technology.

[0131] (Note 1) At least one first memory, The system comprises at least one first processor connected to the at least one first memory, At least one second memory, The system comprises at least one second processor connected to at least one second memory, The at least one first processor is By inputting the input information into a machine learning model, intermediate data is generated. At least a portion of the intermediate data is transmitted to the at least one second memory, The at least one second processor is Using at least a portion of the intermediate data transmitted by the at least one first processor to the at least one second memory, output information of the machine learning model is generated. The ratio of the data transfer rate of the at least one first memory to the calculation speed of the at least one first processor is lower than the ratio of the data transfer rate of the at least one second memory to the calculation speed of the at least one second processor. Information processing system.

[0132] (Note 2) The at least one first memory stores at least some of the parameters of the machine learning model, The at least one second memory stores at least some of the parameters of the machine learning model. The information processing system described in Appendix 1.

[0133] (Note 3) The at least one first processor is The intermediate data is generated by performing the prefill process using the aforementioned machine learning model. The at least one second processor is Using the aforementioned machine learning model, the output information is generated by processing the decoding process. The information processing system described in Appendix 1 or 2.

[0134] (Note 4) The aforementioned machine learning model includes a neural network having an attention mechanism. The information processing system described in Appendix 3.

[0135] (Note 5) At least a portion of the aforementioned intermediate data includes at least a portion of the key vector and value vector used in the attention mechanism, The information processing system described in Appendix 4.

[0136] (Note 6) The at least one first processor is The intermediate data is generated by performing the first process, which is computationally limiting, using the aforementioned machine learning model. The at least one second processor is Using the aforementioned machine learning model, the output information is generated by a second process that is memory-limited. An information processing system as described in any of the appendices 1 to 5.

[0137] (Note 7) The first process described above is a prefill process using the machine learning model, The second process described above is the decoding process using the machine learning model. The information processing system described in Appendix 6.

[0138] (Note 8) The aforementioned machine learning model includes a state-space model, The information processing system described in Appendix 6.

[0139] (Note 9) At least a portion of the aforementioned intermediate data includes at least a portion of the states in the state-space model, The information processing system described in Appendix 8.

[0140] (Note 10) The at least one second processor is The second process is repeatedly executed using the output information generated in the second process that was previously executed. An information processing system as described in any of the appendices 6 to 9.

[0141] (Note 11) The at least one first processor is Before the generation of the intermediate data is completed, the transmission of at least a portion of the intermediate data is started. An information processing system as described in any of the appendices 1 through 10.

[0142] (Note 12) The at least one first processor is After the generation of the intermediate data is completed, the transmission of at least a portion of the intermediate data is started. An information processing system as described in any of the appendices 1 through 11.

[0143] (Note 13) The ratio of the data transfer rate to the calculation rate is the ratio of the total amount of data transferred to the total number of calculations when the entire process related to the machine learning model is executed. An information processing system as described in any of the appendices 1 to 12.

[0144] (Note 14) The ratio of the data transfer rate to the calculation rate is the ratio of the amount of data transferred per unit time to the number of calculations per unit time when processing the machine learning model. An information processing system as described in any of the appendices 1 to 13.

[0145] (Note 15) Multiple of the above-mentioned second memories, It comprises a plurality of the second processors, The at least one first processor is To each of the plurality of the second memories, at least a portion of the intermediate data is transmitted. An information processing system as described in any of the appendices 1 through 14.

[0146] (Note 16) The plurality of the second processors described above constitute a cluster. The information processing system described in Appendix 15.

[0147] (Note 17) Multiple of the above-mentioned first memories, The system comprises a plurality of the first processors, The plurality of first processors are At least a portion of the intermediate data generated by each of the plurality of first processors is combined and transmitted to the at least one second memory. An information processing system as described in any of the appendices 1 through 16.

[0148] (Note 18) The aforementioned information processing system is The machine learning model performs inference processing. An information processing system as described in any of the appendices 1 through 17.

[0149] (Note 19) The aforementioned information processing system is The inference process is performed for the training process of the machine learning model. The information processing system described in Appendix 18.

[0150] (Note 20) A first processing unit having at least one first memory and at least one first processor connected to the at least one first memory, A second processing unit having at least one second memory and at least one second processor connected to the at least one second memory, Having, An information processing system as described in any of the appendices 1 through 19.

[0151] (Note 21) At least one first processor connected to at least one first memory, By inputting the input information into a machine learning model, intermediate data is generated. At least a portion of the aforementioned intermediate data is transmitted to at least one second memory, The at least one second processor connected to the at least one second memory, Using at least a portion of the intermediate data transmitted by the at least one first processor to the at least one second memory, output information of the machine learning model is generated. The ratio of the data transfer rate of the at least one first memory to the calculation speed of the at least one first processor is lower than the ratio of the data transfer rate of the at least one second memory to the calculation speed of the at least one second processor. Information processing methods.

[0152] (Note 22) To at least one first processor connected to at least one first memory, By inputting the input information into a machine learning model, intermediate data is generated. At least a portion of the aforementioned intermediate data is transmitted to at least one second memory. Execute the process, To the at least one second processor connected to the at least one second memory, Using at least a portion of the intermediate data transmitted by the at least one first processor to the at least one second memory, the output information of the machine learning model is generated. Execute the process, The ratio of the data transfer rate of the at least one first memory to the calculation speed of the at least one first processor is lower than the ratio of the data transfer rate of the at least one second memory to the calculation speed of the at least one second processor. program.

[0153] This application claims priority to Japanese Patent Application No. 2024-173485, filed with the Japan Patent Office on 2 October 2024, which is incorporated herein by reference to its entire contents. [Explanation of Symbols]

[0154] 10,20: Server device 100,200: Processing unit 110,210: Processor 120,220: Memory 130,230: Network Interface 1000: Information Processing Systems

Claims

[Claim 1] At least one first memory, The system comprises at least one first processor connected to the at least one first memory, At least one second memory, The system comprises at least one second processor connected to the at least one second memory, The at least one first processor is By inputting the input information into a machine learning model, intermediate data is generated. The at least one second processor is Using at least a portion of the intermediate data generated by the at least one first processor, output information of the machine learning model is generated. The ratio of the data transfer rate of the at least one first memory to the calculation speed of the at least one first processor is lower than the ratio of the data transfer rate of the at least one second memory to the calculation speed of the at least one second processor. Information processing system.

Images