Artificial intelligence (AI) calculation method and system and electronic equipment

Abstract

The invention provides an artificial intelligence (AI) calculation method and system and electronic equipment, in the method, a memory processing PIM module is arranged in the electronic equipment, the PIM module has storage and calculation functions, a system-on-chip (SoC) executes calculation of a first stage in a first model, the SoC can write a calculation result of the first stage into the memory, and the calculation result of the first stage is stored in the memory. The first calculation unit can directly execute the calculation of the second stage by using the data in the memory, so that the time consumption caused by frequent data transmission between the SoC and the memory through a bus when the calculation of the second stage is executed in the prior art is avoided, the calculation efficiency of the second stage can be improved, and the calculation efficiency of the first model is further improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an artificial intelligence (AI) computing method, system, and electronic device. Background Technology

[0002] Artificial intelligence (AI) computing is applied across various fields, and AI computing efficiency is a crucial factor affecting user experience. In the field of human-computer interaction, large language model (LLM) computing is widely used. For example, in response to user input, electronic devices can use LLM to perform calculations and generate outputs to interact with the user.

[0003] The computational efficiency of LLM (Liquidity Logic Model) is a crucial factor affecting user experience. Low LLM efficiency leads to long waiting times for electronic device output, resulting in a poor user experience. Currently, in response to user input, the system-on-chip (SoC) in electronic devices can use LLM for computation to generate output. However, current LLM computations are inefficient. Summary of the Invention

[0004] This application provides an artificial intelligence (AI) computing method, system, and electronic device that can improve the computational efficiency of large language model (LLM).

[0005] Firstly, embodiments of this application provide an AI computing method applied in an electronic device. The electronic device may include a System-on-Chip (SoC) and a Processing Integrated Model (PIM) module in memory. The SoC is connected to the PIM module, which includes a first computing unit and memory. In this application, a PIM module is deployed in the electronic device. The PIM module may include memory and a first computing unit, and it can perform storage and computation functions. In this method, the SoC receives user input, which may include, but is not limited to, text, images, videos, and documents. The SoC uses a first model to perform a first-stage computation on the input, obtaining a computation result. The computation result includes the first character in the output responding to the input, and related data used to calculate other characters in the output.

[0006] The first model is used to perform calculations in response to input to obtain output. The first stage of calculation can be the stage from receiving input to outputting the first character in the output. After obtaining the calculation result of the first stage, the SoC can write the calculation result into memory. In this embodiment, the first computing unit can read the calculation result in memory and, based on the calculation result, use the first model to perform the second stage of calculation to sequentially obtain the other characters in the output. The second stage of calculation can be understood as the stage of sequentially obtaining the other characters in the output. When the first computing unit has sequentially obtained the other characters in the output, it can sequentially send the other characters to the SoC. Correspondingly, the SoC can sequentially output the characters in the output according to the character order, realizing continuous speech in the electronic device.

[0007] In this application, the computational efficiency of the first stage depends on the computing power of the SoC (such as the NPU in the SoC). Without changing the NPU hardware, to improve the computational efficiency of the first model, the computational efficiency of the second stage can be improved. This application can include a PIM module in the electronic device. The PIM module has storage and computation functions. The SoC executes the computation of the first stage in the first model. After obtaining the computation result of the first stage, the SoC can write the result into memory. The PIM module (such as the first computing unit) can directly execute the computation of the second stage using the data in memory (such as the computation result of the first stage). This avoids the time-consuming process caused by frequent data transfer between the SoC and memory during the execution of the second stage computation in the prior art. In this embodiment, the second stage computation can be achieved through interaction between the first computing unit in the PIM module and memory, without frequent data transfer via the bus. Instead, the first computing unit directly uses the data in memory, thus improving the computational efficiency of the second stage and consequently improving the computational efficiency of the first model.

[0008] In one possible implementation, the first model can be a Large Language Model (LLM). The first stage of the LLM may include the stage from receiving user input to the LLM outputting the first character of the answer, and the second stage may include the stage from the second character of the answer to the last character. In some embodiments, the first and second stages of the LLM may be divided in other ways. For example, the first stage may be the stage from receiving user input to the first character of the answer, and the second stage may include the stage from the first character of the answer to the last character.

[0009] Regardless of how the first and second stages of LLM are divided, when a large amount of data needs to be read from memory, the calculations for that stage can be performed by the PIM module. This can reduce the latency caused by data transmission and improve the computational efficiency of LLM.

[0010] In one possible implementation, the SoC adopts a first model. Before performing the first stage of computation on the input, the SoC sets the first computing unit to a first mode. When the first computing unit is in the first mode, the first computing unit is not allowed to access memory.

[0011] In this implementation, before the SoC performs the first stage of computation, the first computing unit can be set to the first mode. The purpose of this is that the SoC needs to access data in memory to perform the first stage of computation. Setting the first computing unit to the first mode can avoid the problem of high memory access pressure caused by the first computing unit accessing memory.

[0012] In one possible implementation, before the first computing unit performs the second stage of computation using the first model based on the computation result, the SoC sets the first computing unit to a second mode. When the first computing unit is in the second mode, the first computing unit is allowed to access memory.

[0013] In this implementation, the SoC sets the first computing unit to the first mode before performing the first stage of computation. However, when the first computing unit performs the second stage of computation, it needs to access data in memory to perform the computation. Therefore, the SoC needs to set the first computing unit to the second mode so that the first computing unit can access memory smoothly and perform the second stage of computation.

[0014] In one possible implementation, the electronic device further includes external memory for storing model parameters of the first model. In this implementation, after the SoC receives user input, it can send a first command to the first computing unit. In response to the first command, the first computing unit loads the model parameters from the external memory. Specifically, the first computing unit can read the model parameters from the external memory and write them into memory. In this implementation, upon completion of model parameter loading, the first computing unit sends feedback information to the SoC, indicating that the model parameter loading is complete.

[0015] In one possible implementation, the SoC employs a first model, and the necessary condition for performing the first stage computation on the input is the model parameters of the first model. Therefore, before performing the first stage computation on the input using the first model, the SoC can also read the model parameters from memory, including the model parameters corresponding to the first stage. Accordingly, the SoC employing the first model and performing the first stage computation on the input includes: the SoC using the model parameters corresponding to the first stage and performing the first stage computation on the input.

[0016] In this implementation, the first computing unit can load the model parameters into memory, so that when the SoC performs the first stage of calculation, it can directly read the model parameters from memory to perform the calculation, avoiding the time-consuming problem of reading model parameters from external memory.

[0017] Accordingly, the model parameters include the model parameters corresponding to the second stage. The first computing unit performs the calculation of the second stage using the first model based on the calculation results, including: the first computing unit performs the calculation of the second stage based on the calculation results and the model parameters corresponding to the second stage.

[0018] In this implementation, the first computing unit can load the model parameters into memory. When the first computing unit performs the first stage of calculation, it can directly read the model parameters from memory to perform the calculation, avoiding the time-consuming problem of reading the model parameters from external storage.

[0019] In one possible implementation, before the SoC sends the first command to the first computing unit, the SoC can start the first computing unit and enable the first computing unit to prepare for computation.

[0020] In one possible implementation, the SoC includes a second computing unit for receiving user input, employing a first model, performing a first-stage computation on the input, and obtaining a computation result. The computation result includes the first character in the output in response to the input, and related data for calculating other characters in the output. Additionally, the second computing unit is also used to write the computation result into memory. For example, the second computing unit is a neural processing unit.

[0021] In this implementation, the Neural Processing Unit (NPU) in the SoC is used to perform the first stage of computation. The NPU has strong computing power, which can improve the efficiency of the first stage of computation.

[0022] Secondly, embodiments of this application provide an AI computing system, which includes: a System-on-a-Chip (SoC) and a Processing Integrated Model (PIM) module in memory. The SoC is connected to the PIM module, and the PIM module includes a first computing unit and memory. The SoC is used to execute the operations performed by the SoC in the first aspect or any possible implementation of the first aspect. The first computing unit is used to execute the operations performed by the first computing unit in the first aspect or any possible implementation of the first aspect.

[0023] In one possible implementation, the AI ​​computing system also includes external memory for storing model parameters.

[0024] In one possible implementation, the AI ​​computing system also includes internal memory that is not the memory in the PIM module.

[0025] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, the memory for storing code instructions, and the processor for running the code instructions to perform the methods described in the first aspect or any possible implementation of the first aspect.

[0026] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program or instructions that, when executed on a computer, cause the computer to perform the methods described in the first aspect or any possible implementation thereof.

[0027] Fifthly, embodiments of this application provide a computer program product including a computer program, which, when run on a computer, causes the computer to perform the methods described in the first aspect or any possible implementation thereof.

[0028] Sixthly, this application provides a chip or chip system including at least one processor and a communication interface. The communication interface and the at least one processor are interconnected via a circuit. The at least one processor is used to run computer programs or instructions to perform the methods described in the first aspect or any possible implementation thereof. The communication interface in the chip can be an input / output interface, pins, or circuits, etc.

[0029] In one possible implementation, the chip or chip system described above in this application further includes at least one memory storing instructions. The memory can be an internal storage unit of the chip, such as a register or cache, or it can be a storage unit of the chip itself (e.g., read-only memory, random access memory, etc.).

[0030] It should be understood that the second to sixth aspects of this application correspond to the technical solutions of the first aspect of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be repeated here. Attached Figure Description

[0031] Figure 1 A schematic diagram illustrating a scenario where an electronic device uses LLM to interact with the user.

[0032] Figure 2 A schematic diagram of the structure of an electronic device;

[0033] Figure 3 A flowchart illustrating LLM computation;

[0034] Figure 4 A flowchart illustrating LLM calculations for electronic devices;

[0035] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0036] Figure 6 A flowchart illustrating one embodiment of the AI ​​computing method provided in this application;

[0037] Figure 7 A flowchart illustrating another embodiment of the AI ​​computing method provided in this application;

[0038] Figure 8 This is a flowchart illustrating another embodiment of the AI ​​computing method provided in this application. Detailed Implementation

[0039] Large language model (LLM) is a natural language processing model based on deep learning that can handle a variety of language tasks, such as text classification, question answering, dialogue, and machine translation. Figure 1 A schematic diagram illustrating a scenario where an electronic device uses LLM to interact with the user. Figure 1 Taking the China-Israel question-and-answer task as an example, refer to Figure 1 In the example 'a', the electronic device can display a question-and-answer page 101, which may include an input box 11, a sending control 12, and a dialog box 13. The user can enter a question in the input box 11, for example... Figure 1 In example 'a', we take the user entering "Question 1" in input box 11 as an example. After the user enters "Question 1" in input box 11, they can operate the send control 12.

[0040] In response to user interaction, send control 12, refer to Figure 1 In step b, the electronic device can display question 1 in dialog box 13. The electronic device can use LLM to parse and process question 1 and generate answer 1 corresponding to question 1. Specifically, when generating answer 1 for question 1, LLM can first generate the first character of answer 1, and then generate subsequent characters based on the first character. In other words, when generating answer 1 for question 1, LLM can generate the characters in answer 1 sequentially based on already generated characters, context information, and language rules. Correspondingly, because LLM does not obtain answer 1 all at once, but generates the characters in answer 1 sequentially, when the electronic device outputs answer 1 in dialog box 13, it will output the characters in answer 1 sequentially, as can be seen from... Figure 1 As shown in ce. The electronic device outputs the characters in answer 1 sequentially, giving the user the impression that the electronic device is uttering words continuously.

[0041] Understandable, Figure 1Taking a question-and-answer task as an example, a user can input text, and the electronic device can use LLM (Liquidity Management Model) to parse and process the text to generate the corresponding answer. In some embodiments, the user can also input voice, images, or documents. Specifically, when a user inputs voice, the electronic device can convert the voice into text, then use LLM to parse and process the text to generate the corresponding answer. When a user inputs an image, the electronic device can extract key information from the image, convert it into text, and then use LLM to parse and process the text to generate the corresponding answer. When a user inputs at least one document, the user can ask a question about at least one of the documents, and the electronic device can parse and process the corresponding document to generate the corresponding answer. This application does not limit the application scenarios of LLM.

[0042] To facilitate understanding of the artificial intelligence (AI) calculation method provided in the embodiments of this application, the following first introduces the current method of using LLM for calculation in electronic devices:

[0043] Figure 2 This is a schematic diagram of an electronic device. (Refer to...) Figure 2 The electronic device may include: a system on chip (SoC) 21, internal memory 22, and external memory 23.

[0044] At least one processor can be integrated on the SoC 21. For example, the processor may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, a display processing unit (DPU), and / or a neural network processing unit (NPU), etc.

[0045] Internal memory 22 can be simply referred to as RAM. RAM is used to store the programs and data currently running in the electronic device. The processor on SoC 21 can directly access RAM and perform corresponding calculations based on the data and program instructions in RAM. For example, when the electronic device uses LLM for calculations, the NPU on SoC 21 can access the programs and data in RAM and perform the corresponding LLM calculations. The specific calculation process can be referred to the description in the following embodiments.

[0046] In some embodiments, the internal memory 22 may include random access memory (RAM) and read-only memory (ROM). For example, RAM may include dynamic random access memory (DRAM), static random access memory (SRAM), and double data rate synchronous dynamic random access memory (DDR SDRAM). For example, ROM may include programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), and electrically erasable programmable read-only memory (EEPROM).

[0047] External storage device 22 may include, but is not limited to, disks, hard disks, flash memory devices, universal flash storage (UFS), etc.

[0048] In this embodiment, the external memory 22 is used to store the model parameters of the LLM, which can be referred to the description in the following embodiments.

[0049] Internal memory 22 corresponds to external memory 23. Internal memory 22 is directly connected to SoC 21 via a bus and is directly accessed by the processor on SoC 21. In contrast, external memory 23 can be connected to SoC 21 via an interface, and its data access speed is relatively slower. Figure 2 The bus is represented by a rectangle.

[0050] The following is based on Figure 2 The structure of the electronic device is shown, illustrating the current methods for LLM computation in electronic devices. In some embodiments, refer to... Figure 3 LLM computation can include two computational phases, namely the first phase and the second phase.

[0051] The first stage can be called the summary stage. The summary stage includes the period from receiving user input to the LLM outputting the first character of the answer. The second stage can be called the generation stage. The generation stage can be understood as the LLM's continuous pronunciation stage; for example, the LLM sequentially generates the second character of the answer until the last character. Correspondingly, the user can see the electronic device sequentially outputting the answer characters on the device's interface until the answer is completely output.

[0052] The summary stage primarily involves parsing and processing the input. In some embodiments, the summary stage can automatically extract key information from the text using specific algorithms and model structures, generating a concise and accurate summary that accurately reflects the main content of the input. The summary stage also generates the first character of the answer based on the summary.

[0053] The generation phase is primarily used to generate the answer, specifically by continuously generating the remaining characters in the answer. These remaining characters refer to all characters except the first character of the answer. In LLM, the generation phase generates a grammatically and semantically coherent answer by understanding the input and the first character of the answer. In some embodiments, the answer can maintain grammatical, semantic, and stylistic consistency with the input.

[0054] Reference Figure 3 Here is a brief description of the LLM calculation process:

[0055] It is understood that an LLM comprises multiple computational layers that work together to process input and generate corresponding outputs (such as answers). In some embodiments, an LLM may include: an input layer, an input embedding layer, a self attention layer, a positional encoding layer, and an output layer. This application does not limit the division of computational layers in an LLM; the division of computational layers in this application is merely an example.

[0056] The input layer performs a series of preprocessing steps on the input (such as text), such as word segmentation, stop word removal, and stemming, to obtain a series of tokens, which can be called the input sequence. A token is the basic unit in text processing and can be a character, word, punctuation mark, etc. This series of tokens can be input to the embedding layer. The embedding layer converts each token into a numerical form that the LLM can process, such as an embedding vector, to obtain the embedding vector corresponding to each token. Through the embedding layer, the LLM can capture semantic and syntactic information in the text, providing input for subsequent computational layers. Because the LLM does not include loops or convolutional structures, a positional encoding layer is needed to provide the positional information of the tokens in the input (text). In some embodiments, the positional encoding layer is often used in conjunction with the embedding layer. The positional encoding layer can incorporate the positional information of the tokens into the embedding vector so that the LLM can distinguish tokens at different positions in the text.

[0057] For example, the input is "China's largest city". After processing by the input layer, we can obtain the tokens of the 7 characters in the phrase "China's largest city". After processing by the embedding layer, we can obtain the embedding vectors corresponding to these 7 characters.

[0058] The embedding vector corresponding to each token passes through a self-attention layer, which uses a self-attention mechanism to process the embedding vector corresponding to each token. In the following embodiments, the embedding vector corresponding to each token will be referred to simply as "token," and the process of the self-attention layer processing the embedding vector corresponding to each token will be described.

[0059] The self-attention mechanism allows the self-attention layer to focus on other positions while processing the token at each position (i.e., the embedding vector corresponding to each token), thereby capturing long-distance dependencies in the text. The processing procedure of the self-attention layer is briefly described below:

[0060] 1. For each token in the input sequence, the self-attention layer can compute the corresponding query (Q) vector, key (K) vector, and value (V) vector. In some embodiments, the self-attention layer can multiply each token by three different weight matrices to obtain the Q vector, K vector, and V vector. For example, for the input "China's largest city", the self-attention layer can obtain the Q vector, K vector, and V vector for each of the 7 tokens.

[0061] 2. For each of the Q, K, and V vectors of the 7 tokens in the input, they can serve as context information for generating a new token. The new token can be regarded as the first character of the answer (such as "three"). The self-attention layer can generate a new token, as well as the Q, K, and V vectors of the new token, based on the Q, K, and V vectors of each of the 7 tokens.

[0062] 3. After generating a new token, the self-attention layer can continue to generate new tokens based on the tokens that have been generated to achieve the effect of continuous speech. For example, the new token generated by the self-attention layer in "2" is the first character of the answer. The self-attention layer can generate the second character of the answer (such as "sand") based on the Q vector of this first character and the K and V vectors of 8 characters (including the 7 input characters and the first character of the answer). This second character can be called the new token. Similarly, the self-attention layer can also generate the Q, K, and V vectors of the new token.

[0063] It should be understood that the specific process of how the self-attention layer in the embodiments of this application generates a new token and the Q, K, and V vectors of the new token will not be elaborated here, and reference can be made to the relevant descriptions of the current self-attention mechanism.

[0064] Referring to the above processing process of the self-attention layer, when the self-attention layer generates a new token each time, the K and V vectors of the previous tokens need to be used. To avoid repeatedly calculating the K and V vectors of each previous token when generating a new token, in some embodiments, the K and V vectors of the tokens can be cached to obtain a KV cache (Key Value cache). Among them, in the KV cache, the K and V vectors of the tokens can be sorted according to the order of the tokens respectively. When the self-attention layer generates a new token, the K and V vectors of the new token can be concatenated to the end of the existing KV cache to update the KV cache. In this way, when the self-attention layer generates a new token each time, by accessing the KV cache, the K and V vectors of each previous token can be obtained. Exemplarily, the KV cache can include the K vectors sorted according to the order of the tokens, and the V vectors sorted according to the order of the tokens. After generating the K and V vectors of the new token, the new K vector can be concatenated to the end of the K vectors, and the new V vector can be concatenated to the end of the V vectors.

[0065] Similarly, each time the self-attention layer generates a new token, it needs to use the Q-vector of the previous token. Therefore, in this embodiment, the Q-vector of the previous token can also be stored. Thus, each time the self-attention layer generates a new token, it accesses the KV cache to retrieve the K and V vectors of each previous token and reads the stored Q-vector of the previous token. The self-attention layer can then generate a new token based on the KV cache and the Q-vector of the previous token. After generating a new token, the self-attention layer can replace the Q-vector of the previous token with the Q-vector of the new token to update the stored Q-vector for use in generating the new token.

[0066] In summary, by repeating the above calculation steps, the self-attention layer can continuously generate new tokens until the last character of the answer is displayed. It can be understood that each time the self-attention layer generates a new token, it can output the new token to the output layer, so that the output layer can convert the new token into a character output. This way, the user can see the electronic device display the characters of the answer sequentially until the last character.

[0067] The following is combined Figure 4 This section details the process of LLM calculations performed on electronic devices. Figure 4 A flowchart illustrating LLM calculations for electronic devices. (Refer to...) Figure 4 The process may include:

[0068] Step 1: In response to user input, the NPU reads model parameters from external storage and writes them to a preset location in memory.

[0069] In response to user input, the electronic device can perform calculations using LLM (Liquidity Management Model). Since the model parameters are stored in the electronic device's external memory, the NPU (Neural Processing Unit) needs to read the model parameters from the external memory into main memory before performing calculations. Specifically, the NPU can read the model parameters from the external memory and write them to a preset location in main memory for use during LLM calculations. In some embodiments, for example, the NPU can write the model parameters to a preset location in DRAM.

[0070] In some embodiments, model parameters may include weights for each computational layer. For example, weights may exist in the form of numerical values, weight matrices, etc. The weights for each computational layer are used to perform the computations of that layer.

[0071] Step 2: The NPU reads the model parameters from memory and performs LLM calculations based on the user's input.

[0072] Reference Figure 3The description of the computational layers in LLM is as follows: The NPU can access memory, read the weights of the input layer from memory, process the input, and obtain at least one token. After obtaining at least one token, the NPU can access memory, read the weights of the embedding layer from memory, process each token, and obtain the embedding vector corresponding to each token. After obtaining the embedding vector corresponding to each token, the NPU can access memory, read the weights of the self-attention layer from memory, and process each token, as described in 1-3 of the above embodiments. After obtaining a new token, the NPU can access memory, read the weights of the output layer from memory, and output the character corresponding to the new token.

[0073] In the first stage, the NPU accesses memory to read the weights of the input layer and the embedding layer to process the input. The number of memory accesses by the NPU in the first stage is relatively small; therefore, the computational efficiency of the first stage depends on the NPU's computing power. In the second stage (i.e., the continuous speech generation stage), referring to the description of generating new tokens in the self-attention layer above, the KV cache and the Q-vector of the previous token are both in memory. Each time the self-attention layer generates a new token, the NPU needs to access memory once to retrieve the KV cache and the Q-vector of the previous token for generating the new token.

[0074] Typically, the output (i.e., the answer) can contain many characters. In the second stage, the NPU needs to repeatedly access memory to retrieve the KV cache and the Q-vector of the previous token. The NPU also needs to repeatedly compute multiple new tokens. Because the NPU accesses memory more frequently in the second stage, and the amount of data transferred between the SoC and memory increases with the amount of data in the KV cache, the computational efficiency of the second stage depends on the data transfer speed of the bus between the SoC and memory. Since the data transfer speed of the bus is related to the bus bandwidth, the computational efficiency of the second stage is dependent on the bus bandwidth. Currently, the low bandwidth of the bus between the SoC and memory in electronic devices affects the computational efficiency of the second stage in LLM, thus leading to low computational efficiency in LLM.

[0075] In summary, the computational efficiency of the first stage currently relies on the computing power of the NPU. To improve the computational efficiency of LLM without changing the NPU hardware, improving the computational efficiency of the second stage can be considered. This is because the computational efficiency of the second stage depends on the data transfer speed and bandwidth of the bus between the SoC and memory. In some embodiments, the bandwidth of the bus between the SoC and memory can be increased to improve the data transfer speed, but this inevitably increases the cost of the electronic device.

[0076] Because the NPU needs to access memory multiple times to retrieve the KV cache and the Q vector of the previous token during the second computation phase, in order to improve access and data transfer efficiency, in some embodiments, a processing in memory (PIM) module can be deployed in the electronic device. The PIM module can be understood as containing memory and integrating computational units within that memory; that is, the PIM module integrates the functions of both memory and computational units. In this way, the PIM module can directly execute the second phase using data in its own memory without frequent data transfer between the SoC and memory, thereby improving the computational efficiency of the second phase and consequently improving the computational efficiency of LLM.

[0077] In some embodiments, computing units can also be embedded in the memory of the electronic device (such as internal memory 22). In this way, the computing units can also directly execute the second stage using the data in memory without frequent data transfers between the SoC and memory, which can also improve the computing efficiency of the second stage, and thus improve the computing efficiency of LLM.

[0078] The following embodiments use the deployment of a PIM module in an electronic device as an example. The memory in the following embodiments can be understood as the memory in the PIM.

[0079] Before introducing the artificial intelligence (AI) calculation method provided in the embodiments of this application, the structure of the electronic device provided in the embodiments of this application will be introduced first. Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. (Refer to...) Figure 5 The electronic device may include: a SoC 51, internal memory 52, external memory 53, and a PIM module 54. The SoC 51, internal memory 52, and external memory 53 can be referred to respectively... Figure 2 Description of SoC 21, internal memory 22, and external memory 23.

[0080] In this embodiment, when the electronic device executes the second stage of LLM, the PIM module can directly read the data in the memory of the PIM module to execute the second stage, avoiding the process of frequent data transfer between the SoC and the internal memory 22, reducing the time spent on data transfer, improving the computational efficiency of the second stage, and thus improving the computational efficiency of LLM.

[0081] It's understandable that an NPU can be integrated and configured on the SoC 51. Other processors can also be integrated and configured on the SoC 51; see [link to relevant documentation] for details. Figure 2 The description in the text, Figure 5 The SoC 51 includes the NPU involved in the embodiments of this application as an example.

[0082] In this embodiment, the NPU is used to execute the first stage of the LLM. In some embodiments, the NPU can be regarded as a second computing unit.

[0083] PIM module 54 is used to perform the second stage in LLM. In some embodiments, PIM module 54 may include a first computing unit and memory. In some embodiments, to facilitate the distinction between the memory in PIM module 54 and the internal memory 22 in the electronic device, the internal memory 22 may be referred to as the first memory, and the memory in PIM module 54 may be referred to as the second memory. The memory in the following embodiments can be understood as the second memory.

[0084] It is understood that the actions performed by the first computing unit and memory in the following embodiments can be regarded as actions performed by the PIM module 54.

[0085] The functions of the NPU and PIM module 54 can be referred to the description in the following embodiments. In some embodiments, the actions performed by the NPU can be regarded as the actions performed by the SoC 51.

[0086] The AI ​​computing method provided in this application will be described below with reference to specific embodiments. These embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The AI ​​computing method provided in this application will be introduced below from the perspective of the interaction between the NPU, PIM module, and external memory.

[0087] Figure 6 This is a flowchart illustrating one embodiment of the AI ​​computing method provided in this application. (Refer to...) Figure 6 The AI ​​calculation method provided in this application embodiment may include:

[0088] S601, in response to user input, the NPU starts the PIM module.

[0089] In response to user input, the NPU can instruct the power supply module of the electronic device to power on the PIM module. For example, the NPU can send a first instruction to the power supply module, instructing it to power on the PIM module. In response to the first instruction, the power supply module supplies power to the PIM module. After the PIM module is powered on, code can be loaded and started running, enabling the NPU to start the PIM module.

[0090] S602, the NPU sends a first command to the PIM module, which instructs the PIM module to load model parameters from external memory.

[0091] After the NPU starts the PIM module, the NPU can send a first command to the PIM module to instruct the PIM module to load model parameters from external memory.

[0092] Model parameters may include the weights of each computational level of the LMM. The computational levels of the LMM and the weights can be referred to the description in the above embodiments.

[0093] S603, in response to the first command, the PIM module loads model parameters from external memory.

[0094] In some embodiments, in response to a first command, the PIM module can read model parameters from external memory and write them to a preset location in the PIM module's memory. For example, the PIM module can access the preset location in external memory to read model parameters, wherein the preset location in external memory is used to store model parameters.

[0095] In some embodiments, the preset location in memory used to store model parameters may be referred to as the second preset location.

[0096] In some embodiments, S602 is an optional step. For example, model parameters may be pre-stored in a second preset location in memory. In this example, in response to user input, the NPU may launch the PIM module and execute S605. In some embodiments, when S605 is an optional step, in response to user input, the NPU may launch the PIM module and execute S606.

[0097] S604, the PIM module sends feedback information to the NPU, which indicates that the model parameters have been loaded.

[0098] Once the PIM module has finished loading the model parameters from external memory, i.e., once the model parameters have been loaded, the PIM module sends feedback information to the NPU to indicate that the PIM module has finished loading the model parameters.

[0099] In response to feedback information, the S605 NPU sets the PIM module to the first mode.

[0100] Specifically, when the PIM module is in its first mode, the first computational unit within the PIM module is not allowed to access the memory within the PIM module. In other words, when the PIM module is in its first mode, the first computational unit cannot read model parameters from memory, meaning the PIM module cannot perform calculations within the LLM.

[0101] In some embodiments, in response to feedback information, the NPU can send a second instruction to the memory in the PIM module. This second instruction instructs the memory to set the first computational unit in the PIM module to be inaccessible. In response to the second instruction, the memory can set itself to disallow access to the first computational unit in the PIM module, i.e., set the PIM module to a first mode. Specifically, when the memory is set to disallow access to the first computational unit in the PIM module, if the first computational unit in the PIM module attempts to access the memory, the memory determines, based on the access permission settings, that the first computational unit in the PIM module is not allowed to access, and the memory can send feedback information indicating that access is not permitted to the first computational unit in the PIM module.

[0102] In some embodiments, S605 is an optional step.

[0103] S606, the NPU reads the model parameters corresponding to the first stage from memory.

[0104] The memory in S606 can be understood as the memory in the PIM module. In this embodiment, the NPU is configured to perform the first stage of LLM calculation. After the NPU sets the PIM module to the first mode, the NPU can read the model parameters corresponding to the first stage from memory. For example, the NPU can access a second preset location in memory to read the model parameters corresponding to the first stage from memory.

[0105] In some embodiments, the first stage is the period from receiving user input to generating the first character in the output. The model parameters corresponding to the first stage may include: the weights of the input layer, the weights of the positional encoding layer, the weights of the embedding layer, and the weights of the self-attention layer.

[0106] S607, the NPU performs the first calculation on the input based on the model parameters corresponding to the first stage, and obtains the calculation result.

[0107] Referring to the description of the LLM calculation process in the above embodiments, the NPU can perform the first calculation on the input according to the model parameters corresponding to the first stage, and obtain the calculation result of the first stage, which is referred to as the calculation result.

[0108] Among them, the calculation result may include: the first character in the output in response to the input, and the relevant data for calculating other characters in the output. Referring to the example in the above embodiment, for example, the output is "Sansha City", the first character in the output may be "San", and the relevant data for calculating other characters in the output may include: the token corresponding to the input, and the K vector and V vector of each token, as well as the Q vector, K vector, and V vector of the first character (token). In other words, the relevant data for calculating other characters in the output may include: the KV cache, and the Q vector corresponding to the first character in the output. Among them, the KV cache may include the K vector and V vector of the token corresponding to the input, and the K vector and V vector of the newly generated token (the first character).

[0109] S608, the NPU writes the calculation result to the first preset position in the memory.

[0110] The memory in S6 can be understood as the memory in the PIM module. The NPU writes the calculation result to the first preset position in the memory so that the PIM module can perform the second-stage calculation based on the calculation result. <0OO0258>In some embodiments, the first preset position may be the same as or different from the second preset position.

[0112] S609, the NPU sets the PIM module to the second mode.

[0113] The memory in S609 can be understood as the memory in the PIM module. Among them, when the PIM module is in the second mode, the first calculation unit in the PIM module is allowed to access the memory. In other words, when the PIM module is in the second mode, the first calculation unit in the PIM module can access the memory to read the model parameters from the memory and perform the second-stage calculation in the LLM.

[0114] In some embodiments, after the NPU writes the calculation result to the first preset position in the memory, the NPU may send a third instruction to the memory, and the third instruction is used to instruct the memory to set the first calculation unit in the PIM module to be allowed to access. In response to the third instruction, the memory can set the first calculation unit in the PIM module to be allowed to access the memory, that is, set the PIM module to the second mode. Among them, when the first calculation unit in the PIM module accesses the memory in the case where the memory sets the first calculation unit in the PIM module to be allowed to access the memory, the memory determines that the first calculation unit in the PIM module is allowed to access according to the access permission setting, and the memory can send data to the first calculation unit in the PIM module, where the data may include the calculation result and the model parameters corresponding to the second stage.

[0115] In some embodiments, S609 is an optional step.

[0116] S610, the PIM module (such as the first computing unit) accesses the first preset location in memory to obtain the calculation result.

[0117] S611, the PIM module (such as the first computing unit) accesses the second preset location in memory to obtain the model parameters corresponding to the second stage.

[0118] The embodiments of this application do not limit the order of S611 and S610.

[0119] When the PIM module is in the second mode, the PIM module (such as the first computational unit in the PIM module) can access memory. Specifically, the PIM module accesses a first preset location in memory to obtain the calculation results of the first stage, and the PIM module (such as the first computational unit in the PIM module) can access a second preset location in memory to obtain the model parameters corresponding to the second stage.

[0120] In some embodiments, the model parameters corresponding to the second stage may include: the weights of the self-attention layer and the weights of the output layer.

[0121] S612, the PIM module performs the second-stage calculation based on the calculation results and the model parameters corresponding to the second stage, and obtains the other characters in the output in sequence.

[0122] The PIM module performs the second-stage calculation based on the calculation results and the model parameters corresponding to the second stage. For example, the PIM module can perform the following steps:

[0123] Step 1: Based on the KV cache and the Q vector of the previous token, generate a new token, along with its corresponding Q vector, K vector, and V vector. The previous token can be the token corresponding to the first character in the output.

[0124] Step 2: The K vector and V vector corresponding to the new token can be concatenated to the KV cache to update the KV cache.

[0125] Step 3: Replace the Q vector of the previous token with the Q vector corresponding to the new token.

[0126] Step 4: Repeat steps 1-4 until the token corresponding to the last character in the output is generated, along with the Q vector, K vector, and V vector corresponding to that token.

[0127] After the PIM module generates tokens corresponding to characters other than the first character in the sequential output, the PIM module can sequentially convert the tokens corresponding to the other characters into the corresponding characters according to the weights of the output layer, that is, the PIM module can sequentially obtain the other characters in the output.

[0128] In some embodiments, S605 and S609 are optional steps. In this example, the PIM module can be configured to: in response to the NPU writing the calculation result to the first preset location in the memory, the PIM module can access the second preset location in the memory to obtain the model parameters of the second stage, and the PIM module can access the first preset location in the memory to obtain the calculation result of the NPU. The PIM module can perform the calculation of the second stage according to the calculation result and the model parameters of the second stage.

[0129] S613, the PIM module sequentially sends the other characters in the output to the NPU.

[0130] Because the PIM module can sequentially obtain the other characters in the output, the PIM module can sequentially send the other characters in the output to the NPU. Exemplarily, taking the input "the largest city in China" as an example, the output is "Sansha City". The output can include three characters: "San", "Sha", and "City". Among them, after the second-stage calculation by the PIM module, it can first obtain the character "Sha", and then obtain the character "City". Among them, after the PIM module obtains the character "Sha", it can send the character "Sha" to the NPU. After the PIM module obtains the character "City", it can send the character "City" to the NPU.

[0131] Correspondingly, the NPU can sequentially obtain the other characters in the output.

[0132] S614, the NPU sequentially outputs the characters in the output in the order of the characters.

[0133] After receiving the other characters in the output, the NPU can sequentially output the characters in the output. Among them, the characters in the output can include the first character in the input and the other characters. Among them, the other characters can be regarded as the characters in the output except the first character.

[0134] In some embodiments, the manner in which the NPU outputs the characters in the output can be: the NPU controls the display screen of the electronic device to display the characters in the output, or the NPU controls the speaker of the electronic device to play the characters in the output. The embodiments of the present application do not limit the manner in which the electronic device outputs the characters in the output.

[0135] In some embodiments, taking the example of an electronic device sequentially displaying characters in the output, since the NPU can perform the calculations in the first stage, the NPU can obtain the first character in the output. After obtaining the first character, the NPU can control the electronic device to display the first character. After the NPU sequentially receives other characters from the PIM module, the NPU can control the electronic device to sequentially display the other characters in the output in the order of receiving the characters.

[0136] Exemplarily, taking the output as "Sansha City" as an example, the NPU performs the calculations in the first stage. After obtaining the character "San", the electronic device can display the character "San". During the process of the PIM module performing the second calculation, the NPU module can first receive the character "Sha" from the PIM module, and the electronic device can display the character "Sha". Then the NPU receives the character "Shi", and the electronic device can display the character "Shi". Accordingly, the user's perception is that the electronic device sequentially displays the characters in the output, that is, the electronic device speaks continuously.

[0137] Figure 7 It is a schematic flowchart of another embodiment of the AI calculation method provided by the embodiments of the present application. It should be understood that Figure 7 the internal memory 52 (i.e., the first memory) is not shown. Referring to Figure 7 , the AI calculation method provided by the embodiments of the present application may include:

[0138] Step 1A: The NPU instructs the PIM module to write the model parameters to the second preset position in the memory.

[0139] Step 1A can be referred to the description in S602 - S604.

[0140] In some embodiments, in response to the user's input, the NPU can directly read the model parameters from the external memory, and the NPU writes the model parameters to the second preset position in the memory. In this example, Step 1A can be replaced with: The NPU writes the model parameters to the second preset position in the memory.

[0141] Step 2A: The NPU performs the calculations in the first stage according to the model parameters in the first stage, and writes the calculation results of the first stage to the first preset position in the memory.

[0142] Step 2A can be referred to the description in S606 - S608.

[0143] Step 3A: The PIM module performs the calculations in the second stage according to the calculation results and the model parameters in the second stage to obtain the other characters in the output.

[0144] Step 3A can be referred to the description in S610 - S612.

[0145] It should be understood that, Figure 7 This section primarily describes how the NPU and PIM modules perform calculations at different stages of the LLM separately. The remaining steps can be found in [the relevant documentation / reference]. Figure 6 The description in the illustrated embodiments.

[0146] In this embodiment, a PIM module can be set in the electronic device, and the NPU performs the first stage of calculation in LLM. After the NPU obtains the calculation result of the first stage, the NPU can write the calculation result of the first stage into memory. The PIM module can use the data in memory (such as the calculation result of the first stage) to directly perform the second stage calculation, which avoids the time-consuming process caused by frequent data transfer between SoC and memory when performing the second stage calculation in the prior art. This embodiment can improve the calculation efficiency of the second stage, thereby improving the calculation efficiency of LLM.

[0147] The AI ​​computing method provided in this application was described above from the perspective of the interaction between the NPU and PIM modules. The following describes the AI ​​computing method provided in this application from the perspective of an electronic device. In some embodiments, refer to... Figure 8 The AI ​​calculation method provided in this application embodiment may include:

[0148] S801, the SoC receives user input.

[0149] Reference Figure 5 As described in the text, the electronic device includes a system-on-a-chip (SoC) and a PIM module. The SoC is connected to the PIM module, and the PIM module includes a first computing unit and memory.

[0150] In some embodiments, user input may include, but is not limited to, text, images, videos, documents, etc.

[0151] S802, the SoC uses the first model to perform the first stage of calculation on the input and obtain the calculation result, which includes: the first character in the output in response to the input, and related data used to calculate other characters in the output.

[0152] Specifically, the SoC uses a first model to perform the first stage of computation on the input. Specifically, the SoC uses the model parameters corresponding to the first stage in the first model to perform the first stage of computation on the input.

[0153] S802 can be referred to in the descriptions in S606-S607.

[0154] S803, the SoC writes the calculation results into memory.

[0155] S803 can be referred to the description in S608.

[0156] S804, the first computing unit performs the second stage of computing based on the computing results in memory, using the first model, and sequentially obtains the other characters in the output.

[0157] The first computing unit can read the calculation results from memory and, based on the calculation results, execute the second stage of calculation using the first model. Specifically, the first computing unit executes the second stage of calculation using the model parameters corresponding to the second stage in the first model, based on the calculation results.

[0158] S802 can be referred to in the descriptions in S610-S612.

[0159] S805, the first computing unit sends other characters to the SoC in sequence.

[0160] S805 can be referred to the description in S613.

[0161] The S806 SoC outputs the characters in the output sequentially according to their order.

[0162] S806 can be referred to the description in S614.

[0163] The embodiments of this application have the same characteristics as... Figure 6 The same technical effects are achieved in the embodiments shown, and can be referred to the description in the above embodiments.

[0164] As described in the above embodiments, the first stage of the LLM may include: the stage from receiving user input to the LLM outputting the first character of the answer, and the second stage may include: the stage from the second character of the answer to the last character in sequence. In some embodiments, the first and second stages may be divided in other ways. For example, the first stage may be: the stage from receiving user input to the first character of the answer output by the LLM, and the second stage may include: the stage from the first character of the answer to the last character in sequence.

[0165] In this example, the first stage of LLM calculation can be performed by the NPU, and the second stage calculation can be performed by the PIM module. Unlike the previous embodiment, in this embodiment, the calculation result of the first stage can include the Q-vector, K-vector, and V-vector corresponding to each token in the input. The Q-vector, K-vector, and V-vector corresponding to each token in the input are used to generate a new token, and the Q-vector, K-vector, and V-vector of the new token. Generating the new token, and the Q-vector, K-vector, and V-vector of the new token, constitutes the second stage calculation process, which can be executed and obtained by the PIM module.

[0166] In this embodiment, although the division of the first and second stages in LLM is different, the computational efficiency of the first stage depends on the computing power of the NPU, while the computational efficiency of the second stage depends on the data transfer speed of the bus between the SoC and memory. In this embodiment, without changing the NPU hardware, a PIM module can be embedded in memory. The PIM can use the data in memory (such as the Q vector, K vector, and V vector corresponding to each token in the input) to directly perform the computation of the second stage, avoiding the time-consuming process caused by frequent data transfer between the SoC and memory when performing the computation of the second stage in the prior art. This embodiment can improve the computational efficiency of the second stage, thereby improving the computational efficiency of LLM.

[0167] It should be noted that the data involved in this application (including but not limited to data used for analysis, data stored, data displayed, etc.) are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0168] This application provides an electronic device, which includes a processor and a memory; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory, causing the electronic device to perform the above-described method.

[0169] This application provides an electronic device that may include a SoC and a PIM module. The SoC and PIM module can execute the corresponding steps in the above embodiments, so that the electronic device performs the above methods.

[0170] In some embodiments, the electronic device may further include external memory for storing model parameters of the first model. In some embodiments, the electronic device may further include internal memory.

[0171] This application provides an AI computing system, which may include a SoC and a PIM module. The SoC and PIM module can execute the corresponding steps in the above embodiments, so that the AI ​​computing system performs the above methods.

[0172] In some embodiments, the AI ​​computing system may further include external memory for storing model parameters of the first model. In some embodiments, the AI ​​computing system may further include internal memory.

[0173] The AI ​​calculation method of the present application embodiments has been described above. The apparatus for executing the above method provided in the present application embodiments is described below. Those skilled in the art will understand that the methods and apparatus can be combined and referenced with each other, and the related apparatus provided in the present application embodiments can execute the steps in the above AI calculation method.

[0174] This application provides a chip. The chip includes a processor, which is used to call a computer program in memory to execute the technical solutions in the above embodiments. Its implementation principle and technical effects are similar to those in the related embodiments described above, and will not be repeated here.

[0175] This application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, it implements the methods described above. The methods described in the above embodiments can be implemented wholly or partially by software, hardware, firmware, or any combination thereof. If implemented in software, the functionality can be stored as one or more instructions or code on or transmitted over the computer-readable medium. The computer-readable medium can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium accessible by a computer.

[0176] In one possible implementation, a computer-readable medium may include random access memory (RAM), read-only memory (ROM), compact discread-only memory (CD-ROM) or other optical disc storage, magnetic disk storage or other magnetic storage devices, or any other medium targeted to carry or to store required program code in the form of instructions or data structures, and accessible by a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. As used herein, disks and optical discs include optical discs, laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs optically reproduce data using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0177] This application provides a computer program product, which includes a computer program that, when run, causes a computer to perform the above-described method.

[0178] It should be noted that the modules or components described in the above embodiments can be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs), etc. Furthermore, when a module is implemented through processing element scheduler code, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processors capable of calling program code, such as a controller. Additionally, these modules can be integrated together to implement a system-on-a-chip (SOC).

[0179] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0180] The term "multiple" in this document refers to two or more. The term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Furthermore, the character " / " in this document generally indicates an "or" relationship between the preceding and following related objects; in formulas, " / " indicates a "division" relationship. Additionally, it should be understood that in the description of this application, words such as "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or order.

[0181] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

[0182] It is understood that, in the embodiments of this application, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

Claims

1. An artificial intelligence (AI) computing method, comprising: The method is applied to an electronic device, the electronic device comprising a system on chip (SoC) and a processing in memory (PIM) module, the SoC being connected to the PIM module, the PIM module comprising a first computing unit and a memory, the method comprising: The SoC receives an input of a user; The SoC performs a first stage of computation on the input using a first model to obtain a computation result, the computation result comprising: a first character in an output responsive to the input, and related data for computing other characters in the output; The SoC writes the computation result into the memory; The first computing unit performs a second stage of computation on the computation result using the first model to obtain the other characters in the output in sequence; The first computing unit sends the other characters to the SoC in sequence; The SoC outputs the characters in the output in sequence according to the order of the characters.

2. The method of claim 1, wherein, Before the SoC performs the first stage of computation on the input using the first model, the method further comprises: The SoC sets the first computing unit to a first mode, when the first computing unit is in the first mode, the first computing unit is not allowed to access the memory.

3. The method according to claim 1 or 2, characterized in that, Before the first computing unit performs the second stage of computation on the computation result using the first model, the method further comprises: The SoC sets the first computing unit to a second mode, when the first computing unit is in the second mode, the first computing unit is allowed to access the memory.

4. The method according to any one of claims 1 to 3, characterized in that, The electronic device further comprises an external memory for storing model parameters of the first model, after the SoC receives the input of the user, the method further comprises: The SoC sends a first command to the first computing unit; In response to the first command, the first computing unit loads the model parameters from the external memory; In a case where the loading of the model parameters is completed, the first computing unit sends feedback information to the SoC, the feedback information being used to indicate that the loading of the model parameters is completed.

5. The method of claim 4, wherein, After the first computing unit loads the model parameters from the external memory, the method further comprises: The first computing unit writes the model parameters into the memory; Before the SoC performs the first stage of computation on the input using the first model, the method further comprises: The SoC reads the model parameters from the memory, the model parameters comprising model parameters corresponding to the first stage; The SoC performs the first stage of computation on the input using the first model, comprising: The SoC performs the first stage of computation on the input using the model parameters corresponding to the first stage.

6. The method of claim 4, wherein, The model parameters comprise model parameters corresponding to the second stage, and the first computing unit performs the second stage of computation on the computation result using the first model, comprising: The first computing unit performs the second stage of computation on the computation result and the model parameters corresponding to the second stage.

7. The method according to any one of claims 4-6, characterized in that, Before the SoC sends the first command to the first computing unit, the method further comprises: The SoC starts the first computing unit.

8. The method according to any one of claims 1-7, characterized in that, The SoC comprises a second computing unit, the second computing unit is configured to: receive an input of a user; perform a first-stage computation on the input using a first model to obtain a computation result, the computation result comprising: a first character in an output responsive to the input, and related data for computing other characters in the output; write the computation result to the memory.

9. The method of claim 8, wherein, The second computing unit is a neural processing unit.

10. The method according to any one of claims 1-9, characterized in that, The first model is a large language model.

11. An artificial intelligence (AI) computing system, comprising: Comprise: a system on chip (SoC) and a processing-in-memory (PIM) module, the SoC being connected to the PIM module, the PIM module comprising a first computing unit and a memory; wherein the SoC is configured to perform the operations performed by the SoC in any one of claims 1-10, and the first computing unit is configured to perform the operations performed by the first computing unit in any one of claims 1-10. 12.The AI computing system of claim 11, wherein, The AI computing system further comprises an external memory configured to store model parameters. 13.The AI computing system of claim 12, wherein, The AI computing system further comprises an internal memory.

14. An electronic device, comprising: The electronic device comprises the AI computing system according to any one of claims 11-13. The AI computing system further comprises an external memory configured to store model parameters. The AI computing system further comprises an internal memory. The electronic device comprises the AI computing system according to any one of claims 11-13.

Images