Artificial intelligence model inference method, system, storage medium and program product

Abstract

An AI model inference method and system, a storage medium and a program product, relate to the technical field of artificial intelligence. A general processor obtains inference data, and determines, according to the inference data, information of an acceleration card storing an inference vector corresponding to the inference data and position information of the inference vector in the acceleration card. The general processor sends the position information to the acceleration card indicated by the information of the acceleration card. The acceleration card obtains the inference vector according to the position information, and performs inference operation by using an AI model according to the inference vector. Since the data amount of the position information is usually much smaller than that of the inference vector, the time consumption of transmitting the position information between the general processor and the acceleration card is much smaller than that of transmitting the inference vector, so that the overall time delay of the acceleration card in determining the inference vector corresponding to the inference data can be effectively reduced, the overall time delay of the AI model in outputting an inference result can be reduced, and the inference performance of the AI model can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an AI model reasoning method, system, storage medium, and program product. Background Technology

[0002] Some artificial intelligence (AI) models typically include embedding operations. Embedding involves mapping high-dimensional sparse features to low-dimensional dense vectors based on an embedding table. This allows each entity in the AI ​​model's inference data (such as words, usernames, product names, etc.) to be mapped to a corresponding low-dimensional dense vector (which can be a fixed-length vector). Furthermore, the semantic relationships and similarities between different entities are captured based on the similarity between their corresponding low-dimensional dense vectors.

[0003] Currently, AI models can typically run on accelerator cards, such as neural processing units (NPUs). During the execution of an AI model, a general-purpose processor can perform embedding operations on the inference data of the AI ​​model to determine the vector corresponding to the inference data, and send the vector to the accelerator card so that the accelerator card can continue to execute subsequent calculations in the AI ​​model based on the received vector (i.e., perform inference based on the vector).

[0004] However, this way of running AI models results in high latency and poor inference performance when the AI ​​model outputs inference results. Summary of the Invention

[0005] This application provides an AI model inference method to reduce the latency of the AI ​​model's output inference results and improve inference performance. Furthermore, this application also provides a corresponding data processing system, a computer-readable storage medium, and a computer program product.

[0006] Firstly, this application provides an AI model inference method. In this method, the inference vector is stored in an accelerator card. Specifically, a general-purpose processor (such as a CPU) acquires inference data and, based on the inference data, determines the information of the accelerator card (such as an NPU) storing the inference vector corresponding to the inference data, as well as the position information of the inference vector in the accelerator card (such as the offset of the inference vector stored in a file). Then, the general-purpose processor sends the position information to the accelerator card indicated by the accelerator card's information. Accordingly, the accelerator card acquires the inference vector based on the position information, such as querying the inference vector from multiple locally stored vectors, and performs inference operations using the AI ​​model based on the inference vector.

[0007] This application stores the inference vector in an accelerator card. During the process of determining the inference vector corresponding to the inference data, the general-purpose processor sends data to the accelerator card that includes the location information of the inference vector within the accelerator card. The accelerator card then obtains the inference vector based on this location information. Furthermore, since the amount of location information is typically much smaller than the amount of inference vector data, the time spent transmitting location information between the general-purpose processor and the accelerator card is significantly less than the time spent transmitting the inference vector. This reduces the communication latency between the general-purpose processor and the accelerator card, effectively lowering the overall latency of the accelerator card in determining the inference vector corresponding to the inference data. Consequently, it reduces the overall latency of the AI ​​model outputting the corresponding inference result based on the inference data and improves the performance of the inference service provided by the data processing system, which includes the general-purpose processor and the accelerator card.

[0008] In one possible implementation, the accelerator card includes a first vector file, and the location information includes file information of the first vector file and the position of the inference vector in the first vector file. Thus, the amount of location information data sent by the general-purpose processor to the accelerator card is much smaller than the amount of inference vector data, thereby reducing the overall latency of the accelerator card in determining the inference vector corresponding to the inference data, and consequently reducing the overall latency of the AI ​​model in outputting the corresponding inference result based on the inference data.

[0009] In one possible implementation, the method further includes: converting a dynamic hash table used to store the relationship between feature values ​​(such as feature IDs) and vectors into a static vector table, wherein the static vector table sequentially stores the vector corresponding to each feature value; generating a first vector file based on the static vector table, wherein the first vector file includes vectors corresponding to a portion of the feature values ​​in the static vector table; setting offset information for each feature value in the first vector file, wherein the offset information of each feature value indicates the position of the vector corresponding to each feature value in the first vector file; and storing the first vector file and the position information of each feature value in the first vector file into an accelerator card; wherein the process of converting the static vector table to storing the position information of each feature value into the accelerator card can be executed by a general-purpose processor or the accelerator card, or by other devices. Then, when the accelerator card obtains the inference vector based on the position information, it can specifically access the first vector file based on file information and obtain the inference vector based on the offset information in the position information. In this way, a portion of the vectors in the dynamic hash table can be moved to the accelerator card, thereby reducing the overall latency of the accelerator card in obtaining the inference vector.

[0010] In one possible implementation, the general-purpose processor determines the information of the accelerator card storing the inference vector corresponding to the inference data and the location information of the inference vector in the accelerator card based on the inference data. This includes: the general-purpose processor determining the feature value corresponding to the inference data based on the inference data; and the general-purpose processor can also determine the file information of the first vector file to which the inference vector corresponding to the feature value belongs, and the location information of the inference vector corresponding to the feature value in the first vector file, based on the feature value. In this way, the general-purpose processor can determine the first vector file to which the inference vector belongs and the specific location of the inference vector in the first vector file based on the feature value, so that the accelerator card can subsequently query the corresponding inference vector from the first vector file based on this information.

[0011] In one possible implementation, before the general-purpose processor acquires the inference data, a computational graph of the AI ​​model can be acquired. This computational graph includes dynamic files and dynamic lookup operators. The dynamic files are used to store vectors based on dynamic data structures, and the dynamic lookup operators are used to find vectors from the dynamic files. The dynamic files in the computational graph are replaced with a first vector file, and the dynamic lookup operators in the computational graph are replaced with static lookup operators, which are used to find vectors from the first vector file. The process of replacing operators and files in the computational graph can be performed by a general-purpose processor or accelerator card, or by other devices. Thus, by replacing files and operators in the computational graph, a correct computational graph can be obtained. By executing the correct computational graph, the inference vectors corresponding to the inference data can be retrieved, and the correct inference process can be executed.

[0012] In one possible implementation, before generating the AI ​​model based on the replaced computation graph, the initialization or recovery operators associated with dynamic files in the replaced computation graph are deleted. These operators can be deleted by a general-purpose processor or accelerator card, or by other devices. By deleting unnecessary operators, the structure of the AI ​​model can be simplified, reducing the memory and computational overhead required for the accelerator card to run the AI ​​model.

[0013] In one possible implementation, the AI ​​model includes a first vector file and a second vector file. The first vector file includes inference vectors, and the second vector file stores vectors based on a dynamic data structure. The method further includes: before the general-purpose processor acquires inference data, obtaining the memory resource amount of the accelerator card, and based on the accelerator card's memory resource amount, deploying the first vector file to the accelerator card and the second vector file to the general-purpose processor. Thus, based on the accelerator card's memory resource amount, it is determined that a portion of the vectors will be deployed to the accelerator card, while the remaining portion will be stored in the general-purpose processor. This avoids the vectors deployed to the accelerator card occupying too much of the accelerator card's memory resources, thereby affecting the accelerator card's operation of the AI ​​model.

[0014] In one possible implementation, the AI ​​model includes a first vector file and a second vector file. The first vector file includes inference vectors, and the second vector file stores vectors based on a dynamic data structure. The method further includes: outputting an interactive interface; and, in response to a user's configuration operation on the vector files via the interactive interface, deploying the first vector file to an accelerator card and the second vector file to a general-purpose processor. Thus, by utilizing the interactive interface, users can select which vectors are deployed to the accelerator card, thereby improving the flexibility of vector deployment.

[0015] In one possible implementation, the AI ​​model is used for information recommendation or natural language processing, or the AI ​​model is a GNN (Graph Neural Network) model.

[0016] Secondly, this application provides a data processing system, which includes a general-purpose processor and an artificial intelligence accelerator card; the general-purpose processor is used to acquire inference data; based on the inference data, determine the information of the accelerator card storing the inference vector corresponding to the inference data and the position information of the inference vector in the accelerator card; send the position information to the accelerator card of the information instruction; the accelerator card is used to acquire the inference vector based on the position information; and perform inference operations using an AI model based on the inference vector.

[0017] In one possible implementation, the accelerator card includes a first vector file, and the location information includes file information of the first vector file and the position of the inference vector in the first vector file.

[0018] In one possible implementation, the system further includes a preprocessing device for: converting a dynamic hash table used to store the relationship between feature values ​​and vectors of data into a static vector table, wherein the static vector table sequentially stores the vector corresponding to each feature value; generating a first vector file based on the static vector table, wherein the first vector file includes vectors corresponding to some feature values ​​in the static vector table; setting offset information for each feature value in the first vector file, wherein the offset information for each feature value indicates the position of the vector corresponding to each feature value in the first vector file; storing the first vector file and the position information of each feature value in the first vector file to an accelerator card; wherein, when the accelerator card obtains the inference vector based on the position information, it is specifically used for: accessing the first vector file based on file information and obtaining the inference vector based on the offset information in the position information.

[0019] The preprocessing device can be a general-purpose processor, an accelerator card, or other devices in the data processing system that are independent of the general-purpose processor and the accelerator card; there is no limitation on this.

[0020] In one possible implementation, when the general-purpose processor determines the information of the accelerator card storing the inference vector corresponding to the inference data and the position information of the inference vector in the accelerator card based on the inference data, it specifically performs the following: determining the feature value corresponding to the inference data based on the inference data; determining the file information of the first vector file to which the inference vector corresponding to the feature value belongs, and the position information of the inference vector corresponding to the feature value in the first vector file based on the feature value.

[0021] In one possible implementation, the system further includes a preprocessing device for: acquiring a computation graph of the AI ​​model before the general-purpose processor acquires inference data, the computation graph including dynamic files and dynamic lookup operators, the dynamic files being used to store vectors based on dynamic data structures, and the dynamic lookup operators being used to look up vectors from the dynamic files; replacing the dynamic files in the computation graph with a first vector file, and replacing the dynamic lookup operators in the computation graph with a static lookup operator, the static lookup operator being used to look up vectors from the first vector file.

[0022] In one possible implementation, the preprocessing device is further configured to: delete initialization operators or recovery operators associated with dynamic files in the replaced computation graph before generating an AI model based on the replaced computation graph.

[0023] In one possible implementation, the AI ​​model includes a first vector file and a second vector file, the first vector file including inference vectors and the second vector file storing vectors based on a dynamic data structure; the system also includes a preprocessing device for: obtaining the amount of memory resources of the accelerator card before the general-purpose processor obtains the inference data; and deploying the first vector file to the accelerator card and the second vector file to the general-purpose processor according to the amount of memory resources of the accelerator card.

[0024] In one possible implementation, the AI ​​model includes a first vector file and a second vector file, the first vector file including inference vectors and the second vector file storing vectors based on a dynamic data structure; the system also includes a preprocessing device for: outputting an interactive interface; and, in response to a user's configuration operation on the vector files on the interactive interface, deploying the first vector file to an accelerator card and the second vector file to a general-purpose processor.

[0025] In one possible implementation, the AI ​​model is used for information recommendation or natural language processing, or the AI ​​model is a graph neural network (GNN) model.

[0026] Thirdly, this application provides a computer-readable storage medium storing instructions that, when executed on a computing device, cause the computing device to perform the operation steps of the AI ​​model inference method described in the first aspect or any implementation thereof.

[0027] Fourthly, this application provides a computer program product containing instructions that, when run on a computing device, causes the computing device to perform the operational steps of the AI ​​model inference method described in the first aspect or any implementation thereof.

[0028] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0029] Figure 1 A schematic diagram of the structure of an exemplary data processing system provided in this application

[0030] Figure 2 A flowchart illustrating an AI model inference method provided in this application;

[0031] Figure 3a A schematic diagram of a dynamic file including vectors and eigenvalues;

[0032] Figure 3b A schematic diagram of the vector files and mapping relationships provided for this application;

[0033] Figure 4 This is a schematic diagram illustrating multiple processing layers in a recommendation scenario.

[0034] Figure 5 This is a schematic diagram illustrating multiple processing layers in a natural language processing scenario.

[0035] Figure 6 This is a schematic diagram of a graph neural network scenario that includes multiple processing layers. Detailed Implementation

[0036] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate; this is merely a method of distinction used in describing objects with the same attributes in the embodiments of this application.

[0037] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0038] See Figure 1 This is a schematic diagram of the structure of an exemplary data processing system 10. Figure 1 As shown, the data processing system 10 includes a general-purpose processor 100 and at least one accelerator card. Figure 1 The example includes two accelerator cards, namely accelerator card 200 and accelerator card 300. Furthermore, the general-purpose processor 100 can communicate with each accelerator card via a bus.

[0039] For example, the bus through which the general-purpose processor 100 communicates with each accelerator card may be, for example, a peripheral component interconnect express (PCIe) bus, a non-volatile memory express (NVMe) bus, a compute express link (CXL) bus, an HCCS (Huawei Collective Communication Server) bus, a unified bus (UB or Ubus), or other types of buses.

[0040] The general-purpose processor 100 may be, for example, a central processing unit (CPU), or any type of processor such as an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a system-on-chip (SoC), a software-defined infrastructure (SDI) chip, or any combination thereof.

[0041] Accelerator card 200 (and similarly accelerator card 300) can be any type of accelerator card capable of running AI models. For example, accelerator card 200 can be a neural network processing unit (NPU), graphics processing unit (GPU), tensor processing unit (TPU), deep learning processing unit (DPU), data processing unit (DPU), etc. In practical applications, accelerator card 200 can also be other types of accelerator cards capable of supporting AI model execution, such as artificial intelligence (AI) chips; there is no limitation in this regard.

[0042] Accelerator card 200 can be used to run AI models. Accelerator card 200 can run a complete AI model independently, for example, a graph neural network (GNN) model or a natural language processing (NLP) model. Alternatively, accelerator card 200 can work in conjunction with accelerator card 300 to run an AI model, for example, a large language model (LLM), a large language model meta AI (LLaMA) model, a bidirectional encoder representations from transformers (BERT) model, or a generative pre-trained transformer 3 (GPT-3) model, or other types of models such as GPT-4, etc. For ease of understanding, the following explanation uses accelerator card 200 running an AI model independently as an example.

[0043] During the process of running an AI model using accelerator cards 200 (and 300), the AI ​​model typically performs inference based on the vectors corresponding to the inference data and outputs the corresponding inference results. This inference data can be, for example, the input data of the AI ​​model. In practical applications, the vectors corresponding to the inference data can be determined by searching a vector file. If the general-purpose processor 100 searches for the vector corresponding to the inference data (hereinafter referred to as the inference vector for ease of distinction and description) from multiple vectors based on the inference data and sends the found inference vector to the accelerator card 200 to execute the model inference process, the communication capability between the general-purpose processor 100 and the accelerator card 200 can easily become a bottleneck restricting the performance of the AI ​​model inference. Specifically, the amount of data in the inference vector corresponding to the inference data is usually large, which results in a long data transmission time during the process of the general-purpose processor 100 sending the queried inference vector to the accelerator card 200, leading to a longer overall inference time for the AI ​​model and reducing its inference performance. Moreover, during the transmission of the inference vector, the accelerator card 200 is usually idle for a long time, which reduces its resource utilization.

[0044] Based on this, Figure 1 In the data processing system 10 shown, the inference performance of the AI ​​model is improved by reducing the data transmission time between the general-purpose processor 100 and each accelerator card.

[0045] In a specific implementation, the accelerator card 200 can store vectors corresponding to the data, and different vectors corresponding to different data can be stored in different locations within the accelerator card. For example, they can be stored in different locations within a vector file stored on the accelerator card, such as a vector table. After obtaining the inference data of the AI ​​model, the general-purpose processor 100 determines the information of the accelerator card storing the inference vectors corresponding to the inference data, as well as the location information of the inference vectors within the accelerator card. Then, the general-purpose processor 100 sends the determined location information to the accelerator card indicated by the information of that accelerator card, assuming it is accelerator card 200. Accelerator card 200 then obtains the inference vectors based on the location information and performs inference operations using the AI ​​model running on it, obtaining the corresponding inference results.

[0046] In determining the inference vector corresponding to the inference data, the data sent from the general-purpose processor 100 to the accelerator card 200 can be the position information of the inference vector within the accelerator card 200, which then obtains the inference vector based on this position information. Since the amount of position information is typically much smaller than the amount of inference vector data, the time spent transmitting position information between the general-purpose processor 100 and the accelerator card 200 is much less than the time spent transmitting the inference vector. This reduces the communication latency between the general-purpose processor 100 and the accelerator card 200, effectively lowering the overall latency of the accelerator card 200 in determining the inference vector corresponding to the inference data. Consequently, it reduces the overall latency of the AI ​​model outputting the corresponding inference result based on the inference data, improves the overall inference performance of the AI ​​model, and ultimately enhances the performance of the inference service provided by the data processing system 10. Simultaneously, the inference accuracy of the AI ​​model does not decrease.

[0047] Furthermore, with lower data transmission latency, the idle time of the accelerator card 200 can be reduced, thereby improving its resource utilization. Simultaneously, by utilizing the memory in the accelerator card 200 to store various data vectors (including inference vectors), the memory space occupied by the general-purpose processor 100 is reduced. This allows the general-purpose processor 100 to process more services based on more memory resources, thereby improving its performance.

[0048] It is worth noting that the above Figure 1 The data processing system 10 shown is merely illustrative and not intended to be limiting. For example, in other possible implementations, the data processing system may include a greater number of general-purpose processors and accelerator cards, and the types of the multiple general-purpose processors and accelerator cards included in the data processing system may be the same or different. Furthermore, in other possible implementations, the data processing system may also include more types of devices, such as switches, to facilitate communication between different accelerator cards; or, the data processing system may also include preprocessing equipment for preprocessing the AI ​​model before it is run.

[0049] For ease of understanding, embodiments of the AI ​​model inference method provided in this application are described below with reference to the accompanying drawings.

[0050] See Figure 2 , Figure 2 This application provides a flowchart illustrating an AI model inference method, which can be applied to... Figure 2The data processing system 10 described above can also be applied to other applicable data processing systems. For ease of explanation, this embodiment uses an application... Figure 2 The data processing system 10 shown is illustrated using the example of running an AI model with an accelerator card 200.

[0051] in, Figure 2 The AI ​​model inference method shown may specifically include the following steps.

[0052] S201: General-purpose processor 100 acquires inference data.

[0053] In this embodiment, the general-purpose processor 100 and the accelerator card 200 can work together to provide inference services. The AI ​​model can be deployed on the accelerator card 200, and the general-purpose processor 100 can provide the input information required for the AI ​​model's inference to the accelerator card 200, so that the accelerator card 200 can run the AI ​​model and provide the corresponding inference services.

[0054] In one possible implementation, the data processing system 10 can provide a client, which may be, for example, an application running on a user's device or a web browser. The data processing system 10 can output an interactive interface to the client, which then presents this interface to the user. In this way, the user can provide data for inference on the interactive interface; hereinafter referred to as inference data. For example, the inference data can be text-type data, such as user-inputted words, phrases, sentences, or paragraphs; or, the inference data can be other types of data such as images, audio, or video. The client can then send the inference data to the general-purpose processor 100, enabling the general-purpose processor 100 to obtain the inference data for the AI ​​model.

[0055] In real-world applications, the general-purpose processor 100 can also acquire inference data through other means, and there are no restrictions on this.

[0056] S202: The general-purpose processor 100 determines, based on the inference data, the information of the accelerator card 200 storing the inference vector corresponding to the inference data and the position information of the inference vector in the accelerator card 200.

[0057] Typically, AI models perform calculations based on the vectors corresponding to the inference data to obtain the inference results. For ease of distinction, the vectors corresponding to the inference data will be referred to as inference vectors below. In this embodiment, after acquiring the inference data, the general-purpose processor 100 can preprocess the inference data to determine the relevant information of the inference vectors corresponding to the inference data. This relevant information is the information of the accelerator card 200 where the inference vector is located and the position information of the inference vector in the accelerator card 200.

[0058] In one possible implementation, the AI ​​model may include at least one vector file, and this at least one vector file includes a first vector file, which may include multiple vectors, each vector corresponding to a piece of data. Furthermore, the first vector file may be pre-deployed in the accelerator card 200 before the accelerator card 200 runs the AI ​​model. Since the data processing system 10 includes multiple accelerator cards, after acquiring inference data, the general-purpose processor 100 can first determine that the inference vector corresponding to the inference data is located in the accelerator card 200 (specifically, it can determine that the inference vector is located in the first vector file in the accelerator card 200), and then determine the position of the inference vector in the first vector file in the accelerator card 200, so that the accelerator card 200 can subsequently determine the inference vector corresponding to the inference data from the first vector file based on this position.

[0059] In practical implementation, the general-purpose processor 100 can first extract features from the inference data to obtain feature values. For example, the general-purpose processor 100 can perform a hash operation on the inference data and use the calculated hash value as the feature value of the inference data. Then, the general-purpose processor 100 can determine one or more accelerator cards where the inference vector corresponding to the feature value is located based on the feature value. For example, in practical applications, the data can be partitioned in advance according to the range of feature values, and the vectors corresponding to feature values ​​in different partitions can be deployed on different accelerator cards. For example, the vector corresponding to feature values ​​in the range of 0 to 5000 is located on accelerator card 200, and the vector corresponding to feature values ​​in the range of 5001 to 10000 is located on accelerator card 300, etc. In this way, the general-purpose processor 100 can determine the accelerator card where the inference vector corresponding to the inference data is located based on the range of feature values ​​of the inference data. Furthermore, when multiple vectors deployed on various accelerator cards are saved as vector files, since the vector files on each accelerator card are fixed, the general-purpose processor 100 can directly determine the file information of the first vector file to which the inference vector belongs based on the feature values ​​of the inference data. This file information may, for example, be the file name of the first vector file. Optionally, this file information may further include the identifier of the accelerator card where the first vector file is located (such as the name of the accelerator card).

[0060] Furthermore, the general-purpose processor 100 can look up the mapping relationship based on the feature values ​​corresponding to the inference data. This mapping relationship can be pre-configured in the general-purpose processor 100, and it is a correspondence between feature values ​​and offset information. The offset information for each feature value indicates the position of the vector corresponding to that feature value in the vector file deployed in the accelerator card. That is, this mapping relationship is used to record the position of the feature value of each data point and the corresponding vector in the vector file. After determining that the inference vector corresponding to the feature value is located in the first vector file, the general-purpose processor 100 can also determine the offset information corresponding to the feature value of the inference data by looking up the mapping relationship. This offset information indicates the position of the inference vector corresponding to the inference data in the first vector file. The offset information and the file information of the first vector file constitute the position information of the inference vector in the accelerator card 200. For example, the offset information can be, for instance, the number of the inference vector in the first vector file, such as the inference vector being 50, indicating the 50th vector in the first vector file. Alternatively, the offset information can be the storage location of the inference vector recorded in the first vector file in the memory of the accelerator card 200, such as the offset. In this embodiment, the specific implementation method of the offset information is not limited.

[0061] In practical applications, the general-purpose processor 100 can also determine the position information of the inference vector in the first vector file in other ways, without limitation.

[0062] When the inference data is text-based, if it includes a single character or word, the corresponding inference vector can be a vector. In this case, the offset information determined by the general processor 100 for the inference data indicates the position of a vector in the first vector file. If the inference data includes multiple words, such as a sentence or paragraph, the general processor 100 can first perform word segmentation on the inference data to obtain multiple different words (a single character can also be considered a word). Then, the general processor 100 can determine the position of the vector corresponding to each word in the first vector file. The offset information determined by the general processor 100 then indicates the positions of multiple vectors. In practical applications, this offset information can specifically indicate the positions of multiple vectors in the first vector file, or it can indicate the positions of the multiple vectors in multiple vector files respectively.

[0063] The first vector file in the AI ​​model can be generated during the model training phase. For example, taking the generation of the first vector file by the general-purpose processor 100 as an example, during the training phase of the AI ​​model, the inference data in each set of training samples can be encoded into vectors. Then, during multiple rounds of iterative training of the AI ​​model, since the training samples used in each round may be different, the general-purpose processor 100 can record the vectors corresponding to newly appearing inference data in each round of training samples and save these vectors in the corresponding files. Thus, after the AI ​​model completes training, when it begins to provide inference services, for the same inference data as in the training samples, the vectors corresponding to that inference data can be retrieved from this file without needing to generate them temporarily through encoding or other methods.

[0064] For example, the general-purpose processor 100 can record the vectors corresponding to each data point in the training samples based on a dynamic data structure and generate a dynamic file, such as a mutable hash table. Thus, during the training phase of the AI ​​model, as the number of training samples increases, the number of vectors corresponding to the data also increases. The general-purpose processor 100 can then dynamically allocate memory space to store the vectors corresponding to each data point, supporting the dynamic insertion of vector data. This avoids wasting significant memory resources during the AI ​​model training phase.

[0065] After the AI ​​model training is complete, the general-purpose processor 100 can convert the dynamic file using dynamic data structures into a static vector table, which sequentially stores the vector corresponding to each feature value. For example, ... Figure 3aAs shown, the dynamic hash table generated by the general-purpose processor 100 records not only vectors but also the corresponding feature values ​​of the vectors. That is, the dynamic hash table can be used to record the relationship between vectors and feature values. The general-purpose processor 100 can then convert the dynamic hash table into a static vector table, which sequentially stores the vector corresponding to each feature value in the dynamic hash table. For example, the general-purpose processor 100 can construct a static vector table (such as a static vector table), derive all vectors from the dynamic hash table, and then save the derived vectors to the static vector table. For instance, multiple vectors can be sequentially stored in the static vector table based on a dense linear permutation. Next, the general-purpose processor 100 can generate a first vector file based on the static vector table. This first vector file includes vectors corresponding to some or all feature values ​​in the static vector table. For example, it can export and save a portion of the vectors from the static vector table as the first vector file. In practical applications, when multiple vectors in the dynamic hash table are distributed across multiple accelerator cards, the general-purpose processor 100 can also generate multiple vector files (including the first vector file) based on the static vector table, allowing the general-purpose processor 100 to deploy different vector files to different accelerator cards. Then, for the first vector file in the accelerator card 200, the general-purpose processor 100 can set offset information for the feature values ​​corresponding to each vector in the first vector file. The offset information for each feature value indicates the position of the vector corresponding to each feature value in the first vector file. Next, the general-purpose processor 100 can send the generated first vector file and the position information of each feature value in the first vector file (i.e., the position information of each vector in the first vector file) to the accelerator card 200 for storage, saving the mapping relationship locally on the general-purpose processor 100. Finally, the general-purpose processor 100 can also create the above mapping relationship, which is used to map the feature values ​​corresponding to each vector to the position of that vector in the static first vector file, such as... Figure 3b As shown, this allows the general-purpose processor 100 to determine the position of each vector in the first vector file based on the mapping relationship.

[0066] Furthermore, since the inference data encountered during the training phase of the AI ​​model cannot cover all the inference data encountered during the inference phase, the general-purpose processor 100 can add a default vector to the first vector file during the generation process. This default vector can be added, for example, to the beginning or end position of the first vector file, or to other positions within the first vector file. Moreover, when the AI ​​model encounters inference data not included in the training samples during the inference phase, the corresponding inference vector can be the default vector from the first vector file. For example, the default vector can be, for instance, a vector with all dimensions set to zero, or other pre-defined vectors; there are no limitations on this.

[0067] S203: The general-purpose processor 100 sends location information to the accelerator card 200 as indicated by the information.

[0068] In a specific implementation, after determining that the inference vector is located on the accelerator card 200, the general-purpose processor 100 can send the location information to the accelerator card 200 based on the bus connected to the accelerator card 200.

[0069] In practical applications, the inference vectors corresponding to the inference data typically include a large number of dimensional information, such as 64- or 128-dimensional vectors. Therefore, the amount of location information data sent from the general-purpose processor 100 to the accelerator card 200 is much smaller than the amount of data in the inference vector itself. Consequently, the time taken for the location information to be transmitted between the general-purpose processor 100 and the accelerator card 200 is usually much shorter than the time taken for the inference vector to be transmitted between the two devices.

[0070] S204: Accelerator Card 200 obtains the inference vector corresponding to the inference data based on the location information.

[0071] In one possible implementation, after acquiring the location information, the accelerator card 200 can determine that the inference vector is located in the first vector file based on the file information of the first vector file included in the location information, and access the first vector file in the accelerator card 200 according to the file information. Then, the accelerator card 200 queries the vector indicated by the offset information from multiple vectors included in the first vector file according to the offset information in the location information, which is the inference vector corresponding to the inference data.

[0072] When the offset information includes multiple offsets, the offset information can be used to indicate the positions of multiple vectors in the first vector file. In this case, the accelerator card 200 can use its parallel computing power to search for the corresponding multiple vectors in parallel from the first vector file based on the multiple positions, thereby improving the overall efficiency of the accelerator card 200 in searching for the inference vectors corresponding to the inference data.

[0073] As an implementation example, the accelerator card 200 can execute a static lookup operator to find multiple vectors in parallel from a first vector file; these multiple vectors are the inference vectors corresponding to the inference data. For example, when the data processing system 10 supports the TensorFlow framework, the accelerator card 200 can utilize its vector computing power to perform the `tf.gather` operation, which is used to select specific sub-tensors from a multidimensional tensor. Thus, by performing the `tf.gather` operation at multiple locations, the accelerator card 200 can select multiple vectors (i.e., sub-tensors) indicated by those multiple locations from the first vector file (i.e., the multidimensional tensor).

[0074] S205: Accelerator Card 200 uses an AI model to perform inference operations based on the inference vector to obtain the inference result.

[0075] After determining the inference vector, the accelerator card 200 can run the AI ​​model to perform subsequent calculations based on the inference vector. Specifically, it can execute each operator in the computation graph corresponding to the AI ​​model in sequence to realize the inference process of the AI ​​model and obtain the final inference result.

[0076] As an implementation example, the computation graph of the AI ​​model executed by the accelerator card 200 includes static lookup operators and a first vector file. Furthermore, the computation graph of the AI ​​model can be obtained by modifying the operators in the computation graph of the initial AI model, which could be, for example, the computation graph generated when the AI ​​model completes training. For ease of understanding, the following explanation uses the example of the general-purpose processor 100 modifying the computation graph corresponding to the initial AI model.

[0077] In specific implementation, the general-purpose processor 100 can obtain an initial AI model, which may be an AI model trained using training samples. In practical applications, the initial AI model can be saved in a first format, such as the SavedModel format. This initial AI model includes a computation graph containing dynamic files and dynamic lookup operators. The dynamic files allow the initial AI model to dynamically insert vectors corresponding to new data during training. Then, the general-purpose processor 100 can replace the dynamic files in the computation graph with the aforementioned first vector file. The method by which the general-purpose processor 100 generates the first vector file is described in the relevant sections above and will not be repeated here. Furthermore, the general-purpose processor 100 can also use the static lookup operators supported by the accelerator card 200 to replace the dynamic lookup operators in the computation graph. For example, it can replace the LookupTableFindV2 operator (dynamic lookup operator) in the computation graph with the tf.gather operator (static lookup operator), thereby generating a new computation graph (i.e., the replaced computation graph). In practical applications, the general-purpose processor 100 can further save the initial AI model in a second format, such as the new.SavedModel format, based on this new computation graph to obtain a new AI model.

[0078] Furthermore, the computation graph of the initial AI model may also include initialization operators or restore operators associated with the dynamic file. The initialization operator is used to initialize the dynamic file, such as defining the memory space corresponding to the dynamic file during the initialization phase and an expansion threshold. The memory space indicates the size of the memory space allocated to the dynamic file during creation, and the expansion threshold indicates that new memory space will be allocated to the dynamic file when the number of vectors stored in the dynamic file (or the size of the memory space used) exceeds the threshold value. The restore operator is used to restore the dynamic file to a normal state when data loss occurs in the vectors recorded in the dynamic file. Since replacing the dynamic file in the computation graph with a static first vector file can lead to the inability to find the dynamic file when executing the initialization and restore operators, execution errors may occur, such as the computation graph terminating due to the inability to find the dynamic file. Therefore, the general-purpose processor 100 can also delete the initialization or restore operators associated with the dynamic file in the computation graph (or both), and generate a new AI model based on the computation graph with the initialization or restore operators deleted. In this way, by removing operators that do not need to be executed, the structure of the AI ​​model can be simplified, reducing the memory and computational overhead required when the accelerator card 200 runs the AI ​​model.

[0079] After generating a new AI model, the accelerator card 200 can load and run the AI ​​model to provide corresponding inference services. For example, after generating an inference result based on the vectors corresponding to the inference data, the accelerator card 200 can output the inference result, such as outputting the inference result to the general-purpose processor 100, which then presents the inference result to the user through a client, thereby providing the user with corresponding inference services.

[0080] Thus, during the inference service provided by the data processing system 10, by sinking the first vector file to the accelerator card 200, the amount of data sent from the general-purpose processor 100 to the accelerator card 200 can be effectively reduced, and the data transmission latency can be reduced. This can effectively improve the efficiency of determining the vector corresponding to the inference data and improve the overall performance of the inference service provided by the data processing system 10.

[0081] Furthermore, the first vector file can be a static file containing a fixed number of vectors. This allows the memory space of the first vector file to be allocated according to the number of vectors that the first vector file needs to record when it is created, thereby avoiding excessive memory resources allocated to the first vector file and resulting in memory fragmentation.

[0082] Furthermore, by finding the vectors corresponding to the inference data through static vector files, the file dependency on dynamic data structures can be eliminated. This can reduce the complexity of AI model deployment, simplify the structure of AI models, and thus reduce the workload of serialization and deserialization during model deployment.

[0083] It is understandable that in practical applications, an AI model may include multiple vector files, each of which can store the vectors corresponding to the data. These multiple vector files include the first vector file mentioned above. For example, the inference data in the training samples can be divided into multiple segments, and a vector file can be generated when training the AI ​​model using the training samples of each segment.

[0084] In the first implementation example, multiple vector files can be deployed on the accelerator card 200. The general-purpose processor 100 can be configured with mapping relationships corresponding to these multiple vector files. Each mapping relationship is used to map the feature value of the data to the position of the corresponding vector in the vector file. Then, the general-purpose processor 100 and the accelerator card 200 can, referring to the method described above for finding the first vector file, find the vector corresponding to the inference data from the second vector file, the third vector file, and other vector files among the multiple vector files.

[0085] Accordingly, when modifying the dynamic files and dynamic lookup operators in the computation graph of the initial AI model, the general-purpose processor 100 can traverse all files and all operation nodes in the computation graph to determine all dynamic files and all dynamic lookup operators, and use the corresponding multiple vector files to replace the multiple dynamic files in the computation graph, and use the corresponding multiple static lookup operators to replace the multiple dynamic lookup operators in the computation graph.

[0086] In the second implementation example, due to the limited memory space of the accelerator card 200, it may be difficult to deploy all vector files on the accelerator card 200. In this case, some vector files from multiple vector files can be deployed on the accelerator card 200, and the remaining vector files can be deployed on the general-purpose processor 100. Taking the deployment of the first vector file from multiple vector files on the accelerator card 200 and the deployment of the second vector file from multiple vector files on the general-purpose processor 100 as an example, when the vector corresponding to the inference data is stored in the first vector file, the general-purpose processor 100 and the accelerator card 200 can refer to the above method to find the vector corresponding to the inference data from the first vector file. When the vector corresponding to the inference data is stored in the second vector file, the general-purpose processor 100 can directly query the locally stored second vector file according to the feature value of the inference data, find the vector corresponding to the inference data from the second vector file, and then send the found vector to the accelerator card 200 so that the accelerator card 200 can perform subsequent calculations based on the vector.

[0087] Taking the deployment of vector files on the general-purpose processor 100 as an example, the general-purpose processor 100 can pre-obtain the memory resource amount of the accelerator card 200, and based on this memory resource amount, determine which portion of the vector files in the AI ​​model will be deployed to the accelerator card 200 (such as the first vector file), and the remaining portion of the vector files will be deployed to the general-purpose processor 100 (such as the second vector file). This avoids deploying too many vector files to the accelerator card 200, which could lead to insufficient memory resources when the accelerator card 200 runs the AI ​​model.

[0088] In this embodiment, the general-purpose processor 100 can, based on the memory resources of the accelerator card 200, deploy vector files with higher execution order to the accelerator card 200 and the remaining vector files with lower execution order to the general-purpose processor 100 according to their execution order in the computation graph. Alternatively, the general-purpose processor 100 can, based on the memory resources of the accelerator card 200, select vector files with higher importance for deployment to the accelerator card 200 and deploy vector files with lower importance to the general-purpose processor 100; the importance of the vector files can be measured by the number of times the data is repeated in the training samples. Alternatively, the general-purpose processor 100 can, based on the memory resources of the accelerator card 200, randomly select some vector files for deployment to the accelerator card 200 using a random algorithm and deploy the remaining vector files to the general-purpose processor 100. In this embodiment, the specific implementation method for allocating vector files between the general-purpose processor 100 and the accelerator card 200 is not limited.

[0089] Alternatively, the general-purpose processor 100 can output an interactive interface, allowing the user to configure which vectors from the static vector table will be deployed to the accelerator card 200 and which vectors will be stored on the general-purpose processor 100. In this case, the general-purpose processor 100 can respond to the user's configuration operations on the vector files through the interactive interface, generating a first vector file based on the user-selected vectors and deploying it to the accelerator card 200. It can also generate a second vector file based on the unselected vectors and deploy it to the general-purpose processor 100. In practical applications, other methods can be used to deploy multiple vector files from the AI ​​model to different accelerator cards or the general-purpose processor 100; this is not limited.

[0090] It is worth noting that the above Figure 2 The embodiments shown are merely illustrative examples and are not intended to be limiting. Other embodiments similar to this one are described below.

[0091] 1. Figure 2 In this embodiment, the general-purpose processor 100 is used as an example to illustrate the operations of generating the first vector file and modifying the computation graph. In other embodiments, the accelerator card in the data processing system 10 may also perform the operations of generating the first vector file and modifying the computation graph, or other devices deployed separately in the data processing system 10 may perform these operations, such as the preprocessing device mentioned above. There is no limitation on this.

[0092] 2. Figure 2 The illustrated embodiment uses accelerator card 200 running a single AI model as an example. In other embodiments, the data processing system 10 can also utilize multiple accelerator cards to collaboratively run the same AI model. The AI ​​model can include multiple vector files, and these multiple vector files can be deployed to accelerator card 200 and accelerator card 300 respectively, such as deploying the first vector file to accelerator card 200, the second vector file to accelerator card 300, etc. In this case, each accelerator card can refer to the above method to find the vector file according to the received location to determine the vector corresponding to the inference data. Furthermore, some vector files in the AI ​​model can also be stored in a general-purpose processor 100, etc.

[0093] When the inference data corresponds to multiple inference vectors, and these vectors are located in vector files on different accelerator cards, the general-purpose processor 100 can refer to the above. Figure 2The process of the illustrated embodiment involves sending different location information to different accelerator cards, so that each accelerator card can query a portion of the inference vector from its own stored vector file. Then, the different accelerator cards can exchange the vectors they have queried, so that some or all of the accelerator cards can obtain the complete inference vector corresponding to the inference data.

[0094] For example, suppose the inference data includes data a and data b, and the vector corresponding to data a is located in the vector file of accelerator card 200, and the vector corresponding to data b is located in the vector file of accelerator card 300. Then, the general-purpose processor 100 can refer to the above process to send the location information corresponding to data a to accelerator card 200 and the location information corresponding to data b to accelerator card 300. Accelerator card 200 can query the vector corresponding to data a based on the location information corresponding to data a, and accelerator card 300 can query the vector corresponding to data b based on the location information corresponding to data b. Thus, accelerator card 200 and accelerator card 300 can interact with the queried vectors so that accelerator card 200 and / or accelerator card 300 can obtain the complete inference vector corresponding to the inference data (including the vector corresponding to data a and the vector corresponding to data b).

[0095] To facilitate a further understanding of the technical solution of this application, the process of the data processing system 10 providing inference services is described below in conjunction with the accompanying drawings and the following three specific application scenarios.

[0096] 1. Information Recommendation Scenario. In this application scenario, the AI ​​model running on the accelerator card 200, specifically a recommendation model, is used to recommend item information to the user.

[0097] like Figure 4 As shown, the information recommendation service (i.e., inference service) provided by the data processing system 10 includes a user interaction layer, a feature extraction layer, a static embedding lookup layer, a model inference layer, and a recommendation result generation layer. The user interaction layer interacts with the user to obtain inference data provided by the user. This inference data can be, for example, relationship data between the user and items (such as goods), such as the user's browsing history, purchase history, or review history for goods. The feature extraction layer extracts feature values ​​from the inference data and provides these feature values ​​to the static embedding lookup layer. In the static embedding lookup layer, the general-purpose processor 100 can determine the position information of the inference vector corresponding to the inference data in the vector file based on the feature value, and send this position information to the accelerator card 200. The accelerator card 200 can then use this position information to search the vector file to determine the inference vector corresponding to the inference data. The specific implementation process is described above. Figure 2The relevant parts of the illustrated embodiment are described below. The model inference layer utilizes the recommendation model to perform the corresponding inference process based on the retrieved inference vectors, and provides the inference results to the recommendation result generation layer. These inference results can be, for example, identifiers indicating items to be recommended to the user. The recommendation result generation layer obtains relevant information about multiple items based on the inference results, such as actual product images and descriptive text, and presents this information to the user to recommend items the user might like.

[0098] Thus, by accelerating the determination of the inference vector corresponding to the inference data in the static embedding lookup layer, the overall efficiency and real-time performance of the data processing system 10 in recommending information to users can be improved.

[0099] 2. Natural Language Processing Scenarios. In this application scenario, the AI ​​model running on the accelerator card 200 can specifically be a natural language processing (NLP) model, which can be used to classify or machine translate the input text.

[0100] like Figure 5 As shown, the natural language processing service (i.e., inference service) provided by the data processing system 10 may include an input text layer, a word segmentation and parsing layer, a static embedding lookup layer, an NLP model inference layer, and a result processing layer. The input text layer is used to acquire inference data for the AI ​​model. This inference data is text-based, such as a sentence input by the user. The word segmentation and parsing layer is used to segment the input text into multiple words, and then parse these words to obtain the feature values ​​corresponding to each segment. In the static embedding lookup layer, the general-purpose processor 100 can determine the position of the vector corresponding to each segment in the vector file based on the feature values ​​of each segment, and send the positions of the multiple segments to the accelerator card 200. The accelerator card 200 can then search the vector file based on the positions of the multiple segments to determine the vector corresponding to each segment, thereby obtaining the inference vector corresponding to the inference data. The specific implementation process is described above. Figure 2 The relevant parts of the illustrated embodiment are described below. The NLP model inference layer utilizes the NLP model to perform the corresponding inference process based on the found inference vectors, and provides the inference result to the result processing layer. This inference result can be, for example, a text classification result of the inference data or a translation result for the inference data. The result processing layer generates corresponding natural language processing results based on the inference result, such as generating a corresponding response text based on the text category to which the inference data belongs, or generating a translated text corresponding to the inference data.

[0101] Thus, by accelerating the determination of the inference vector corresponding to the inference data in the static embedding lookup layer, the performance of the data processing system 10 in providing users with natural language processing services such as text allocation and machine translation can be improved, and its performance is particularly outstanding in real-time tasks such as machine translation and intelligent question answering.

[0102] 3. Graph Neural Network Scenarios. In this application scenario, the AI ​​model running on the accelerator card 200 can specifically be a graph neural network (GNN) model, which can be used to construct knowledge graphs or analyze social networks based on input graph data.

[0103] like Figure 6 As shown, the graph neural network service (i.e., inference service) provided by the data processing system 10 may include a graph input layer, a static embedding lookup layer, a GNN model inference layer, a graph aggregation / pooling layer, and a result processing layer. The graph input layer is used to acquire the inference data of the AI ​​model, which is graph data including nodes and edges. In the static embedding lookup layer, the general-purpose processor 100 can use the identifiers of each node and edge in the graph data as feature values, and determine the position of the vector corresponding to each node / edge in the vector file based on the feature values ​​of each node / edge. It then sends the positions of each node / edge to the accelerator card 200, which can then search the vector file based on the positions of each node / edge to determine the vector corresponding to each node / edge, thereby obtaining the inference vector corresponding to the inference data. The specific implementation process is described above. Figure 2 The relevant parts of the illustrated embodiment are described below. The GNN model inference layer utilizes the GNN model to perform the corresponding inference process based on the found inference vectors, and provides the inference results to the graph aggregation / pooling layer. These inference results can be, for example, the relationships between different nodes (such as social relationships). The graph aggregation / pooling layer aggregates or reduces the dimensionality of the node information in the inference results. For example, it can aggregate information from adjacent nodes in the inference results using methods such as max pooling to obtain an aggregated representation of each node, thereby reducing redundant information in the node information. The result processing layer processes the aggregated or dimensionality-reduced node information, such as adding category labels to nodes and predicting whether edges exist between different nodes.

[0104] Thus, by accelerating the determination of the inference vector corresponding to the inference data in the static embedding lookup layer, the performance of the data processing system 10 in providing users with services for building knowledge graphs or analyzing social networks can be improved, especially in the process of processing hundreds of millions of nodes in large-scale social networks or knowledge graphs, which can produce a significant acceleration effect.

[0105] It is worth noting that the above Figures 4 to 6The application scenarios shown are merely illustrative examples and are not intended to be limiting. In practical applications, the above... Figure 2 The illustrated embodiments can also be applied to other applicable application scenarios.

[0106] It is worth noting that other reasonable combinations of steps that can be conceived by those skilled in the art based on the above description also fall within the scope of protection of this application. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to this application.

[0107] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the aforementioned AI model inference method.

[0108] This application also provides a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in this application are generated.

[0109] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another website, computer, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0110] The computer program product can be a software installation package. When any of the aforementioned AI model inference methods is required, the computer program product can be downloaded and executed on a computing device.

[0111] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0112] The terminology used in the above embodiments is for the purpose of describing specific embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the embodiments of this application, “one or more” refers to one, two, or more; the character “ / ” generally indicates that the preceding and following objects are in an “or” relationship. In the embodiments of this application, “simultaneously” means within the same time period, including situations where they are at the same moment.

[0113] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0114] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An artificial intelligence (AI) model reasoning method, characterized in that, The method includes: General-purpose processors acquire inference data; The general-purpose processor determines, based on the inference data, the information of the accelerator card storing the inference vector corresponding to the inference data and the position information of the inference vector in the accelerator card; The general-purpose processor sends the location information to the accelerator card indicated by the information of the accelerator card; The acceleration card obtains the inference vector based on the location information; The accelerator card performs inference operations using the AI ​​model based on the inference vector.

2. The method according to claim 1, characterized in that, The accelerator card includes a first vector file, and the location information includes the file information of the first vector file and the position of the inference vector in the first vector file.

3. The method according to claim 2, characterized in that, The method further includes: The dynamic hash table used to store the relationship between feature values ​​and vectors is converted into a static vector table, which sequentially stores the vector corresponding to each feature value. The first vector file is generated based on the static vector table, and the first vector file includes vectors corresponding to some feature values ​​in the static vector table. Offset information is set for each feature value in the first vector file, wherein the offset information for each feature value indicates the position of the vector corresponding to each feature value in the first vector file; Store the first vector file and the location information of each feature value in the first vector file into the accelerator card; The accelerator card obtains the inference vector based on the location information, including: The accelerator card accesses the first vector file based on the file information and obtains the inference vector based on the offset information in the location information.

4. The method according to any one of claims 1 to 3, characterized in that, The general-purpose processor determines, based on the inference data, the information of the accelerator card storing the inference vector corresponding to the inference data and the position information of the inference vector in the accelerator card, including: The general-purpose processor determines the feature value corresponding to the inference data based on the inference data; The general-purpose processor determines the file information of the first vector file to which the inference vector corresponding to the feature value belongs, and the position information of the inference vector corresponding to the feature value in the first vector file, based on the feature value.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Before the general-purpose processor acquires the inference data, the computation graph of the AI ​​model is acquired. The computation graph includes dynamic files and dynamic lookup operators. The dynamic files are used to store vectors based on dynamic data structures, and the dynamic lookup operators are used to look up vectors from the dynamic files. The dynamic file in the computation graph is replaced with the first vector file, and the dynamic search operator in the computation graph is replaced with a static search operator, which is used to search for vectors from the first vector file.

6. The method according to claim 5, characterized in that, The method further includes: Before generating the AI ​​model based on the replaced computation graph, delete the initialization or recovery operators associated with the dynamic file in the replaced computation graph.

7. The method according to any one of claims 1 to 6, characterized in that, The AI ​​model includes a first vector file and a second vector file. The first vector file includes the inference vectors, and the second vector file stores vectors based on a dynamic data structure. The method further includes: Before the general-purpose processor acquires the inference data, the amount of memory resources of the accelerator card is acquired; Based on the memory resources of the accelerator card, the first vector file is deployed to the accelerator card, and the second vector file is deployed to the general-purpose processor.

8. The method according to any one of claims 1 to 7, characterized in that, The AI ​​model includes a first vector file and a second vector file. The first vector file includes the inference vectors, and the second vector file stores vectors based on a dynamic data structure. The method further includes: Output the interactive interface; In response to the user's configuration operation for the plurality of vector files on the interactive interface, the first vector file is deployed to the AI ​​accelerator card, and the second vector file is deployed to the general-purpose processor.

9. The method according to any one of claims 1 to 8, characterized in that, The AI ​​model is used for information recommendation or natural language processing, or the AI ​​model is a graph neural network (GNN) model.

10. A data processing system, characterized in that, The data processing system includes a general-purpose processor and an artificial intelligence accelerator card; The general-purpose processor is used to acquire inference data; determine, based on the inference data, the information of the accelerator card storing the inference vector corresponding to the inference data and the position information of the inference vector in the accelerator card; and send the position information to the accelerator card indicated by the information of the accelerator card. The acceleration card is used to obtain the inference vector based on the location information; and to perform inference operations using the AI ​​model based on the inference vector.

11. The system according to claim 10, characterized in that, The accelerator card includes a first vector file, and the location information includes the file information of the first vector file and the position of the inference vector in the first vector file.

12. The system according to claim 11, characterized in that, The system also includes a preprocessing device for: The dynamic hash table used to store the relationship between feature values ​​and vectors is converted into a static vector table, which sequentially stores the vector corresponding to each feature value. The first vector file is generated based on the static vector table, and the first vector file includes vectors corresponding to some feature values ​​in the static vector table. Offset information is set for each feature value in the first vector file, wherein the offset information for each feature value indicates the position of the vector corresponding to each feature value in the first vector file; Store the first vector file and the location information of each feature value in the first vector file into the accelerator card; When the accelerator card obtains the inference vector based on the location information, it is specifically used for: Access the first vector file based on the file information, and obtain the inference vector based on the offset information in the location information.

13. The system according to any one of claims 10 to 12, characterized in that, When the general-purpose processor determines, based on the inference data, the information of the accelerator card storing the inference vector corresponding to the inference data and the position information of the inference vector in the accelerator card, specifically for: The feature value corresponding to the inference data is determined based on the inference data; the file information of the first vector file to which the inference vector corresponding to the feature value belongs, and the position information of the inference vector corresponding to the feature value in the first vector file are determined based on the feature value.

14. The system according to any one of claims 10 to 13, characterized in that, The system also includes a preprocessing device for: Before the general-purpose processor acquires the inference data, the computation graph of the AI ​​model is acquired. The computation graph includes dynamic files and dynamic lookup operators. The dynamic files are used to store vectors based on dynamic data structures, and the dynamic lookup operators are used to look up vectors from the dynamic files. The dynamic file in the computation graph is replaced with the first vector file, and the dynamic search operator in the computation graph is replaced with a static search operator, which is used to search for vectors from the first vector file.

15. The system according to claim 14, characterized in that, The pretreatment equipment is also used for: Before generating the AI ​​model based on the replaced computation graph, delete the initialization or recovery operators associated with the dynamic file in the replaced computation graph.

16. The system according to any one of claims 10 to 15, characterized in that, The AI ​​model includes a first vector file and a second vector file. The first vector file includes the inference vectors, and the second vector file stores vectors based on a dynamic data structure. The system also includes a preprocessing device for: Before the general-purpose processor acquires the inference data, the memory resource amount of the accelerator card is acquired; based on the memory resource amount of the accelerator card, the first vector file is deployed to the accelerator card, and the second vector file is deployed to the general-purpose processor.

17. The system according to any one of claims 10 to 16, characterized in that, The AI ​​model includes a first vector file and a second vector file. The first vector file includes the inference vectors, and the second vector file stores vectors based on a dynamic data structure. The system also includes a preprocessing device for: Output the interactive interface; In response to a user's configuration operation for a vector file on the interactive interface, the first vector file is deployed to the accelerator card, and the second vector file is deployed to the general-purpose processor.

18. The system according to any one of claims 10 to 17, characterized in that, The AI ​​model is used for information recommendation or natural language processing, or the AI ​​model is a graph neural network (GNN) model.

19. A computer-readable storage medium, characterized in that, Includes instructions that, when executed on at least one computing device, cause the at least one computing device to perform the steps of the method as described in any one of claims 1 to 9.

20. A computer program product containing instructions, characterized in that, When it is run on at least one computing device, it causes the at least one computing device to perform the method as described in any one of claims 1 to 9.

Images