Method and apparatus for running ai model, and program product and storage medium

Abstract

The present application belongs to the technical field of machine learning. Provided are a method and apparatus for running an AI model, and a program product and a storage medium. The method is applied to a host, wherein the host comprises a processor and is connected to a computing card. The method comprises: a processor determining, from two adjacent groups of input data for an AI model, first data that is present in a second data group but is not present in a first data group, wherein the second data group is trained after the first data group; prefetching to a first memory of the processor a first embedding vector corresponding to the first data, and determining first embedding vector information corresponding to the first data; and during the processing of the second data group on a computing card, prefetching the first embedding vector from the first memory to a second memory of the computing card on the basis of the first embedding vector information. By means of the solution shown in the present application, a processing delay caused by the transfer of embedding vectors can be reduced.

Description

AI model operation method, device, program product and storage medium

[0001] This application claims priority to a Chinese patent application filed on March 5, 2024, with application number 202410254670.X and entitled “AI model operation method, device, program product and storage medium”. The entire contents of this patent application are incorporated into this application by reference. Technical Field

[0002] The present application relates to the field of machine learning technology, and in particular to an operating method, device, program product, and storage medium for an AI model. Background Art

[0003] In the Internet scenario, in order to express an object input into the network, an artificial intelligence (AI) model usually uses a low-dimensional dense vector in the model parameters to represent the object. This low-dimensional dense vector is called an embedding vector. The number of parameters of the embedding vector increases with the number of feature types of the AI ​​model. The host's computing card may not be able to store all of them. Therefore, the embedding vector is stored in multi-level storage media. For example, the host's computing card, the host's double data rate synchronous dynamic random access memory (DDR) and solid state drive (SSD). In this way, since the embedding vector needs to be transported to the computing card for calculation, the way the embedding vector is transported is crucial. Summary of the Invention

[0004] The present application provides an AI model operation method, device, program product and storage medium, which can pre-fetch the required embedding vectors into the memory of the computing card before processing the next data group, thereby reducing the processing delay caused by moving the embedding vectors.

[0005] In a first aspect, the present application provides a method for operating an AI model, which is applied to a host, the host including a processor and connected to a computing card, the method comprising: the processor determining the first data in the second data group but not in the first data group in two adjacent groups of input data of the AI ​​model, wherein the second data group is trained after the first data group; pre-fetching the first embedding vector corresponding to the first data to the first memory of the processor, and determining the first embedding vector information corresponding to the first data; and pre-fetching the first embedding vector from the first memory to the second memory of the computing card according to the first embedding vector information during processing of the first data group on the computing card.

[0006] In the solution shown in the present application, the first data group and the second data group are adjacent data groups during training, and the second data group is processed after the first data group. Before the second data group is processed, the first data in the second data group but not in the first data group is determined, and the first embedding vector corresponding to the first data is pre-fetched into the first memory. During the processing of the first data group on the computing card, the first embedding vector corresponding to the first data is pre-fetched into the second memory of the computing card. In this way, the first embedding vector is pre-fetched into the first memory in advance, and when the first embedding vector is really needed to be transferred from the host to the computing card, the first embedding vector exists in the memory of the host and does not need to be obtained from other storage media such as SSD. Moreover, the first embedding vector is pre-fetched into the memory of the computing card in advance, and can be used directly when the first embedding vector is used for processing without waiting, which can reduce the processing delay caused by transferring the embedding vector.

[0007] In an optional embodiment, the method includes: determining second data that is in the first data group but not in the second data group, and after the first data group is processed on the computing card, swapping a second embedding vector corresponding to the second data from the second memory to the first memory.

[0008] In the solution shown in the present application, after the processing of the first data group is completed, the embedded vectors that are not needed for the second data group are swapped out to the host's memory for storage, and the embedded vectors stored in the host can be updated.

[0009] In an optional manner, the first embedded vector information is an offset address of the first embedded vector in the second memory when the first data is processed on the computing card.

[0010] In an optional manner, the host stores a lock-free queue, which is used to store idle offset addresses in the second memory; determining the first embedded vector information corresponding to the first data includes: selecting the first embedded vector information in the lock-free queue; after selecting the first embedded vector information, the method also includes: deleting the first embedded vector information in the lock-free queue.

[0011] In the solution shown in the present application, the host maintains an idle offset address in the second memory, and obtains the offset address corresponding to the first embedded vector from the idle offset address, so as to quickly obtain the corresponding offset address for the first embedded vector.

[0012] In one optional embodiment, a host stores at least one hash bucket for storing input data corresponding to embedded vector information in the input data of the AI ​​model; the processor determines first data in two adjacent groups of input data of the AI ​​model that is in the second data group but not in the first data group, including: the processor determining a hash value corresponding to each input data in the second data group; and determining first data in the second data group that does not belong to the at least one hash bucket based on the hash value corresponding to each input data. After determining the first embedded vector information corresponding to the first data, the method also includes the processor adding the first data to the corresponding hash bucket based on the hash value corresponding to the first data.

[0013] In the solution shown in the present application, a hash bucket is set up to quickly find the first data that is in the second data group but not in the first data group, thereby improving data search efficiency. In addition, the input data belonging to the first embedding vector are all present in the hash bucket, which facilitates the pre-fetching of the embedding vector of the next data group after the second data group.

[0014] In an optional approach, the space capacity of each hash bucket is equal to the cache capacity of the processor's L1 cache line. This allows the processor to read data from one hash bucket at a time, improving data search efficiency.

[0015] In an optional manner, the second memory corresponds to a first indicator bit and a second indicator bit, the first indicator bit is used to indicate an offset address corresponding to a third embedded vector in the second memory, the third embedded vector includes the embedded vector corresponding to each input data in the first data group and the first embedded vector, the second indicator bit is used to indicate an offset address corresponding to a fourth embedded vector in the second memory, the fourth embedded vector includes the embedded vector corresponding to each input data in the second data group, and the determination of the second data in the first data group but not in the second data group includes: the processor determines the offset address corresponding to the second data in the second memory according to the first indicator bit and the second indicator bit, and obtains the second data based on the offset address corresponding to the second data.

[0016] In the solution shown in the present application, for the same offset address, by maintaining two indicator bits, the second data that needs to be swapped out can be quickly and accurately determined.

[0017] In a second aspect, the present application provides an AI model operation device, which has the function of implementing the above-mentioned first aspect or any optional method of the first aspect. The device includes at least one module, and the at least one module is used to implement the AI ​​model operation method provided by the above-mentioned first aspect or any optional method of the first aspect.

[0018] In a third aspect, the present application provides a computer-readable storage medium storing at least one computer instruction, which is read by a processor to enable a computing device to execute the method for operating the AI ​​model provided in the first aspect or any optional method of the first aspect.

[0019] In a fourth aspect, the present application provides a computing device comprising a processor and a memory, wherein: the memory stores computer instructions; the processor executes the computer instructions so that the computing device executes the method for operating the AI ​​model provided in the first aspect or any optional method of the first aspect.

[0020] In a fifth aspect, the present application provides a computer program product comprising computer instructions stored in a computer-readable storage medium. A processor of a computing device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computing device to execute the method for operating the AI ​​model provided in the first aspect or any optional embodiment of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] FIG1 is a schematic diagram of the use of an embedding vector provided by an exemplary embodiment of the present application;

[0022] FIG2 is a schematic diagram of the architecture of a host provided by an exemplary embodiment of the present application;

[0023] FIG3 is a schematic diagram of the hardware structure of a host provided by an exemplary embodiment of the present application;

[0024] FIG4 is a flowchart of an AI model operation method provided by an exemplary embodiment of the present application;

[0025] FIG5 is a schematic diagram of determining first data provided by an exemplary embodiment of the present application;

[0026] FIG6 is a schematic diagram of a swap-in and swap-out framework provided by an exemplary embodiment of the present application;

[0027] FIG7 is a schematic diagram of a hash bucket provided by an exemplary embodiment of the present application;

[0028] FIG8 is a schematic diagram of an initialization state provided by an exemplary embodiment of the present application;

[0029] FIG9 is a schematic diagram of a state of swapping in a first data group provided by an exemplary embodiment of the present application;

[0030] FIG10 is a schematic diagram of a state of switching into a second data group provided by an exemplary embodiment of the present application;

[0031] FIG11 is a schematic diagram of a state of swapping out a first data group provided by an exemplary embodiment of the present application;

[0032] FIG12 is a schematic diagram illustrating a timing diagram provided by an exemplary embodiment of the present application;

[0033] FIG13 is a schematic structural diagram of an AI model operating device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0035] In internet search and recommendation scenarios, AI models typically use an embedding vector in their model parameters to represent input data (such as an object or a person). This embedding vector is updated as the model trains. For example, as shown in Figure 1, the input data for the AI ​​model includes three car brands, Car Brand 1 through Car Brand 3, with embedding vectors of [0.5, 0.4, 0.3, 0.1], [0.2, 0.7, 0.9, 0.8], and [0.4, 0.8, 0.5, 0.5], respectively. Embedding vectors are currently widely used. However, as the number of features trained on AI models increases, the amount of embedding vector data also surges. The host's computing card may not be able to store all the embedding vectors. Therefore, the embedding vectors are stored in multiple storage media, such as the host's computing card, the host's DDR memory, and an SSD, with the SSD plugged into the host. Because the embedding vectors need to be transferred to the computing card for calculation, the transfer method directly affects computational efficiency, making the transfer method crucial.

[0036] In related technologies, during model training, to reduce the time overhead of frequently swapping embedding vectors between storage media, the training dataset is preprocessed and split into multiple groups of data. Before training each group of data, the embedding vectors for that group of data are loaded into the high-bandwidth memory (HBM) of the computing card. Training for that group of data begins, and after training is complete, the trained embedding vectors for that group of data are transferred to the host side for updating. This process, because adjacent data groups may contain duplicates, requires waiting for the embedding vectors of the previous group of data to be updated to ensure the accuracy of the training data's embedding vectors, resulting in a longer training time.

[0037] In an embodiment of the present application, multiple input data are split into multiple data groups. When the computing card uses the current data group for training, it is determined that the embedding vectors in the next data group are not stored in the input data of the computing card. The embedding vectors of these input data are first pre-fetched into the memory of the processor. Then, during the training of the current data group, the embedding vectors are pre-fetched from the memory of the processor to the memory of the computing card. In this way, after the training of the current data group is completed, the embedding vectors of the next data group are all stored in the computing card, and the next data group can be trained directly without waiting for transfer.

[0038] In the following, the technical solution provided in the embodiment of this application is described in detail in the order of the host architecture, the hardware structure of the host, and the operation method flow of the AI ​​model.

[0039] The following describes the architecture of the host provided in the embodiments of the present application.

[0040] Figure 2 provides a schematic diagram of the host architecture. The host can be a computing device such as a server or a computer. Referring to Figure 2, the host is connected to at least one computing card and at least one SSD. In this way, it is equivalent to having three levels of storage media, including the host's DDR, the computing card's HBM, and the SSD. Alternatively, the host is connected to at least one computing card and is also connected to a network storage system. In this way, it is equivalent to having three levels of storage media, including the host's DDR, the computing card's HBM, and the network storage system. Alternatively, the host is connected to at least one computing card and at least one SSD, and is also connected to a network storage system. In this way, it is equivalent to having four levels of storage media, including the host's DDR, the computing card's HBM, the SSD, and the network storage system. When the embedding vector management method is applied to the model training process, the computing card is called a training card. When the embedding vector management method is applied to the model inference process, the computing card is called an inference card. Computing cards include but are not limited to graphics processing units (GPUs) and neural-network processing units (NPUs).

[0041] Furthermore, "at least one computing card is connected to the host" can be understood as meaning that the computing card is plugged into the host, or that the computing card is a remote computing card, and the host and the remote computing card can communicate over a network, such as when the host communicates with the host into which the remote computing card is plugged, thereby enabling communication with the remote computing card. Thus, the host can be understood as the host of the computing card, and the computing card can be understood as a device of the host.

[0042] The following is an introduction to the hardware structure of the host.

[0043] As shown in FIG3 , the host can be optionally implemented by a general bus architecture, and includes at least one processor 301 , a communication bus 302 , a memory 303 , and at least one network interface 304 .

[0044] The processor 301 is, for example, a general-purpose central processing unit (CPU), a network processor (NP), a GPU, an NPU, a data processing unit (DPU), a microprocessor, or one or more integrated circuits for implementing the solution of the present application. For example, the processor 301 includes an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD is, for example, a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0045] Communication bus 302 is used to transmit information between the above components. Communication bus 302 can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in Figure 3, but this does not mean that there is only one bus or one type of bus.

[0046] The memory 303 is, for example, a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 303 is, for example, independent and connected to the processor 301 via the communication bus 302. The memory 303 can also be integrated with the processor 301.

[0047] Optionally, the memory 303 is used to store the embedded vector mentioned below. When the host needs to use the embedded vector, the processor 301 accesses the memory 303 to obtain the embedded vector stored in the memory 303.

[0048] The network interface 304 uses any transceiver-like device for communicating with other devices or communication networks. The network interface 304 includes a wired network interface and may also include a wireless network interface. For example, the wired network interface may be an Ethernet interface. The Ethernet interface may be an optical interface, an electrical interface, or a combination thereof. The wireless network interface may be a wireless local area network (WLAN) interface, a cellular network interface, or a combination thereof.

[0049] In a specific implementation, as an example, the processor 301 may include one or more CPUs.

[0050] In a specific implementation, as an example, a host may include multiple processors. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0051] In a specific implementation, as an example, the host may further include an output device and an input device. The output device communicates with the processor 301 and can display information in a variety of ways. For example, the output device can be a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector. The input device communicates with the processor 301 and receives user input in a variety of ways. For example, the input device can be a mouse, a keyboard, a touch screen device, or a sensor device.

[0052] In some embodiments, the memory 303 is used to store the program code 3031 for executing the embedded vector management in this application, and the processor 301 executes the program code 3031 stored in the memory 303. That is, the host can implement the operation method of the AI ​​model provided by the method embodiment through the processor 301 and the program code 3031 in the memory 303.

[0053] The following describes the process of the operation method of the AI ​​model. The AI ​​model can be any neural network model with an embedding vector, including but not limited to a recommendation model and / or a search model, which is not limited in the embodiments of the present application. For example, the AI ​​model is a car brand search model, a recommendation model for a shopping application (such as when a user opens a shopping application, the content displayed on the homepage is based on the recommendation model), a search model for a shopping application, or a model for recommending information in a sharing area of ​​a social application, etc.

[0054] FIG4 provides a flow chart of the method for running the AI ​​model, see steps 401 to 403. The execution subject of the method for running the AI ​​model is a processor.

[0055] In step 401 , the processor determines the first data in two adjacent sets of input data for an AI model that is in the second data set but not in the first data set, wherein the second data set is trained after the first data set.

[0056] Among them, the input data is the embedding vector mentioned above.

[0057] In this embodiment, when the operation method of the AI ​​model is applied to the AI ​​model training process, the input data (i.e., training data) is divided into multiple data groups, each data group includes a batch (batch) or multiple batches of training data, and the amount of data included in each data group is set according to the experience value or the actual application scenario. Among the multiple data groups, the first data group is adjacent to the second data group, and the second data group is trained after the first data group, that is, after the first data group is trained, the second data group is trained. In this embodiment of the present application, the first data group and the second data group are any two adjacent data groups among the multiple data groups.

[0058] The host computer determines the second data group in advance, determines whether each input data in the second data group belongs to the first data group, and determines that the input data in the second data group but not in the first data group is the first data. For example, referring to Figure 5, the first data group includes input data 1 to input data 5, the second data group includes input data 4 to input data 8, and the first data group includes input data 6 to input data 8.

[0059] Step 402: Pre-fetch a first embedded vector corresponding to the first data into a first memory of a processor, and determine first embedded vector information corresponding to the first data.

[0060] In this embodiment, after determining the first data, the processor determines the embedded vector corresponding to the first data, which is called the first embedded vector. If the first embedded vector exists in the SSD or network storage system, the first embedded vector is pre-fetched into the first memory of the processor. In this way, the embedded vectors of these input data are pre-fetched from the remote network storage system or SSD to the first memory of the processor in advance. The first memory can be the DDR of the host. When the embedded vectors really need to be transferred from the first memory to the computing card, the required embedded vectors all exist in the first memory, and the first embedded vector can be hit in the first memory, saving the waiting time for transfer.

[0061] And the processor determines the first embedding vector information corresponding to the first data. The first embedding vector information is the address of the first embedding vector in the second memory of the computing card when the first data is trained on the computing card. The address can be an offset address, indicating the offset of the storage location relative to a first address. The second memory is the memory allocated on the computing card for storing the embedded vector, which belongs to the HBM of the computing card. Here, in order to not affect the first data group when pre-fetching the first embedded vector to the computing card, in the second memory, the offset address of the first embedded vector is different from the offset address of the embedded vector of the first data group. The host is configured with information of the second memory of the computing card for storing the embedded vector. In the second memory, the processor assigns an offset address to the first embedded vector in the unused offset address of the embedded vector of the first data group. In this way, when the first embedded vector is pre-fetched for the computing card, the training process of the first data group will not be affected.

[0062] It should be noted that, in step 402, there is no order between determining the first embedded vector information and pre-fetching the first embedded vector into the first memory.

[0063] Step 403: Prefetch the first embedded vector from the first memory to the second memory of the computing card according to the first embedded vector information. The first embedded vector is prefetched to the second memory during the processing of the first data group.

[0064] In this embodiment, after determining the first embedding vector, the processor sends the first embedding vector and first embedding vector information to the computing card. The computing card uses the first embedding vector information to store the first embedding vector in the second memory. After the computing card completes training on the first data group, it trains the second data group. Alternatively, the processor can monitor the training process of the computing card and, after determining that training for the first data group has begun, send the first embedding vector and first embedding vector information to the computing card. The computing card uses the first embedding vector information to store the first embedding vector in the second memory. After the computing card completes training on the first data group, it trains the second data group.

[0065] For each data set, the training process is as follows: the computing card uses the embedding vectors of the data in that data set for forward inference, then calculates the loss, updates the embedding vector based on the loss, and replaces the original embedding vector with the updated embedding vector. It should be noted that to avoid the delay caused by the transfer of embedding vectors, the storage space used to store the embedding vectors on the computing card is greater than or equal to the storage space for storing the embedding vectors of the two data sets. This way, even if two adjacent data sets do not have the same input data, the embedding vectors of the later-trained data set can be swapped into the computing card in advance.

[0066] Optionally, when swapping the first embedded vector into the second memory, the processor sends the first embedded vector to a queue of a corresponding channel of the second memory, and the computing card obtains the first embedded vector from the queue using the first embedded vector information.

[0067] Using the process shown in Figure 4, during the training of the first data group, the embedding vectors of the input data in the second data group that are not stored in the computing card are asynchronously pre-fetched into the memory of the computing card. In this way, when the computing card uses the second data group for training, it can start training without waiting, which can save waiting time. Moreover, when starting training, the process shown in Figure 4 is executed, and the first embedding vector to be swapped in is determined by data analysis. The process of pre-fetching the first embedding vector to the computing card is parallel to the data analysis and graph execution process. Since the process of determining the first embedding vector only depends on data analysis, it can be significantly ahead of the graph execution process of AI model training. Basically, during the training of the first data group on the computing card, the first embedding vector can be swapped into the second memory. Of course, when the data analysis process is relatively fast and there is enough space in the second memory, the first embedding vector can be swapped into the second memory before training the first data group.

[0068] The process shown in FIG4 is described in detail below.

[0069] In an optional method, the storage space of the second memory of the computing card is limited. When the AI ​​model operation method is applied to the model training process, the embedding vector of the input data will be updated. After each data group training is completed, the embedding vector not used in the next data group can be moved to the host for storage. The processing process is:

[0070] After the processor completes processing on the computing card using the first data set, it determines second data that is in the first data set but not in the second data set. After training the first data set on the computing card, the processor swaps the second embedding vector corresponding to the second data from the second memory to the first memory for storage. This allows the processor to obtain the latest embedding vector corresponding to the second data when it subsequently retrieves the embedding vector corresponding to the second data.

[0071] Optionally, the processor sends the second embedded vector to a queue of a corresponding channel of the first memory, and stores the embedded vector in the first memory as long as there is an embedded vector in the queue.

[0072] Alternatively, when the second memory is relatively large, the second embedded vector may not be swapped out to the host's storage space for storage. Instead, it may be swapped out to the first memory for storage when the computing card's storage space for the embedded vector is insufficient. For example, assuming the second data group is the i-th data group, during the use of the i-th data group, if the embedded vector of the input data in the i+1-th data group is swapped into the computing card and insufficient storage space is available, the second embedded vector may be swapped out to the host for storage. Exemplarily, the least recently used (LRU) principle may be used when swapping out the second embedded vector.

[0073] It should be noted that the full set of embedding vectors is stored in the SSD or network storage system. If the second memory is large, during the training process, updated embedding vectors are updated to the SSD or network storage system at regular intervals. Alternatively, during the training process, updated embedding vectors are updated to the SSD or network storage system every time a target number of data sets are trained. The target number can be preset, such as 10.

[0074] To better understand the embodiment of the present application, when applied to the model training process, Figure 6 provides a schematic diagram of the swap-in and swap-out framework. In Figure 6, it is determined that the embedding vectors of the input data that need to be swapped in for the second data group are C and D, C is stored in the SSD, C is loaded from the SSD to the DDR, and the offset address corresponding to C is determined to be 12, and the offset address corresponding to D is 34. C and D are sent to the queue of the corresponding channel of the HBM of the computing card, C is stored at 12, and D is stored at 34, so that the computing card can obtain C and D from the HBM.

[0075] After the training of the first data group is completed, the identifier of the second data that needs to be swapped out is determined, and the offset addresses corresponding to the embedded vectors A and B of the second data, including 56 and 78, are determined. 56 and 78 are added to the queue of the corresponding channel of the HBM so that the computing card obtains 56 and 78 from the queue, reads A and B at the addresses indicated by 56 and 78, and adds them to the queue of the corresponding channel of the DDR. Based on the identifier of the data that needs to be swapped out, A and B are updated in the SSD and DDR.

[0076] Optionally, in FIG6 , since B is no longer used for a period of time, B is updated to the SSD so that the host's DDR stores the most recently used embedding vectors.

[0077] In an optional manner, in order to quickly determine the data swapped in (first data), the offset address corresponding to the first data, the data swapped out (second data) and the offset address corresponding to the second data, an embodiment of the present application provides the following data structure and interface function.

[0078] Data structure 1: Hash bucket of key-value pairs. This hash bucket is used to store key-value pairs corresponding to input data with offset addresses in the input data of the AI ​​model. The key is the identifier of the input data. The identifiers of different input data are different. The identifier of the input data is set according to actual needs. Any identifier that can represent an input data can be used as the identifier of the input data. For example, the input identifier can be a 64-bit integer (int64), etc. The value is the offset address of the embedded vector of the input data when it is stored in the computing card. The hash bucket can store elements with hash conflicts through a linked list or other data structure. When a hash conflict occurs, the new key-value pair will be inserted into the corresponding hash bucket to form a linked list. When searching, the hash value calculated by the hash function is first calculated to obtain the corresponding hash bucket, and then a linear search is performed in the hash bucket until the specified key-value pair is found or the end of the linked list is found. In addition, the hash bucket also includes a lock flag bit. The lock flag bit is used to set the lock flag bit to a first value when a process operates the hash bucket. The first value is used to indicate that other processes cannot operate the hash bucket. After the process completes the operation on the hash bucket, the lock flag bit is set to a second value. The second value is used to indicate that other processes can operate the hash bucket. In this way, conflicts caused by multiple processes accessing the hash bucket simultaneously can be reduced. Here, the hash bucket can be set to one or more according to actual needs. See Figure 7, which shows three hash buckets.

[0079] Optionally, the storage capacity of each hash bucket is equal to the cache capacity of the processor's first-level cache line. In this way, when the host processor reads data from the hash bucket, it can read all the data in a hash bucket, and can greatly reduce the number of times the linked list nodes are created and traversed after the hash conflict. Each key-value pair in the hash bucket is a linked list node.

[0080] Data structure 2: first indicator bit. The first indicator bit is used to record the occupied offset addresses and free offset addresses in the second memory. The second memory includes the offset addresses of the addresses used by the computing card to store embedded vectors. The number of elements in the second memory is equal to the maximum cache size of embedded vectors in the computing card. Among them, the first indicator bit of each offset address in the second memory is initialized to the first value, indicating that no embedded vector is stored at each offset address. When an embedded vector is stored at a certain offset address, the first indicator bit of the offset address is updated to the second value. In this way, when the first indicator bit of a certain offset address takes the first value, it means that the address indicated by the offset address does not store an embedded vector. When the first indicator bit of a certain offset address takes the second value, it means that the address indicated by the offset address stores an embedded vector. The first value and the second value are different. For example, the first value is 0 and the second value is 1, or the first value is 1 and the second value is 0.

[0081] Data structure 3: second indicator bit. The second indicator bit corresponds to the data group and is reset when a new second data group is loaded. The second indicator bit is used to record the offset address occupied by the second data group and the idle offset address in the second memory. The second indicator bit of each offset address in the second memory is initialized to the second value, indicating that no embedded vector is stored at each offset address. When an embedded vector is stored at a certain offset address, the second indicator bit of the offset address is updated to the first value. In this way, when the second indicator bit of a certain offset address takes the first value, it means that the address indicated by the offset address stores an embedded vector. When the second indicator bit of a certain offset address takes the second value, it means that the address indicated by the offset address does not store an embedded vector. The first value and the second value are different. For example, the first value is 0 and the second value is 1, or the first value is 1 and the second value is 0.

[0082] Data Structure 4: Data Queue. The data queue, also known as the offset vector queue, is equal in length to the total number of offset addresses in the second memory. It is used to record the identifier of the input data corresponding to the target offset address. The input data corresponding to the target offset address is the input data belonging to the embedded vector stored at the target offset address. The target offset address includes the offset address in the second memory corresponding to the input data. The data queue is initialized to all -1 or other fixed values. When an embedded vector is stored at an offset address, the identifier of the data occupying that offset address is located at the corresponding position in the data queue.

[0083] Data structure 5: lock-free queue. The lock-free queue is used to record the offset addresses of the idle state in the second memory.

[0084] Interface function 1: a function used to search for the first data and change the first indicator bit and the second indicator bit of the offset address. The process of calling interface function 1 is as follows: for each input data in the second data group, the host calls interface function 1 and searches the hash bucket of the key-value pair for the existence of the input data. If it exists, the second indicator bit of the offset address corresponding to the input data is set to the first value, and the value indicating the existence of the key-value pair is returned. If it does not exist, an idle offset address that can be used is obtained in the lock-free queue, the idle offset address is deleted in the lock-free queue, and the key-value pair of the identifier of the input data and the corresponding offset address is inserted into the hash bucket of the key-value pair, the second indicator bit of the offset address corresponding to the input data is set to the first value, the first indicator bit of the offset address corresponding to the input data is set to the second value, the position corresponding to the offset address in the data queue is set to the identifier of the input data, and the value indicating that the key-value pair does not exist is returned.

[0085] Interface function 2: a function for obtaining the key-value pair of input data and offset address swapped out from the computing card to the first memory. The process of calling interface function 2 is: performing an "AND operation" on the first indicator bit and the second indicator bit of each offset address in the second memory, determining the offset address with a result of 1 as the offset address of the second data that needs to be swapped out (i.e., the first offset address), finding the identifier of the second data corresponding to the first offset address in the data queue, obtaining the key-value pair corresponding to the second data, and returning the obtained key-value pair, which includes the identifier of the second data and the first offset address. The above is explained using the example that the first numerical value and the second numerical value are different. In practice, the first numerical value and the second numerical value may also be the same, except that after performing an "AND operation" on the first indicator bit and the second indicator bit, the offset address with a result of 0 is determined as the offset address of the second data that needs to be swapped out.

[0086] Interface function 3: A function used to remove a key-value pair from the hash bucket. The process of calling interface function 3 is as follows: after the embedded vector corresponding to the second data is swapped out from the computing card to the first memory of the host, the key-value pair corresponding to the second data is removed from the hash bucket, the first indicator bit of the offset address in the key-value pair is set to the first value, the value of the corresponding position in the data queue is updated to -1, and the offset address is added to the lock-free queue.

[0087] Interface function 4 is used to reset the second indicator bit of each offset address in the second memory. The process of calling interface function 4 is as follows: when the second data set is loaded into the computing card, the second indicator bit of each offset address in the second memory is first set to the second value, and then the second indicator bit of the offset address corresponding to each embedded vector in the second data set is set to the first value.

[0088] The following is a specific example to illustrate the process of moving the embedded vector. Assume that the number of elements in the second memory is 7, that is, the maximum cache size is 7 embedded vectors. Here, it is considered that there are at least two identical embedded vectors in adjacent data groups. According to the definition of the above data structure, the state initialization is shown in Figure 8. In Figure 8, the lock-free queue includes 0, 1, 2, 3, 4, 5, and 6. The first indicator bit of each offset address in the second memory is 0000000, and the second indicator bit of each offset address in the second memory is 1111111. The data queue includes -1, -1, -1, -1, -1, -1, and the hash bucket of the key-value pair is "empty".

[0089] Assume that the first data group includes 5 input data, identified as 1, 2, 3, 4, and 5, respectively. Call interface functions 1 to 4 and update the state in Figure 8 to the state shown in Figure 9. The specific process is as follows: call interface function 1, find that input data 1 to input data 5 do not exist in the hash bucket, determine that the offset addresses corresponding to input data 1 to input data 5 are 0 to 4 in the lock-free queue, update the first indicator bit of the first 5 offset addresses in the second memory to 1, and obtain the first indicator bit of each offset address in the second memory as 1111100, update the second indicator bit of the first 5 offset addresses in the second memory to 0, and obtain the second indicator bit of each offset address in the second memory as 0000011. The lock-free queue corresponding to the first data group includes 5 and 6. The key-value pairs that need to be swapped into the computing card are 1-0, 2-1, 3-2, 4-3, and 5-4, and return the value used to indicate that the key-value pair does not exist. Call interface function 2 to perform an AND operation on the first indicator bit and the second indicator bit of the offset address of the second memory. The results are all 0, indicating that there is no key-value pair that needs to be swapped out to the first memory.

[0090] Before the computing card trains the second data group, it determines that the next training data group is the second data group. The second data group includes 5 input data, which are identified as 1, 2, 4, 6, and 8. Interface functions 1 to 4 are called to update the update shown in Figure 9 to the state shown in Figure 10. The specific process is as follows: first call interface function 4, set the second indicator bit of each offset address in the second memory to 1, call interface function 1, find that input data 6 and 8 do not exist in the hash bucket, determine in the lock-free queue that the offset addresses corresponding to input data 6 and 8 are 5 and 6, update the first indicator bit of the 6th and 7th offset addresses in the second memory to 1, and obtain the first indicator bit of each offset address in the second memory to 1111111, and update the second indicator bit of the offset address in the second memory except the 3rd and 5th offset addresses to 0, and obtain the second indicator bit of each offset address in the second memory to 0010100. At this point, the lock-free queue is empty. The key-value pairs that need to be swapped into the compute card are 6-5 and 8-6, and a value is returned indicating that the key-value pairs 6-5 and 8-6 do not exist. Call interface function 2 and perform an AND operation on the first and second indicator bits of the second memory offset address. The result is 0010100. It can be seen that the results for the third and fifth offset addresses are 1. The embedded vectors for the third and fifth offset addresses need to be swapped out. The key-value pairs are 3-2 and 5-4. Then call interface function 3 to remove the key-value pairs 3-2 and 5-4 from the hash bucket and update the first indicator bits of offset addresses 2 and 4 to 0. In the data queue, the values ​​corresponding to offset addresses 2 and 4 are updated to -1. Offset addresses 2 and 4 are added to the lock-free queue as free offset addresses. The state of the hash bucket and the second data group after swapping in is shown in Figure 11.

[0091] In this embodiment, the forward and reverse execution time of the computation card's computation graph is used to mask the preparation, update, and transfer of the embedding vectors for the next data set, reducing the extra time overhead associated with heterogeneous storage during computation. Furthermore, by precalculating the embedding vectors for the next data set that are not stored on the computation card and only transferring those not stored on the computation card, redundant transfers are avoided, improving transfer efficiency.

[0092] To better understand how the time required to update, query, and transfer embedding vectors is masked by the graph execution time lock when applying the AI ​​model operation method to model training, a timing diagram is provided in Figure 12. The dashed lines in Figure 12 represent the graph execution time for each data group on the compute card. Figure 12 uses the swapping in and out of embedding vectors for the input data in the second data group as an example for illustration. Figure 12 shows that the preparation and swapping in of the embedding vectors for the input data in the second data group are completed in advance during the graph execution of the first data group, and their timing is masked by the graph execution time. The swapping out and updating of the embedding vectors for the input data in the second data group are completed during the graph execution of the third data group, and their timing is masked by the graph execution time. In Figure 12, the dashed boxes represent the calculation of the identifiers and offset addresses of the input data to be swapped in and out. In actual operation, this calculation relies solely on the parsing and preprocessing of the dataset, and is therefore asynchronous and ahead of the computational graph execution. These are generated in advance and stored in the queue used by the compute card to retrieve the embedding vectors. Their placement in the diagram is for ease of understanding only.

[0093] It should be noted that the AI ​​model operation method used in the AI ​​model training process in the previous article can of course also be used in the AI ​​model inference process, so that the next inference data group can be moved to the memory of the computing card in advance, saving the transfer time.

[0094] It should also be noted that the previous article uses a single host to train the AI ​​model as an example. When multiple hosts are training the AI ​​model, each host can use the AI ​​model operation method described in the previous article.

[0095] The following is a schematic diagram of the structure of the operating device of the AI ​​model.

[0096] FIG13 is a block diagram of an operating device for an AI model provided in an embodiment of the present application. The device can be implemented as part or all of a device through software, hardware, or a combination of both. The device is applied to a host, which includes a computing card. The device can implement the process described in FIG4 of the embodiment of the present application. The device includes: a determination module 1310, a pre-fetch module 1320, and a swap module 1330, wherein:

[0097] Determination module 1310, for determining first data in two adjacent sets of input data for the AI ​​model that is in a second data set but not in the first data set, wherein the second data set is trained after the first data set, and specifically for implementing the determination function of step 401 and executing the implicit steps included in step 401;

[0098] The prefetch module 1320 is configured to:

[0099] Pre-fetching a first embedded vector corresponding to the first data into a first memory of the processor, and determining first embedded vector information corresponding to the first data;

[0100] During processing of the first data group on the computing card, the first embedded vector is prefetched from the first memory to the second memory of the computing card according to the first embedded vector information, which can be specifically used to implement the prefetching function of step 402 and step 403 and execute the implicit steps included in step 402 and step 403.

[0101] In an optional manner, the determining module 1310 is further configured to determine second data that is in the first data group but not in the second data group;

[0102] The apparatus further includes a swapping module 1330 for swapping out a second embedding vector corresponding to the second data from the second memory to the first memory after the first data group is processed on the computing card.

[0103] In an optional manner, the first embedding vector information is an offset address of the first embedding vector in the second memory when the first data is trained on the computing card.

[0104] In an optional manner, the host stores a lock-free queue, and the lock-free queue is used to store the idle offset address in the second memory;

[0105] The prefetch module 1320 is configured to select the first embedded vector information from the lock-free queue;

[0106] The prefetch module 1320 is further configured to delete the first embedded vector information from the lock-free queue after selecting the first embedded vector information.

[0107] In an optional manner, the host stores at least one hash bucket, and the at least one hash bucket is used to store input data corresponding to the embedded vector information in the input data of the AI ​​model;

[0108] The determining module 1310 is configured to:

[0109] Determine a hash value corresponding to each input data in the second data group;

[0110] Determining, according to the hash values ​​corresponding to the respective input data, first data in the second data group that does not belong to the at least one hash bucket;

[0111] The determining module 1310 is further configured to: after determining the first embedding vector information corresponding to the first data, the processor adds the first data to a corresponding hash bucket according to a hash value corresponding to the first data.

[0112] In an optional manner, the second memory corresponds to a first indicator bit and a second indicator bit, the first indicator bit is used to indicate an offset address corresponding to a third embedded vector in the second memory, the third embedded vector includes an embedded vector corresponding to each input data in the first data group and the first embedded vector, and the second indicator bit is used to indicate an offset address corresponding to a fourth embedded vector in the second memory, the fourth embedded vector includes an embedded vector corresponding to each input data in the second data group;

[0113] The determining module 1310 is configured to:

[0114] Determining, in the second memory, an offset address corresponding to the second data according to the first indicator bit and the second indicator bit;

[0115] The second data is obtained based on the offset address corresponding to the second data.

[0116] For the detailed process of the AI ​​model running device shown in Figure 13 executing the AI ​​model, please refer to the description in the previous embodiments, which will not be repeated here.

[0117] In some embodiments, a computer program product is provided, the computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computing device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computing device to execute the process shown in FIG4 .

[0118] In some embodiments, a computer-readable storage medium is provided, which stores computer instructions. When the computer instructions in the computer-readable storage medium are executed by a computing device, the computing device executes the process shown in Figure 4.

[0119] Those skilled in the art will appreciate that the various method steps and units described in the embodiments disclosed in this application can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the steps and components of each embodiment have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0120] In the several embodiments provided in this application, it should be understood that the disclosed system architecture, device and method can be implemented in other ways. For example, the device embodiment described above is only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, or it can be an electrical, mechanical or other form of connection.

[0121] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0122] In addition, each module in each embodiment of the present application can be integrated into a processing module, each module can exist physically separately, or two or more modules can be integrated into a module. The above-mentioned integrated modules can be implemented in the form of hardware or software modules.

[0123] If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

[0124] In this application, the terms "first" and "second" are used to distinguish between identical or similar items having substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first" and "second", nor is there any limitation on quantity or order of execution. It should also be understood that although the following description uses the terms first and second, etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the various examples, a first embedded vector may be referred to as a second embedded vector, and similarly, a second embedded vector may be referred to as a first embedded vector. Both the first embedded vector and the second embedded vector may be embedded vectors, and in some cases, may be separate and different embedded vectors.

[0125] The above description is merely an exemplary embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and such modifications or substitutions should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for operating an artificial intelligence (AI) model, characterized in that: The method is applied to a host, the host including a processor and connected to a computing card, and the method includes: The processor determines, among two adjacent sets of input data for the AI ​​model, first data that is in a second data set but not in the first data set, wherein the second data set is trained after the first data set; Pre-fetching a first embedded vector corresponding to the first data into a first memory of the processor, and determining first embedded vector information corresponding to the first data; During processing of the first data group on the computing card, the first embedding vector is pre-fetched from the first memory to a second memory of the computing card according to the first embedding vector information.

2. The method according to claim 1, characterized in that The method includes: determining second data that is in the first data set but not in the second data set; After the first data group is processed on the computing card, the second embedding vector corresponding to the second data is swapped out from the second memory to the first memory.

3. The method according to claim 1 or 2, characterized in that The first embedded vector information is an offset address of the first embedded vector in the second memory when the first data is processed on the computing card.

4. The method according to claim 3, characterized in that The host stores a lock-free queue, and the lock-free queue is used to store the idle offset address in the second memory; The determining first embedding vector information corresponding to the first data includes: The processor selects the first embedded vector information from the lock-free queue; After selecting the first embedding vector information, the method further includes: The processor deletes the first embedded vector information from the lock-free queue.

5. The method according to any one of claims 1 to 4, characterized in that The host stores at least one hash bucket, and the at least one hash bucket is used to store input data corresponding to the embedded vector information in the input data of the AI ​​model; The processor determines, among two adjacent groups of input data for the AI ​​model, first data that is in the second data group but not in the first data group, including: The processor determines a hash value corresponding to each input data in the second data group; The processor determines, based on the hash values ​​corresponding to the respective input data, first data in the second data group that does not belong to the at least one hash bucket; After determining the first embedding vector information corresponding to the first data, the method further includes: The processor adds the first data to the corresponding hash bucket according to the hash value corresponding to the first data.

6. The method according to claim 5, characterized in that The space capacity of each hash bucket is equal to the cache capacity of the first-level cache line of the processor.

7. The method according to claim 2, characterized in that The second memory has a first indicator bit and a second indicator bit corresponding thereto, the first indicator bit being used to indicate an offset address corresponding to a third embedded vector in the second memory, the third embedded vector including the embedded vector corresponding to each input data in the first data group and the first embedded vector, and the second indicator bit being used to indicate an offset address corresponding to a fourth embedded vector in the second memory, the fourth embedded vector including the embedded vector corresponding to each input data in the second data group; The determining of second data that is in the first data group but not in the second data group includes: The processor determines, in the second memory, an offset address corresponding to the second data according to the first indicator bit and the second indicator bit; The second data is obtained based on the offset address corresponding to the second data.

8. An operating device for an artificial intelligence (AI) model, characterized in that: The device is applied to a host, the host includes a processor and is connected to a computing card, and the device includes: A determination module, configured to determine first data in two adjacent groups of input data for an AI model that is in a second data group but not in the first data group, wherein the second data group is trained after the first data group; Prefetch module, used to: Pre-fetching a first embedded vector corresponding to the first data into a first memory of the processor, and determining first embedded vector information corresponding to the first data; During processing of the first data group on the computing card, the first embedding vector is pre-fetched from the first memory to a second memory of the computing card according to the first embedding vector information.

9. The device according to claim 8, characterized in that The determining module is further configured to determine second data that is in the first data group but not in the second data group; The device also includes a swapping module for swapping out a second embedding vector corresponding to the second data from the second memory to the first memory after the first data group is processed on the computing card.

10. The device according to claim 8 or 9, characterized in that The first embedding vector information is an offset address of the first embedding vector in the second memory when the first data is trained on the computing card.

11. The device according to claim 10, characterized in that The host stores a lock-free queue, and the lock-free queue is used to store the idle offset address in the second memory; The prefetch module is configured to select the first embedded vector information from the lock-free queue; The prefetch module is further configured to delete the first embedded vector information from the lock-free queue after selecting the first embedded vector information.

12. The device according to any one of claims 9 to 11, characterized in that The host stores at least one hash bucket, and the at least one hash bucket is used to store input data corresponding to the embedded vector information in the input data of the AI ​​model; The determining module is configured to: Determine a hash value corresponding to each input data in the second data group; Determining, according to the hash values ​​corresponding to the respective input data, first data in the second data group that does not belong to the at least one hash bucket; The determining module is further configured to: after determining the first embedding vector information corresponding to the first data, the processor adds the first data to a corresponding hash bucket according to a hash value corresponding to the first data.

13. The device according to claim 9, characterized in that The second memory has a first indicator bit and a second indicator bit corresponding thereto, the first indicator bit being used to indicate an offset address corresponding to a third embedded vector in the second memory, the third embedded vector including the embedded vector corresponding to each input data in the first data group and the first embedded vector, and the second indicator bit being used to indicate an offset address corresponding to a fourth embedded vector in the second memory, the fourth embedded vector including the embedded vector corresponding to each input data in the second data group; The determining module is configured to: Determining, in the second memory, an offset address corresponding to the second data according to the first indicator bit and the second indicator bit; The second data is obtained based on the offset address corresponding to the second data.

14. A computer program product, characterized in that The computer program product includes computer instructions stored in a computer-readable storage medium; The processor of the computing device reads the computer instructions from the computer-readable storage medium; The processor executes the computer instructions to cause the computing device to perform the method according to any one of claims 1 to 7.

15. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions. When the computer instructions in the computer-readable storage medium are executed by a computing device, the computing device is caused to execute the method according to any one of claims 1 to 7.

Images