Data arrangement method and device and data access method and device

Abstract

The invention provides a data arrangement method and device and a data access method and device, and belongs to the technical field of information. In the scheme provided by the invention, the host can adjust the arrangement sequence of the plurality of pieces of data in the memory of the acceleration chip based on the query frequency of the plurality of pieces of data in the acceleration chip, so that the acceleration chip stores the at least two pieces of target data with relatively high query frequency into the cache. Therefore, when the host accesses the data in the acceleration chip, the acceleration chip can quickly return the data required by the host from the cache, so that the data access speed is effectively improved.

Description

Technical Field

[0001] The present application relates to the field of information technology, and in particular to a data arrangement method, a data access method and a device. Background Art

[0002] In order to improve the efficiency of model training, artificial intelligence (AI) chips (also called accelerator cards) are generally used to work with the host-side processor. The AI ​​chip can store data and parameters for model training, and the host-side processor can access the data and parameters in the AI ​​chip and perform model training.

[0003] However, when the amount of data stored in the AI ​​chip is large and the processor needs to access a large amount of data in the AI ​​chip, the data access speed is low. Summary of the invention

[0004] The present application provides a data arrangement method, a data access method and a device, which can solve the technical problem of low memory access speed of a processor to an acceleration chip in the related art.

[0005] In a first aspect, a data arrangement method is provided, which is applied to a host, the host establishes a communication connection with an acceleration chip, and a plurality of data are stored in the memory of the acceleration chip. The method includes: determining the query frequency of the plurality of data stored in the memory of the acceleration chip, and adjusting the arrangement order of the plurality of data in the memory of the acceleration chip based on the query frequency. The plurality of data after the arrangement order is adjusted is used for the acceleration chip to store at least two target data in a cache. Afterwards, based on the adjusted arrangement order of the plurality of data, the query identification (ID) of the plurality of data is updated, and the query ID is used to indicate the storage location of the data in the acceleration chip.

[0006] In the solution provided by the present application, the host can adjust the arrangement order of the multiple data in the memory of the acceleration chip based on the query frequency of the multiple data in the acceleration chip, so that the acceleration chip stores at least two target data with a higher query frequency in the cache. In this way, it can be ensured that when the host accesses the data in the acceleration chip, the acceleration chip can quickly return the data required by the host from the cache, thereby effectively improving the speed of data access.

[0007] Optionally, the process of adjusting the arrangement order of the multiple data in the memory of the acceleration chip based on the query frequency may include: adjusting the arrangement order of the multiple data in the memory of the acceleration chip so that the multiple data are arranged in sequence from high to low query frequency.

[0008] Based on this, the acceleration chip can directly store the first N data in the memory in the cache, thereby effectively improving the loading efficiency of the cache in the acceleration chip.

[0009] Optionally, the process of determining the query frequency of the plurality of data stored in the memory of the acceleration chip may include: determining the query frequency of the plurality of data stored in the memory of the acceleration chip every statistical period.

[0010] Since the host can periodically count the query frequencies of multiple data in the acceleration chip, the arrangement order of the multiple data can be periodically updated. Accordingly, the acceleration chip can timely update the data stored in the cache to ensure a high cache hit rate.

[0011] Optionally, each of the multiple pieces of data may be an embedding vector. Also, in a model training scenario, the multiple pieces of data may be the weights of the model.

[0012] Optionally, the method may further include: sending a memory access request to the acceleration chip based on the updated query IDs of the multiple pieces of data. The memory access request may carry the query ID of the data to be accessed by the host.

[0013] In a second aspect, a data access method is provided, which is applied to an acceleration chip, wherein a plurality of data are stored in a memory of the acceleration chip, and the acceleration chip has a communication connection with a host. The method comprises: after the host adjusts the arrangement order of the plurality of data in the memory based on the query frequency of the plurality of data, at least two target data among the plurality of data are stored in a cache. The query frequency of the at least two target data is higher than that of other data. Afterwards, if a memory access request sent by the host is received, the data indicated by the memory access request is queried from the cache.

[0014] In the solution provided by the present application, the acceleration chip can store at least two target data with a higher query frequency in the cache after the host adjusts the arrangement order of multiple data in the memory. Thus, it can be ensured that when the host accesses the data in the acceleration chip, the acceleration chip can quickly return the data required by the host from the cache, thereby effectively improving the speed of data access.

[0015] Optionally, the multiple data are arranged in descending order of query frequency, and the process of storing at least two target data among the multiple data into the cache may include: storing the first N target data among the multiple data into the cache, where N is an integer greater than 1.

[0016] Optionally, the method may further include: if the data indicated by the memory access request is stored in the cache, reading the data from the cache and sending it to the host; if the data indicated by the memory access request is not stored in the cache, reading the data from the memory of the acceleration chip and sending it to the host.

[0017] In a third aspect, a host is provided, the host comprising at least one module, and the at least one module can be used to implement the data arrangement method provided in the first aspect.

[0018] In a fourth aspect, an acceleration chip is provided, the acceleration chip comprising at least one module, and the at least one module can be used to implement the data access method provided in the second aspect.

[0019] In a fifth aspect, a data processing device is provided, which may be a host or an acceleration chip. The data processing device comprises: a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, the method provided in any of the above aspects is implemented.

[0020] In a sixth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium. When the instructions are executed on a processor, the processor executes the method provided in any of the above aspects.

[0021] In a seventh aspect, a computer program product comprising instructions is provided, wherein the instructions are executed by a processor to implement the method provided in any of the above aspects.

[0022] In an eighth aspect, an electronic device is provided, which includes a host and an acceleration chip. The host can be used to implement the data arrangement method provided in the first aspect, and the acceleration chip can be used to implement the data access method provided in the second aspect.

[0023] In summary, the present application provides a data arrangement method, a data access method and a device. In the solution provided by the present application, the host can adjust the arrangement order of the multiple data in the memory of the acceleration chip based on the query frequency of the multiple data in the acceleration chip, so that the acceleration chip stores at least two target data with a higher query frequency in the cache. In this way, it can be ensured that when the host accesses the data in the acceleration chip, the acceleration chip can quickly return the data required by the host from the cache, thereby effectively improving the speed of data access. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a schematic diagram of an implementation environment involved in the data arrangement method and data access method provided in the embodiments of the present application;

[0025] Figure 2 is a flow chart of a data arrangement method and a data access method provided in an embodiment of the present application;

[0026] Figure 3 It is a schematic diagram of a host adjusting the arrangement order of multiple data provided by an embodiment of the present application;

[0027] Figure 4 It is a structural schematic diagram of a data arrangement device provided in an embodiment of the present application;

[0028] Figure 5 is a schematic diagram of the structure of an acceleration chip provided in an embodiment of the present application;

[0029] Figure 6 It is a structural diagram of a data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The data arrangement method, data access method and device provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0031] Figure 1 is a schematic diagram of an electronic device provided in an embodiment of the present application, and the electronic device may be a server, for example, a parameter server (PS). Figure 1 As shown, the electronic device may include a host 01 and an acceleration chip 02. The host 01 and the acceleration chip 02 may be connected via a bus or a network. The bus may be a peripheral component interconnect express (PCIe) bus, a compute express link (CXL) bus, a universal serial bus (USB) or a quick path interconnect (QPI) bus, etc. The network between the host 01 and the acceleration chip 02 may be a wired network (such as a network cable) or a wireless network, etc. Furthermore, the acceleration chip 02 may be inserted into a card slot of the host 01, or may be set independently of the host 01, and this embodiment of the present application does not limit this.

[0032] Continue to refer Figure 1The host 01 may include a processor 011, which may be a central processing unit (CPU). The acceleration chip 02 is also called an acceleration card, which includes a memory 021 and a cache 022. The memory 021 may be a high bandwidth memory (HBM), and the cache 022 may be a L2 cache.

[0033] In the embodiment of the present application, the acceleration chip 02 may be a chip dedicated to processing tasks in a specific field. For example, the acceleration chip 02 may be dedicated to processing artificial intelligence (AI) tasks, and accordingly, the acceleration chip 02 may also be referred to as an AI chip. For example, the acceleration chip 02 may be a neural-network processing unit (NPU) or a graphics processing unit (GPU).

[0034] In the process of processing tasks in a specific field, such as in the model training process, the processor 011 in the host 01 needs to perform a large number of memory access operations on the data stored in the memory 021 in the acceleration chip 02. For example, in the training process of a sparse model, the processor 011 can perform sparse memory access on the acceleration chip 02. Among them, sparse memory access can refer to: using a gather sparse read operator and a uss sparse write operator, etc., to read and write non-continuous data to the acceleration chip 02 through address offset. Among them, uss refers to unsorted_segment_sum.

[0035] In order to improve the speed of data access, the acceleration chip 02 usually adopts a cache acceleration solution. For example, the acceleration chip 02 can count the query frequency of the processor 011 for each data (such as embedded vector) in the memory 021 through scalar class calculation. After that, the acceleration chip 02 can put the data with higher query frequency into the cache 022. When the acceleration chip 02 receives the memory access request from the processor 011 again, it can first detect whether the accessed data is cached through Scalar class calculation. If it is cached, the cache 022 is accessed, and if it is not cached, the memory 021 is accessed.

[0036] However, due to the limited computing power of scalar computing in the acceleration chip 02, the performance of the sparse non-continuous small HBM access of the memory transfer engine (MTE) is poor, that is, the above cache acceleration solution has limited effect on improving the memory access speed. Among them, MTE is the memory access module in the acceleration chip 02.

[0037] Optionally, the solution provided in the embodiment of the present application can be applied to big data scenarios or model training scenarios. Among them, model training is a key component of systems such as recommendation, search, advertising and marketing. Model training can produce high-quality models for online reasoning and sorting to support business applications. Since the larger the model, the better the effect, the scale of models currently used in business applications is getting larger and larger. However, the huge number of parameters of large models causes memory access and network communication to become system bottlenecks.

[0038] The present application provides a data arrangement method and a data access method, which can effectively improve the speed at which a processor accesses data in an acceleration chip. The method can be applied to, for example, Figure 1 In the implementation environment shown. Figure 2 As shown, the method includes:

[0039] Step 101: The host determines the query frequency of multiple data stored in the memory of the acceleration chip.

[0040] In the embodiments of the present application, Figure 3 As shown, the acceleration chip may include a memory, which may be a high bandwidth memory (HBM). The memory may store multiple pieces of data for access by the host. Each piece of data may be an embedding (emb) vector. The host may determine the query frequency of the multiple pieces of data stored in the memory of the acceleration chip every statistical period. For example, in a model training scenario, the host may count the query frequency of the multiple pieces of data every N trainings, where N is an integer greater than 1, for example, N may be 2.

[0041] The query frequency of each piece of data may refer to the number of queries, i.e., the number of accesses, of the host to the piece of data within the query period. Both the statistical period and the query period may be pre-set in the host, and the query period may be equal to the statistical period. Since the host may periodically count the query frequencies of multiple pieces of data in the acceleration chip, the arrangement order of the multiple pieces of data may be periodically updated. Accordingly, the acceleration chip may timely update the data stored in the cache to ensure a high cache hit rate.

[0042] Optionally, in the scenario of model training, the multiple pieces of data stored in the memory may be model weights. During the model training phase, the host may read and parse the model samples and count the query frequencies of each model weight stored in the memory of the acceleration chip.

[0043] For example, Figure 3As shown, assuming that a total of X embedded vectors from emb0 to embX are stored in the HBM of the acceleration chip, the host can count the query frequency of each embedded vector separately. Figure 3 The query ID sequence shown in shows the query IDs of the embedding vectors most recently accessed by the host. Referring to the sequence, it can be seen that the host accessed emb3 3 times, emb2 2 times, and emb0, emb1, emb4, emb5 and emb7 once respectively during the query cycle.

[0044] Step 102: The host adjusts the arrangement order of the plurality of data in the memory of the acceleration chip based on the query frequency.

[0045] After determining the query frequency of each piece of data in the acceleration chip, the host can adjust the arrangement order of the multiple pieces of data in the memory of the acceleration chip according to the query frequency, so that the multiple pieces of data are arranged in order of high and low query frequencies.

[0046] Optionally, in an embodiment of the present application, the host may adjust the arrangement order of the plurality of data in the memory of the acceleration chip so that the plurality of data are arranged in descending order according to the query frequency. Figure 3 Since the query frequency of emb0 to emb3 satisfies: emb3>emb2>emb1=emb0, the host can adjust the arrangement order of emb0 to emb3, and arrange the four embedding vectors in the order of emb3, emb2, emb1, emb0.

[0047] It is understandable that the host may also arrange the multiple pieces of data in other orders, for example, in order from low to high query frequency.

[0048] Step 103: The acceleration chip stores at least two target data among the multiple data in a cache.

[0049] In the embodiment of the present application, after the host adjusts the arrangement order of the multiple data, the acceleration chip can store at least two target data among the multiple data in the cache. Moreover, the query frequency of the at least two target data stored in the cache is higher than that of other data. That is, the at least two target data can be the at least two data with the highest query frequency among the multiple data.

[0050] Assuming that after the host adjusts the arrangement order of the multiple data, the multiple data are arranged in order from high to low query frequency, then the at least two target data may be the first N data, where N is an integer greater than 1. Accordingly, in step 103, the acceleration chip can directly store the first N target data in its memory into the cache. Since the acceleration chip no longer needs to count the query frequency of each data, but can directly store the data at a specific location in the memory (such as the data in the front continuous space of the memory) into the cache, the loading efficiency of the cache is effectively improved.

[0051] For example, Figure 3 As shown, assuming that N is 2, the acceleration chip can store the first two embs (ie, emb2 and emb3) in the HBM into the cache.

[0052] It is understandable that the above N can be determined based on the cache size of the acceleration chip and the size of a single piece of data. For example, N can be the upper limit of the number of data that can be stored in the cache of the acceleration chip.

[0053] Step 104: The host updates the query IDs of the plurality of data based on the adjusted arrangement order of the plurality of data.

[0054] It is understandable that each piece of data in the acceleration chip corresponds to a query ID, which is used to indicate the storage location of the data in the acceleration chip. The query ID can also be called an arrangement offset or a query offset, which can indicate the offset position of the data in the memory of the acceleration chip (i.e., the offset relative to the first address).

[0055] Example, reference Figure 3 , because the host adjusts the arrangement order of emb0 to emb3, the arrangement offsets of the four embedded vectors need to be updated accordingly. As shown in Table 1, the original arrangement offset of emb0 is 0, and the updated arrangement offset is 3, the original arrangement offset of emb1 is 1, and the updated arrangement offset is 2, the original arrangement offset of emb2 is 2, and the updated arrangement offset is 1, and the original arrangement offset of emb3 is 3, and the updated arrangement offset is 0.

[0056] Table 1

[0057]

[0058] Step 105: The host sends a memory access request to the acceleration chip based on the updated query IDs of the multiple data pieces.

[0059] In an embodiment of the present application, when the host needs to access data in the acceleration chip, it can send a memory access request to the acceleration chip based on the updated query ID of the multiple data. The memory access request carries the query ID of the data to be accessed. For example, assuming that the host needs to access emb2, a memory access request carrying query ID: 1 can be sent to the acceleration chip.

[0060] Step 106: The acceleration chip queries the cache for data indicated by the memory access request based on the memory access request sent by the host.

[0061] After receiving the memory access request sent by the host, the acceleration chip can first query whether the cache stores the data indicated by the memory access request. For example, assuming that the memory access request sent by the host is used to indicate access to emb2, the acceleration chip can query whether the cache stores emb2.

[0062] Step 107a: If the data indicated by the memory access request is stored in the cache, the acceleration chip reads the data from the cache and sends it to the host.

[0063] If the data indicated by the memory access request is stored in the cache of the acceleration chip (ie, a cache hit), the acceleration chip can directly read the data from the cache and send it to the host.

[0064] Example, reference Figure 3 , assuming that the memory access request sent by the host is used to indicate access to emb2, since emb2 is stored in the cache, the acceleration chip can directly read emb2 from the cache and send it to the host.

[0065] Step 107b: If the data indicated by the memory access request is not stored in the cache, the acceleration chip reads the data from its memory and sends it to the host.

[0066] If the data indicated by the memory access request is not stored in the cache (ie, a cache miss), the acceleration chip may read the data from the memory and send it to the host.

[0067] Example, reference Figure 3 , assuming that the memory access request sent by the host is used to indicate access to emb0, since emb0 is not stored in the cache, the acceleration chip needs to read emb0 from the HBM and send it to the host.

[0068] It is understandable that in the embodiment of the present application, since the memory access address of the acceleration chip is parsed and determined by the host, the host (such as the CPU in the host) can be used to calculate the data with high query frequency (i.e., cache hotspots), and the data arrangement in the acceleration chip can be optimized based on the query frequency. Therefore, the acceleration chip does not need to use scalar calculation to determine the data to be cached. The acceleration chip only needs to store at least two target data with high query frequency into the cache based on the rearranged data.

[0069] Based on the above method embodiments, it can be known that since the scalar computing power of the acceleration chip is usually weak, the embodiment of the present application can count the query frequency of the queried data through the host (such as the processor in the host), and adjust the arrangement of the data in the acceleration chip based on the query hotspot. As a result, the query hotspot can be arranged at a specific location in the memory of the acceleration chip (for example, arranged in the continuous space at the front of the memory), and the acceleration chip can directly cache the data at the specific location without performing scalar type calculations, thereby effectively improving the loading efficiency of the cache in the acceleration chip.

[0070] It is understandable that the method provided in the embodiment of the present application can be used in the sparse memory access operator of model training or reasoning, for example, it can be applied to the gather sparse read operator or the uss sparse write operator. Thus, a large number of repeated accesses of the sparse memory access operator can be optimized, and the performance of the bottleneck operator can be greatly improved in a basically overhead-free manner. For example, after testing, the method provided in the embodiment of the present application can improve the performance of the sparse transport operator by 15%+.

[0071] It is also understandable that the order of the steps of the method provided in the embodiment of the present application can be adjusted appropriately, and the steps can be increased or decreased accordingly. For example, the above step 104 can be performed before step 103.

[0072] In summary, the embodiments of the present application provide a data arrangement method and a data access method. In the above method, the host can adjust the arrangement order of the multiple data in the memory of the acceleration chip based on the query frequency of the multiple data in the acceleration chip, so that the acceleration chip stores at least two target data with a higher query frequency in the cache. Thus, it can be ensured that when the host accesses the data in the acceleration chip, the acceleration chip can quickly return the data required by the host from the cache, thereby effectively improving the speed of data access. In addition, since the method provided by the embodiment of the present application does not require the acceleration chip to count the query frequency of each data, it can effectively improve the loading efficiency of the cache in the acceleration chip, and can avoid the problem of limited computing power of the acceleration chip resulting in poor effect on improving the access speed.

[0073] Figure 4is a schematic diagram of a data arrangement device provided in an embodiment of the present application, which can be applied to Figure 1 The host 01 in the application scenario shown, for example, can be applied to the processor 011 of the host 01. For example, the data arrangement device can be a software development kit (SDK) including sparse model training. Figure 1 The host 01 has established a communication connection with the acceleration chip 02, and the memory 021 of the acceleration chip 02 stores a plurality of data. Figure 4 As shown, the data arrangement device includes:

[0074] The determination module 201 is used to determine the query frequency of the plurality of data stored in the memory of the acceleration chip. The function of the determination module 201 can be realized by referring to the relevant description of step 101 in the above method embodiment.

[0075] The adjustment module 202 is used to adjust the arrangement order of the multiple data in the memory of the acceleration chip based on the query frequency, and the multiple data after the adjustment of the arrangement order are used for the acceleration chip to store at least two target data in the cache. The function of the adjustment module 202 can refer to the relevant description of step 102 in the above method embodiment.

[0076] The updating module 203 is used to update the query identification ID of the plurality of data based on the adjusted arrangement order of the plurality of data, and the query ID is used to indicate the storage location of the data in the acceleration chip. The function of the updating module 203 can be realized by referring to the relevant description of step 104 in the above method embodiment.

[0077] Optionally, the adjustment module 202 may be used to adjust the arrangement order of the plurality of data in the memory of the acceleration chip so that the plurality of data are arranged in order from high to low according to the query frequency.

[0078] Optionally, the determination module 201 may be configured to determine, at every statistical period, a query frequency of the plurality of pieces of data stored in the memory of the acceleration chip.

[0079] Optionally, each of the multiple pieces of data may be an embedding vector.

[0080] Optionally, continue to refer to Figure 4 , the device may also include:

[0081] The sending module 204 is used to send a memory access request to the acceleration chip based on the updated query IDs of the multiple pieces of data. The function of the sending module 204 can be realized by referring to the relevant description of step 105 in the above method embodiment.

[0082] In summary, the embodiment of the present application provides a data arrangement device, which can adjust the arrangement order of the multiple data in the memory of the acceleration chip based on the query frequency of the multiple data in the acceleration chip, so that the acceleration chip stores at least two target data with a higher query frequency in the cache. In this way, it can be ensured that when the host accesses the data in the acceleration chip, the acceleration chip can quickly return the data required by the host from the cache, thereby effectively improving the speed of data access. In addition, since the data arrangement device provided in the embodiment of the present application does not require the acceleration chip to count the query frequency of each data, it can avoid the problem of limited computing power of the acceleration chip resulting in poor effect on improving the access speed.

[0083] Figure 5 is a schematic diagram of the structure of an acceleration chip provided in an embodiment of the present application, and the acceleration chip can be applied to Figure 1 The application scenario shown in Figure 1. Figure 1 As shown, the acceleration chip 02 establishes a communication connection with the host 01. In addition, the memory 021 of the acceleration chip 02 stores a plurality of data.

[0084] like Figure 5 As shown, the acceleration chip includes:

[0085] The processing module 301 is used to store at least two target data of the multiple data in the cache after the host adjusts the arrangement order of the multiple data in the memory based on the query frequency of the multiple data. The query frequency of the at least two target data is higher than that of other data. The functional implementation of the processing module 301 can refer to the relevant description of step 103 in the above method embodiment.

[0086] The query module 302 is used to query the data indicated by the memory access request from the cache if the memory access request sent by the host is received. The function implementation of the query module 302 can refer to the relevant description of step 106 in the above method embodiment.

[0087] Optionally, the multiple pieces of data may be arranged in descending order of query frequency, and the processing module 301 may be used to store the first N pieces of target data among the multiple pieces of data in a cache, where N is an integer greater than 1.

[0088] Alternatively, if Figure 5 As shown, the acceleration chip may further include a sending module 303. The sending module 303 is used to: if the data indicated by the memory access request is stored in the cache, read the data from the cache and send it to the host; if the data indicated by the memory access request is not stored in the cache, read the data from the memory of the acceleration chip and send it to the host.

[0089] The function implementation of the sending module 303 may refer to the related description of step 107a and step 107b in the above method embodiment.

[0090] In summary, the embodiment of the present application provides an acceleration chip, which can store at least two target data with a higher query frequency in the cache after the host adjusts the arrangement order of multiple data in the memory. Thereby, it can be ensured that when the host accesses the data in the acceleration chip, the acceleration chip can quickly return the data required by the host from the cache, thereby effectively improving the speed of data access. In addition, since the acceleration chip does not need to count the query frequency of each data, it can avoid the problem of limited computing power of the acceleration chip resulting in poor improvement of memory access speed.

[0091] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the host, acceleration chip and each module described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0092] It should be understood that the data arrangement device and acceleration chip provided in the embodiments of the present application can also be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), and the PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof. In addition, the method provided in the above method embodiment can also be implemented by software. When the method provided in the above method embodiment is implemented by software, each functional module in the data arrangement device and the acceleration chip can also be a software module.

[0093] Figure 6 is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. The data processing device can be Figure 1 The host 01 in the application scenario shown in the figure can implement the steps performed by the host 01 in the above method embodiment. Alternatively, the data processing device can be Figure 1 The acceleration chip 02 in the application scenario shown can implement the steps performed by the acceleration chip 02 in the above method embodiment. Figure 6 The data processing device includes: a processor 401, a memory 402, a network interface 403 and a bus 404.

[0094] The memory 402 stores a computer program 4021, which is used to implement various application functions. The processor 401 is used to execute the computer program 4021 to implement the data arrangement method or data access method provided by the above method embodiment. For example, the processor 401 can be used to execute the computer program 4021 to implement the above method embodiment. Figure 4 or Figure 5 The functions of each module are shown.

[0095] The processor 401 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), ASIC, FPGA, graphics processing unit (GPU), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0096] The memory 402 may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories. Among them, the nonvolatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DRRAM).

[0097] There may be multiple network interfaces 403, and the network interfaces 403 are used to achieve communication connection (which may be wired or wireless) with other devices, wherein the other devices may be devices such as terminals, servers or VMs.

[0098] The bus 404 is used to connect the processor 401, the memory 402 and the network interface 403. In addition to the data bus, the bus 404 may also include a power bus, a control bus and a status signal bus, etc. However, for the sake of clarity, various buses are labeled as bus 404 in the figure.

[0099] An embodiment of the present application also provides a computer-readable storage medium, which stores instructions. When the instructions are executed by a processor, the processor executes the steps performed by the host or the acceleration chip in the above method embodiment.

[0100] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a processor, enables the processor to execute the steps executed by a host or an acceleration chip in the above method embodiment.

[0101] In the embodiments of the present application, the terms "first", "second" and "third" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. The term "at least one" means one or more, and "plurality" means two or more.

[0102] The above is only an optional implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A data arrangement method, characterized in that: Applied to a host, the host establishes a communication connection with an acceleration chip, and a memory of the acceleration chip stores multiple data; the method includes: Determining a query frequency of the plurality of pieces of data stored in a memory of the acceleration chip; Adjusting the arrangement order of the plurality of data in the memory of the acceleration chip based on the query frequency, and the plurality of data after the adjustment of the arrangement order are used for the acceleration chip to store at least two target data in a cache; Based on the adjusted arrangement order of the plurality of data pieces, query identification IDs of the plurality of data pieces are updated, where the query IDs are used to indicate storage locations of the data in the acceleration chip.

2. The method according to claim 1, characterized in that The adjusting the arrangement order of the plurality of data in the memory of the acceleration chip based on the query frequency includes: Adjust the arrangement order of the multiple pieces of data in the memory of the acceleration chip so that the multiple pieces of data are arranged in order from high to low according to the query frequency.

3. The method according to claim 1 or 2, characterized in that: The determining the query frequency of the plurality of pieces of data stored in the memory of the acceleration chip includes: At every statistical period, a query frequency of the plurality of pieces of data stored in the memory of the acceleration chip is determined.

4. The method according to any one of claims 1 to 3, characterized in that: Each piece of data in the plurality of pieces of data is an embedding vector.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Based on the updated query IDs of the plurality of pieces of data, a memory access request is sent to the acceleration chip.

6. A data access method, characterized in that: Applied to an acceleration chip, a memory of the acceleration chip stores multiple pieces of data, and the acceleration chip establishes a communication connection with a host, the method comprising: After the host adjusts the arrangement order of the plurality of data in the memory based on the query frequency of the plurality of data, at least two target data among the plurality of data are stored in the cache, and the query frequency of the at least two target data is higher than that of other data; If a memory access request sent by the host is received, the data indicated by the memory access request is queried from the cache.

7. The method according to claim 6, characterized in that The plurality of data are arranged in descending order of query frequency, and the storing at least two target data among the plurality of data in the cache includes: The first N target data of the plurality of data are stored in a cache, where N is an integer greater than 1.

8. The method according to claim 6 or 7, characterized in that: The method further comprises: If the data indicated by the memory access request is stored in the cache, reading the data from the cache and sending it to the host; If the data indicated by the memory access request is not stored in the cache, the data is read from the memory of the acceleration chip and sent to the host.

9. A data arrangement device, characterized in that: Applied to a host, the host establishes a communication connection with an acceleration chip, and a memory of the acceleration chip stores multiple data; the device includes: A determination module, used to determine the query frequency of the plurality of data stored in the memory of the acceleration chip; an adjustment module, configured to adjust the arrangement order of the plurality of data in the memory of the acceleration chip based on the query frequency, wherein the plurality of data after the adjustment of the arrangement order is used for the acceleration chip to store at least two target data in a cache; An updating module is used to update the query identification ID of the plurality of data based on the adjusted arrangement order of the plurality of data, wherein the query ID is used to indicate the storage location of the data in the acceleration chip.

10. The device according to claim 9, characterized in that The adjustment module is used for: Adjust the arrangement order of the multiple pieces of data in the memory of the acceleration chip so that the multiple pieces of data are arranged in order from high to low according to the query frequency.

11. The device according to claim 9 or 10, characterized in that The determining module is used to: At every statistical period, a query frequency of the plurality of pieces of data stored in the memory of the acceleration chip is determined.

12. The device according to any one of claims 9 to 11, characterized in that: Each piece of data in the plurality of pieces of data is an embedding vector.

13. The device according to any one of claims 9 to 12, characterized in that: The device also includes: The sending module is used to send a memory access request to the acceleration chip based on the updated query IDs of the multiple data.

14. An acceleration chip, characterized in that: The memory of the acceleration chip stores a plurality of data, and the acceleration chip establishes a communication connection with the host, and the acceleration chip includes: A processing module, configured to store at least two target data of the plurality of data in a cache after the host adjusts the arrangement order of the plurality of data in the memory based on the query frequency of the plurality of data, wherein the query frequency of the at least two target data is higher than that of other data; The query module is used to query the data indicated by the memory access request from the cache if a memory access request sent by the host is received.

15. The acceleration chip according to claim 14, characterized in that: The plurality of data are arranged in descending order of query frequency, and the processing module is used to: The first N target data of the plurality of data are stored in a cache, where N is an integer greater than 1.

16. The acceleration chip according to claim 14 or 15, characterized in that: The acceleration chip further includes a sending module, and the sending module is used to: If the data indicated by the memory access request is stored in the cache, reading the data from the cache and sending it to the host; If the data indicated by the memory access request is not stored in the cache, the data is read from the memory of the acceleration chip and sent to the host.

17. A data processing device, characterized in that: The device comprises: a memory, a processor and a computer program stored in the memory and capable of running on the processor, and the processor implements the method according to any one of claims 1 to 8 when executing the computer program.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a processor, the processor is enabled to execute the method according to any one of claims 1 to 8.

19. A computer program product comprising instructions, characterized in that The instructions are executed by a processor to implement the method according to any one of claims 1 to 8.

20. An electronic device, characterized in that: The electronic device comprises: a host and an acceleration chip, wherein the host is used to implement the method according to any one of claims 1 to 5, and the acceleration chip is used to implement the method according to any one of claims 6 to 8.

Images