Cache updating method and device, equipment and medium

Abstract

The invention provides a cache updating method and device, equipment and a medium, and the method comprises the steps: determining a first tag word address corresponding to each first input text when context reasoning is carried out for a plurality of first input texts at the same time; obtaining a first input text index corresponding to each first input text; binding the first tag word address and the first input text index of the same first input text to obtain a cache index lookup table; and updating a cache value in a preset text type cache according to the change state corresponding to the preset flag bit of each first input text in the context reasoning and the update state of the cache index lookup table. In the embodiment of the invention, the text-type cache value in the text-type cache is designed as the global variable in the inference model, and the cache value of the text-type cache is dynamically updated and managed by setting the plurality of flag bits and the cache index lookup table, so that waste and overflow of a video memory space can be avoided.

Description

Technical Field

[0001] The present invention relates to the field of cache technology, and in particular to a cache update method, device, equipment and medium. Background Art

[0002] With the development of artificial intelligence technology, more and more large language models (LLMs) have emerged and are playing a role in production and life. vLLM is an open source large model inference acceleration framework. It improves the throughput of model inference by efficiently managing the key-value cache tensor of the attention mechanism. It has excellent performance in the scenario of high-concurrency inference service and currently supports multiple large language models.

[0003] In order to reduce the amount of computation of large language models, vLLM saves the key vectors and numerical vectors corresponding to the already calculated tag words in the cache unit, and logically blocks the tag words, and maps the logical blocks to physical blocks through mapping relationships, so as to save video memory and improve throughput. However, this cache method only supports key-value cache. With the development of large language models, the model may need to cache more calculation results. In addition to caching the key vectors and numerical vectors of the local filter attention layer, the model also needs to cache the output tensor of the local filter attention layer.

[0004] However, the existing vLLM-based cache management for input text integrates the local filter cache into the vLLM source code, but the native code is modified significantly, which is not conducive to the subsequent version iteration and upgrade of the vLLM source code. Moreover, this violent modification of the vLLM native sentence class method will have a greater impact on the model without text cache, and generate many unnecessary null values ​​in the cache unit. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a cache update method, device, equipment and medium. The specific technical solution is as follows:

[0006] In a first aspect of the present invention, a cache update method is provided, the method comprising:

[0007] When context reasoning is performed on a plurality of first input texts at the same time, determining a first marked word address corresponding to each of the first input texts;

[0008] Obtain a first input text index corresponding to each first input text;

[0009] Binding the first markup word address of the same first input text with the first input text index to obtain a cache index lookup table;

[0010] The cache value in the preset text-type cache is updated according to the change status corresponding to the flag bits preset in the context reasoning of each of the first input texts and the update status of the cache index lookup table.

[0011] Optionally, determining the first marked word address corresponding to each first input text includes:

[0012] By obtaining the key-value cache address corresponding to each input text in contextual reasoning in a preset vectorized large language framework, the first marker word address corresponding to each input text is determined.

[0013] Optionally, the pre-set flag bit includes an initialization flag bit and an update flag bit;

[0014] The updating of the cache value in the pre-set text-based cache by using the change state corresponding to the flag bits pre-set in the context reasoning of each of the first input texts and the update state of the cache index lookup table includes:

[0015] When the first mark word address is a first preset value, the update flag bit and the initialization flag position corresponding to the first input text are initial values, and the cache index lookup table is set to a null value, the cache value corresponding to the first input text in the preset text type cache is updated to be a zero vector;

[0016] When a plurality of initial marker words generated by the conversion of the first input text are predicted to obtain a first target marker word, the update flag bit and the initialization flag position corresponding to the first input text are set to initial values, the cache index lookup table is set to a null value, and the cache value corresponding to the first input text in the pre-set text-type cache is updated according to the initial value and the null value to be a zero vector;

[0017] When prediction is performed for the first target tag word and the plurality of initial tag words, the cache value in the pre-set text-type cache is updated by the target cache value generated by the first target tag word and the plurality of initial tag words in context reasoning.

[0018] Optionally, before the step of determining the first marked word address corresponding to each of the first input texts when contextual reasoning is performed on a plurality of first input texts at the same time, the method includes:

[0019] When the number of all the first input texts corresponding to the first input texts is greater than a preset threshold, determining an input text queue;

[0020] After the step of updating the cache value in the preset text-based cache according to the change state corresponding to the flag bit preset in the context reasoning of each of the first input texts and the update state of the cache index lookup table, the method includes:

[0021] When there is a second input text in a waiting-for-reasoning state in the input text queue, acquiring the first input text in an incomplete-reasoning state;

[0022] Performing contextual reasoning on the first input text and the second input text that are in an incomplete reasoning state at the same time, and setting a cache value corresponding to the second input text to a zero vector;

[0023] Determine a second marked word address corresponding to each of the second input texts;

[0024] Obtain a second input text index corresponding to each second input text;

[0025] The second markup word address and the second input text index of the same second input text are bound in the cache index lookup table.

[0026] Optionally, setting a binding relationship between a second markup word address corresponding to the second input text and the second input text in the cache index lookup table includes:

[0027] If the second mark word address corresponding to the second input text is the same as the first mark word address, then the second input text index in the cache index lookup table covers the first input text index.

[0028] Optionally, after the step of updating the cache value in the preset text-based cache according to the change state corresponding to the flag bit preset in the context reasoning of each of the first input texts and the update state of the cache index lookup table, the method includes:

[0029] The cache value in the updated preset text-type cache is read through a preset reasoning model, and the first input text and / or the second input text are reasoned to obtain the reasoning result corresponding to the first input text and / or the second input text.

[0030] Optionally, after the step of reading the updated cache value in the preset text-type cache through the preset reasoning model, reasoning the first input text, and obtaining the reasoning result corresponding to the first input text, the method includes:

[0031] Determine the target input text index in the inference state in the cache index lookup table;

[0032] Determine the first input text and / or the second input text in a reasoning completion state through the target input text index;

[0033] The first input text and / or the second input text in the inference completed state is deleted from the preset text type cache, and the cache value in the preset text type cache is updated.

[0034] In a second aspect of the present invention, a cache management device is provided, the device comprising:

[0035] A determination module, configured to determine a first marked word address corresponding to each of the first input texts when contextual reasoning is performed on a plurality of first input texts at the same time;

[0036] An acquisition module, used for acquiring a first input text index corresponding to each first input text;

[0037] A binding module, used for binding the first markup word address of the same first input text with the first input text index to obtain a cache index lookup table;

[0038] An updating module is used to update the cache value in the preset text-type cache according to the change status corresponding to the flag bits preset in the context reasoning of each of the first input texts and the update status of the cache index lookup table.

[0039] In a third aspect of the implementation of the present invention, there is also provided a communication device, comprising: a transceiver, a memory, a processor, and a program stored in the memory and executable on the processor;

[0040] The processor is used to read the program in the memory to implement the cache update method as described in any one of the first aspects.

[0041] In a fourth aspect of the implementation of the present invention, a readable storage medium is further provided, and when the program is executed by a processor, the cache update method as described in any one of the first aspects is implemented.

[0042] The cache update method provided in the embodiment of the present invention includes: when contextual reasoning is performed on several first input texts at the same time, determining the first marker word address corresponding to each first input text; obtaining the first input text index corresponding to each first input text; binding the first marker word address of the same first input text with the first input text index to obtain a cache index lookup table; updating the cache value in the pre-set text-type cache through the change state corresponding to the pre-set flag bit of each first input text in the contextual reasoning and the update state of the cache index lookup table. In the embodiment of the present invention, by designing the text-type cache value in the text-type cache as a global variable in the reasoning model, and by setting multiple flag bits and a cache index lookup table to realize dynamic update management of the cache value of the text-type cache, waste and overflow of video memory space can be avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0044] Figure 1 A flowchart of a cache update method provided by an embodiment of the present invention;

[0045] Figure 2 is a device block diagram of a cache management device provided by an embodiment of the present invention;

[0046] Figure 3 is a schematic diagram of a communication device provided by an embodiment of the present invention;

[0047] Figure 4 It is an exemplary text-type cache diagram provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] To make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in the embodiments of the present invention, many technical details are proposed in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can be implemented. The division of the following embodiments is for the convenience of description, and should not constitute any limitation on the specific implementation of the present invention. The various embodiments can be combined and referenced with each other without contradiction.

[0049] It should be noted that in the embodiment of the present application, the cache strategy of the large language model can be divided into two categories. One is to cache the operation results of the markup words, such as the common key vector and numerical vector cache; the other is to cache the operation results of the input text, such as the local filter cache of the source 2.0 model. At present, vLLM only handles the markup word type cache and lacks a processing strategy for the input text type cache. The text type cache is bound to the input text, but the maximum number of text parallelism allowed by vLLM when performing high-concurrency reasoning services is limited. At the same time, the number of inferred texts cannot exceed the threshold. The uninferred text must wait until the current inference text ends before it can be added to the inference queue. The strategy is controlled by the vLLM framework. During the model operation, it is only responsible for calculation and cannot determine the current inference text sequence number through effective information. Therefore, it is difficult to find the cache data corresponding to the current text.

[0050] Therefore, in order to solve the above problems, the embodiment of the present application proposes a new cache strategy based on the vLLM framework, which designs global variables to save input text cache values, sets multiple flags and lookup tables to calculate the cache values ​​corresponding to the input text currently being inferred, completes the calculation and update of the cache values, and deletes the cache of the ended text at the same time, ensuring that the length of the text cache value is equal to the number of texts currently being processed, thereby avoiding waste of video memory.

[0051] Reference Figure 1 , shows a flow chart of the steps of a cache update method provided by an embodiment of the present invention, the method may include:

[0052] Step 101, when contextual reasoning is performed on a plurality of first input texts at the same time, determining a first marked word address corresponding to each of the first input texts;

[0053] Further, the determining the first markup word address corresponding to each first input text includes:

[0054] By obtaining the key-value cache address corresponding to each input text in contextual reasoning in a preset vectorized large language framework, the first marker word address corresponding to each input text is determined.

[0055] Step 102, obtaining a first input text index corresponding to each of the first input texts;

[0056] Step 103, binding the first markup word address of the same first input text with the first input text index to obtain a cache index lookup table;

[0057] It should be noted that in the embodiment of the present application, in the above steps 101-102, by reading the key value cache address in the vLLM framework, the first marker word address of each input text is bound to the input text index, thereby constructing a lookup table structure.

[0058] Specifically, when contextual reasoning is performed on several first input texts at the same time, that is, multiple different first input texts are reasoned in parallel. For example, there are a total of 4 different input texts, and reasoning and prediction need to be performed on the input texts. Based on the threshold limit of the input text and the limitation of the reasoning model (vLLM has a limit on the number of input texts that can be reasoned at the same time when performing high-concurrency reasoning services. If the number of input texts exceeds the threshold, the texts exceeding the threshold number must wait until the previous text calculation is completed before they can be added to the calculation queue), 2 input texts can be reasoned in parallel at a time.

[0059] Therefore, when contextual reasoning is performed on several first input texts at the same time, the first marker word address corresponding to each first input text can be determined, where the first marker word address is the first marker word address of each first input text, and the second marker word address introduced subsequently is the same, that is, the first marker word address of each second input text.

[0060] Among them, the tag word is a token. During the model reasoning process, the first input text can be converted into multiple initial tag words, and then input into the reasoning model for calculation based on the multiple initial tag words, and the next tag word is predicted. The predicted tag word is then calculated and the next tag word is inferred. When the terminator is inferred, the answer to the input text is completed.

[0061] Input Text Index Length refers to the length of the index sequence that the model converts when processing input text. Usually, the input text is tokenized into multiple tokens, each of which corresponds to an index value. The index length is the number of these tokens.

[0062] For example, for an input text "Hello, how are you?", it may be converted into the following token sequence after word segmentation: ["Hello",",","how","are","you","?"]

[0063] If each token corresponds to an index value, then the index length of this input text is 6.

[0064] Therefore, after obtaining the first marked word address and the first input text index of each first input text, the first marked word address and the first input text index of the same first input text may be bound to obtain a cache index lookup table.

[0065] Step 104: Update the cache value in the preset text-based cache according to the change status corresponding to the flag bits preset in the contextual reasoning of each of the first input texts and the update status of the cache index lookup table.

[0066] Furthermore, the pre-set flag bit includes an initialization flag bit and an update flag bit;

[0067] The updating of the cache value in the pre-set text-based cache by using the change state corresponding to the flag bits pre-set in the context reasoning of each of the first input texts and the update state of the cache index lookup table includes:

[0068] When the first mark word address is a first preset value, the update flag bit and the initialization flag position corresponding to the first input text are initial values, and the cache index lookup table is set to a null value, the cache value corresponding to the first input text in the preset text type cache is updated to be a zero vector;

[0069] It should be noted that, in the embodiment of the present application, the inference service of vLLM can be divided into three parts: initialization stage, pre-filling stage and decoding stage.

[0070] When the first marker word address is a first preset value, the update flag and the initialization flag position corresponding to the first input text are initial values, and the cache index lookup table is set to an empty value, this corresponds to the initialization phase, which is mainly used for the calculation and allocation of video memory.

[0071] When a plurality of initial marker words generated by the conversion of the first input text are predicted to obtain a first target marker word, the update flag bit and the initialization flag position corresponding to the first input text are set to initial values, the cache index lookup table is set to a null value, and the cache value corresponding to the first input text in the pre-set text-type cache is updated according to the initial value and the null value to be a zero vector;

[0072] When the first input text conversion generates several initial marker words and predicts the first target marker word, this is the pre-filling stage. The pre-filling stage is to convert the input text into marker words and input them into the model for calculation to predict the next marker word.

[0073] When prediction is performed for the first target tag word and the plurality of initial tag words, the cache value in the pre-set text-type cache is updated by the target cache value generated by the first target tag word and the plurality of initial tag words in context reasoning.

[0074] When predictions are made for the first target marker word and several of the initial marker words, this is the decoding stage, which is to calculate the predicted marker word and continue to infer the next marker word until the terminator is inferred and the answer to the input text is completed.

[0075] It can be seen that the text cache needs to be 0 in the initialization stage and pre-filling stage, because there is no upper-level calculation result to record in these two stages. In the decoding stage, the result of the previous calculation needs to be taken out of the cache and input into the model for calculation, and the new calculation result needs to be saved in the cache to overwrite the original value for the next calculation. Figure 4 , the update management strategy of the text cache in the embodiment of the present application is as follows Figure 4 As shown, the specific process is as follows:

[0076] When the reasoning process is in the initialization stage or the first pre-filling stage, the update flag is 0, the cache value corresponding to the input text is initialized to the zero vector, and the update flag and the initialization flag are both updated to 1.

[0077] Based on the vLLM framework, it can be known that the address value of the marker word in the initialization phase is the first preset value, wherein the first preset value can be -1, and the marker word address in the pre-filling phase is the correct address value. Therefore, it can be determined whether the current phase is the initialization phase or the pre-filling phase by judging whether the address value is -1. If it is the initialization phase, the update flag bit and the initialization flag bit are reset to 0, and the cache index lookup table is set to empty to prepare for the next pre-filling phase.

[0078] When the reasoning process is in the first pre-filling stage, the process of the previous order when the reasoning process is in the initialization stage or the first pre-filling stage is repeated, but because the mark word address in the pre-filling stage is the correct address value, the flag bit and the cache index lookup table are not reset, preparing for the subsequent decoding stage.

[0079] When the reasoning process is in the decoding stage, the model takes out the cache value from the cache unit and inputs it into each decoding layer for calculation. After the calculation is completed, the new cache value is copied to the corresponding position to complete the cache value update.

[0080] It should be noted that, in the embodiment of the present application, the pre-set flag bit includes an initialization flag bit and an update flag bit.

[0081] Furthermore, the pre-set text-type cache is a three-dimensional matrix, the first dimension corresponding to the three-dimensional matrix is ​​the number of network layers of the pre-set reasoning model, the second dimension corresponding to the three-dimensional matrix is ​​an input text queue composed of a number of the first input texts and the second input texts; the third dimension corresponding to the three-dimensional matrix is ​​the cache value corresponding to each of the input texts.

[0082] By designing the pre-set text cache as a three-dimensional matrix, in which the first dimension corresponds to the number of network layers of the inference model, the second dimension corresponds to the input text queue, and the third dimension corresponds to the cache value of each input text, this structure can efficiently organize and manage cache data, allowing the model to quickly access and update the cache during the inference process, thereby significantly improving the inference speed and efficiency.

[0083] It should be noted that in the embodiment of the present application, the pre-set text type cache is a cache that stores the corresponding sentences (input text) by setting it as a three-dimensional matrix. The first dimension represents the number of network layers of the reasoning model, the second dimension represents the input text queue, and the third dimension is the cache tensor corresponding to each input text. At the same time, three auxiliary parameters are designed to complete the update of the cache value, the cache index lookup table, the initialization flag, and the update flag.

[0084] The two flags are initialized to 0, and the cache index lookup table is initialized to empty, which is used to record the input text number corresponding to the current cache area. Since the real index of the current input text cannot be known during model calculation, this number is not equal to the actual input text queue but will be dynamically updated according to the algorithm.

[0085] Furthermore, before the step of determining the first marked word address corresponding to each of the first input texts when contextual reasoning is performed on a plurality of first input texts at the same time, the method includes:

[0086] When the number of all the first input texts corresponding to the first input texts is greater than a preset threshold, determining an input text queue;

[0087] After the step of updating the cache value in the preset text-based cache according to the change state corresponding to the flag bit preset in the context reasoning of each of the first input texts and the update state of the cache index lookup table, the method includes:

[0088] When there is a second input text in a waiting-for-reasoning state in the input text queue, acquiring the first input text in an incomplete-reasoning state;

[0089] Performing contextual reasoning on the first input text and the second input text that are in an incomplete reasoning state at the same time, and setting a cache value corresponding to the second input text to a zero vector;

[0090] Determine a second marked word address corresponding to each of the second input texts;

[0091] Obtain a second input text index corresponding to each second input text;

[0092] The second markup word address and the second input text index of the same second input text are bound in the cache index lookup table.

[0093] It should be noted that when vLLM is performing high-concurrency inference services, there is a limit on the number of input texts that can be inferred simultaneously. If the number of input texts exceeds the threshold, the texts exceeding the threshold must wait until the previous text calculations are completed before they can be added to the calculation queue. When new texts are added to the calculation queue, they will first undergo a pre-filling phase and then be merged into the calculation queue to perform decoding phase calculations together with other texts.

[0094] Therefore, before performing step 101 , the input text queue may be determined according to the number of input texts, and the input text queue may be updated based on whether the input text is in an inference completion state.

[0095] When multiple input texts are inferred in parallel, the order in which the input texts are completed may vary. In order for the program to maintain parallel inference, the program will pre-fill the unreasoned sentences, and then merge them with the unfinished sentences and decode them in parallel.

[0096] Specifically, in this pre-filling phase, since the cache update flag is 1, a new zero tensor will be created on the basis of the original cache during cache initialization for model reasoning in the pre-filling phase. At the same time, the cache index lookup table will also store the new text index.

[0097] Further, setting the binding relationship between the second markup word address corresponding to the second input text and the second input text in the cache index lookup table includes:

[0098] If the second mark word address corresponding to the second input text is the same as the first mark word address, then the second input text index in the cache index lookup table covers the first input text index.

[0099] It should be noted that, based on the previous explanation, after the queue is updated, if the first marker word address of the new text conflicts with the original address, that is, the second marker word address corresponding to the second input text is the same as the first marker word address, then at this time, the new index number can be directly overwritten with the original one in the cache index lookup table.

[0100] Further, after the step of updating the cache value in the pre-set text-based cache by the change state corresponding to the flag bit pre-set in the context reasoning of each of the first input texts and the update state of the cache index lookup table, the method includes:

[0101] The cache value in the updated preset text-type cache is read through a preset reasoning model, and the first input text and / or the second input text are reasoned to obtain the reasoning result corresponding to the first input text and / or the second input text.

[0102] Further, after the step of reading the cache value in the updated preset text-type cache through the preset reasoning model, reasoning the first input text, and obtaining the reasoning result corresponding to the first input text, the method includes:

[0103] Determine the target input text index in the inference state in the cache index lookup table;

[0104] Determine the first input text and / or the second input text in a reasoning completion state through the target input text index;

[0105] The first input text and / or the second input text in the inference completed state is deleted from the preset text type cache, and the cache value in the preset text type cache is updated.

[0106] It should be noted that in an embodiment of the present application, after the pre-filling stage of the new input text, i.e., the second input text, is completed, the number of cache values ​​stored in the cache unit will exceed the number of texts currently inferred in parallel, causing the index of the cache index lookup table to be out of range, so both need to be updated.

[0107] Through the mapping address of the current marker word, the mapping address of the first character of the input text where the marker word is located can be calculated, and then the cache index lookup table can be used to locate the target input text index currently being inferred. According to the target input text index, the cache value corresponding to the first input text and / or the second input text in the reasoning completion state can be deleted, thereby releasing unnecessary memory space.

[0108] At the same time, the index information of the deleted text can be used to update the cache index lookup table to ensure the accuracy of the lookup table records and avoid index overflow.

[0109] In addition, it should be noted that, in the embodiment of the present application, the above method can also be applied to the case where the number of texts being inferred is less than the maximum parallel threshold.

[0110] The cache update method provided in the embodiment of the present invention includes: when contextual reasoning is performed on several first input texts at the same time, determining the first marker word address corresponding to each first input text; obtaining the first input text index corresponding to each first input text; binding the first marker word address of the same first input text with the first input text index to obtain a cache index lookup table; updating the cache value in the pre-set text-type cache through the change state corresponding to the pre-set flag bit of each first input text in the contextual reasoning and the update state of the cache index lookup table. In the embodiment of the present invention, by designing the text-type cache value in the text-type cache as a global variable in the reasoning model, and by setting multiple flag bits and a cache index lookup table to realize dynamic update management of the cache value of the text-type cache, waste and overflow of video memory space can be avoided.

[0111] Reference Figure 2 , shows a schematic diagram of the structure of a cache update device provided by an embodiment of the present invention, the device comprising:

[0112] A determination module 201, configured to determine a first marked word address corresponding to each of the first input texts when contextual reasoning is performed on a plurality of first input texts at the same time;

[0113] An acquisition module 202 is used to acquire a first input text index corresponding to each first input text;

[0114] A binding module 203, used for binding the first markup word address of the same first input text with the first input text index to obtain a cache index lookup table;

[0115] The updating module 204 is used to update the cache value in the preset text-based cache according to the change status corresponding to the flag bits preset in the context reasoning of each of the first input texts and the update status of the cache index lookup table.

[0116] The cache update device provided in the embodiment of the present invention determines the first marker word address corresponding to each first input text when contextual reasoning is performed on several first input texts at the same time; obtains the first input text index corresponding to each first input text; binds the first marker word address of the same first input text with the first input text index to obtain a cache index lookup table; updates the cache value in the pre-set text-type cache according to the change state corresponding to the flag bits pre-set in the contextual reasoning of each first input text and the update state of the cache index lookup table. In the embodiment of the present invention, by designing the text-type cache value in the text-type cache as a global variable in the reasoning model, and by setting multiple flag bits and a cache index lookup table to achieve dynamic update management of the cache value of the text-type cache, waste and overflow of video memory space can be avoided.

[0117] The embodiment of the present invention also provides a communication device, such as Figure 3 As shown, it includes a processor 301, a communication interface 302, a memory 303 and a communication bus 304, wherein the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304.

[0118] Memory 303, used for storing computer programs;

[0119] The processor 301, when used to execute the program stored in the memory 303, can implement the following steps:

[0120] When context reasoning is performed on a plurality of first input texts at the same time, determining a first marked word address corresponding to each of the first input texts;

[0121] Obtain a first input text index corresponding to each first input text;

[0122] Binding the first markup word address of the same first input text with the first input text index to obtain a cache index lookup table;

[0123] The cache value in the preset text-type cache is updated according to the change status corresponding to the flag bits preset in the context reasoning of each of the first input texts and the update status of the cache index lookup table.

[0124] Among them, the memory and the processor are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and memories together. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits together, which are all well known in the art, so this article will not further describe them. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor can be transmitted on a wired medium or on a wireless medium through an antenna. Further, the antenna also receives data and transmits the data to the processor. The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. The memory can be used to store data used by the processor when performing operations.

[0125] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0126] The communication interface is used for communication between the above terminal and other devices.

[0127] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0128] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0129] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions. When the instructions are executed on a computer, the computer executes the cache update method described in any of the above embodiments.

[0130] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the cache update method described in any one of the above embodiments.

[0131] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0132] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0133] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0134] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A cache update method, characterized in that: The method comprises: When context reasoning is performed on a plurality of first input texts at the same time, determining a first marked word address corresponding to each of the first input texts; Obtain a first input text index corresponding to each first input text; Binding the first markup word address of the same first input text with the first input text index to obtain a cache index lookup table; The cache value in the preset text-type cache is updated according to the change status corresponding to the flag bits preset in the context reasoning of each of the first input texts and the update status of the cache index lookup table.

2. The method according to claim 1, characterized in that The determining of the first marked word address corresponding to each first input text comprises: By obtaining the key-value cache address corresponding to each input text in contextual reasoning in a preset vectorized large language framework, the first marker word address corresponding to each input text is determined.

3. The method according to claim 1, characterized in that The pre-set flag bit includes an initialization flag bit and an update flag bit; The updating of the cache value in the pre-set text-based cache by using the change state corresponding to the flag bits pre-set in the context reasoning of each of the first input texts and the update state of the cache index lookup table includes: When the first mark word address is a first preset value, the update flag bit and the initialization flag position corresponding to the first input text are initial values, and the cache index lookup table is set to a null value, the cache value corresponding to the first input text in the preset text type cache is updated to be a zero vector; When a plurality of initial marker words generated by the conversion of the first input text are predicted to obtain a first target marker word, the update flag bit and the initialization flag position corresponding to the first input text are set to initial values, the cache index lookup table is set to a null value, and the cache value corresponding to the first input text in the pre-set text-type cache is updated according to the initial value and the null value to be a zero vector; When prediction is performed for the first target tag word and the plurality of initial tag words, the cache value in the pre-set text-type cache is updated by the target cache value generated by the first target tag word and the plurality of initial tag words in context reasoning.

4. The method according to claim 1, characterized in that: Before the step of determining the first marked word address corresponding to each of the first input texts when contextual reasoning is performed simultaneously on a plurality of first input texts, the method includes: When the number of all the first input texts corresponding to the first input texts is greater than a preset threshold, determining an input text queue; After the step of updating the cache value in the preset text-based cache according to the change state corresponding to the flag bit preset in the context reasoning of each of the first input texts and the update state of the cache index lookup table, the method includes: When there is a second input text in a waiting-for-reasoning state in the input text queue, acquiring the first input text in an incomplete-reasoning state; Performing contextual reasoning on the first input text and the second input text that are in an incomplete reasoning state at the same time, and setting a cache value corresponding to the second input text to a zero vector; Determine a second marked word address corresponding to each of the second input texts; Obtain a second input text index corresponding to each second input text; The second markup word address and the second input text index of the same second input text are bound in the cache index lookup table.

5. The method according to claim 4, characterized in that The step of setting the binding relationship between the second markup word address corresponding to the second input text and the second input text in the cache index lookup table comprises: If the second mark word address corresponding to the second input text is the same as the first mark word address, then the second input text index in the cache index lookup table covers the first input text index.

6. The method according to claim 5, after the step of updating the cache value in the pre-set text-based cache by the change state corresponding to the flag bit pre-set in the context reasoning of each of the first input texts and the update state of the cache index lookup table, the method comprises: The cache value in the updated preset text-type cache is read through a preset reasoning model, and the first input text and / or the second input text are reasoned to obtain the reasoning result corresponding to the first input text and / or the second input text.

7. The method according to claim 6, characterized in that After the step of reading the updated cache value in the preset text-type cache through the preset reasoning model, reasoning the first input text, and obtaining the reasoning result corresponding to the first input text, the method includes: Determine the target input text index in the inference state in the cache index lookup table; Determine the first input text and / or the second input text in a reasoning completion state through the target input text index; The first input text and / or the second input text in the inference completed state is deleted from the preset text type cache, and the cache value in the preset text type cache is updated.

8. A cache management device, characterized in that: The device comprises: A determination module, configured to determine a first marked word address corresponding to each of the first input texts when contextual reasoning is performed on a plurality of first input texts at the same time; An acquisition module, used for acquiring a first input text index corresponding to each first input text; A binding module, used for binding the first markup word address of the same first input text with the first input text index to obtain a cache index lookup table; An updating module is used to update the cache value in the preset text-type cache according to the change status corresponding to the flag bits preset in the context reasoning of each of the first input texts and the update status of the cache index lookup table.

9. A communication device, characterized in that: include: A transceiver, a memory, a processor, and a program stored on the memory and executable on the processor; The processor is used to read the program in the memory to implement the cache update method as described in any one of claims 1-7.

10. A readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the cache updating method as described in any one of claims 1 to 7 is implemented.

Images