Data processing and scheduling method and device based on dynamic sparse attention calculation

Abstract

The application relates to a data processing and scheduling method and device based on dynamic sparse attention calculation. The method comprises the following steps: splitting a target prompt text to obtain a plurality of subtexts; for each subtext, the query feature of each first word element in the subtext and the key feature of each second word element in the current traversed subtext are sequentially used to determine a preset number of target word elements corresponding to the first word element, and the global maximum attention weight of the first word element is determined; the ratio of the attention weight of the first word element under each target word element to the sum of the attention weight of the target word element and the attention weight of the subtext to which the target word element belongs satisfies a preset condition; and the response information of the target prompt text is determined according to the global maximum attention weight of each first word element, the key feature and the attention weight of each target word element. The method can reduce invalid calculation and memory occupation, and improve processing efficiency.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, and in particular to a data processing and scheduling method and apparatus based on dynamic sparse attention computation. Background Technology

[0002] As the application scenarios of large language models continue to expand, the demand for text reasoning with long contexts continues to grow. Attention computation, as a core computational component of large-scale language models (such as the Transformer model), determines the overall performance of long text processing through its computational efficiency and memory overhead.

[0003] In traditional techniques, self-attention mechanisms require calculating the attention score between every two words in the text to be processed and generating a complete global attention matrix to perform weight normalization and weighted summation operations. However, as the length of the text to be processed increases, the size of the attention matrix grows rapidly exponentially, which not only brings high computational costs but also generates a large amount of intermediate data, causing significant data handling and storage pressure, and thus severely restricting the response speed of long text reasoning.

[0004] Therefore, this attention computing scheme has technical problems such as high computing power and high memory consumption, resulting in low data processing efficiency in long text scenarios and difficulty in meeting the high real-time long context processing requirements. Summary of the Invention

[0005] Therefore, it is necessary to provide a data processing and scheduling method and apparatus based on dynamic sparse attention computation to address the above-mentioned technical problems. This method and apparatus can reduce the computational overhead and memory usage of attention computation, thereby improving the data processing efficiency in long text scenarios and meeting the requirements of high real-time long context processing.

[0006] Firstly, this application provides a data processing and scheduling method based on dynamic sparse attention computation, including:

[0007] The target prompt text is split into multiple subtexts; each subtext includes multiple words from the target prompt text.

[0008] For each subtext, following the traversal order of different subtexts, a preset number of target words corresponding to the first word word are determined based on the query features of each first word word in the subtext and the key features of each second word word in the currently traversed subtext. The global maximum attention weight of the first word word is also determined. The ratio of the attention weight of the first word word under each corresponding target word word to the sum of the attention weights of the subtext to which the target word word belongs satisfies a preset condition. The sum of attention weights is the sum of the attention weights of the first word word under each second word word in the subtext to which the target word word belongs. Based on the global maximum attention weight of each first word word, and the key features and attention weights of each target word word, the response information of the target prompt text is determined.

[0009] In one embodiment, for each subtext, according to the traversal order of different subtexts, a preset number of target words corresponding to the first word are determined sequentially based on the query features of each first word in the subtext and the key features of each second word in the currently traversed subtext. This includes: for each subtext, taking the first subtext in different subtexts as the first traversed subtext, and alternately selecting the current subtext to be traversed from the beginning and end of each untraversed subtext; and, when the current subtext to be traversed is selected, determining the target words corresponding to the first word in the currently traversed subtext based on the query features of each first word in the subtext and the key features of each second word in the currently traversed subtext; if the number of determined target words is less than the preset number, returning to the operation of taking the first subtext in different subtexts as the first traversed subtext and alternately selecting the current subtext to be traversed from the beginning and end of each untraversed subtext, until the number of determined target words reaches the preset number.

[0010] In one embodiment, the target word corresponding to the first word in the currently traversed subtext is determined sequentially based on the query features of each first word in the subtext and the key features of each second word in the currently traversed subtext. This includes: determining the attention weight of the first word under the second word based on the query features of the first word and the key features of each second word in the currently traversed subtext; and selecting a second word from the currently traversed subtext whose determined ratio satisfies a preset condition as the target word corresponding to the first word based on the ratio of the attention weight of the first word under each second word in the currently traversed subtext to the sum of the attention weights corresponding to the first word in the currently traversed subtext.

[0011] In one embodiment, determining the global maximum attention weight of the first word element includes: for each first word element in the subtext, selecting the maximum value from the attention weights of the first word element under each second word element in the currently traversed subtext as the local undetermined maximum value; comparing the local undetermined maximum value with the local historical maximum value corresponding to the currently traversed first word element; if the local undetermined maximum value is greater than the local historical maximum value, determining the local undetermined maximum value as the new local historical maximum value, and traversing the next subtext corresponding to the currently traversed subtext; if the local undetermined maximum value is not greater than the local historical maximum value, retaining the local historical maximum value, and traversing the next subtext corresponding to the currently traversed subtext; and after traversing all second word elements in all subtexts, determining the current local historical maximum value as the global maximum attention weight of the first word element.

[0012] In one embodiment, the response information of the target prompt text is determined based on the global maximum attention weight of each first word element and the key features and attention weight of each target word element. This includes: for each subtext, obtaining the value features of the reference word element based on the key features of each reference word element corresponding to the subtext; and determining the response information of the subtext based on the global maximum attention weight of each first word element in the subtext, the value features of each reference word element, and the attention weight corresponding to each reference word element; wherein, each reference word element corresponding to the subtext is a target word element corresponding to each first word element in the subtext; the attention weight corresponding to each reference word element is the attention weight of the first word element corresponding to the reference word element in the subtext under the reference word element; and determining the response information of the target prompt text based on the sum of the response information of each subtext.

[0013] In one embodiment, the response information of the subtext is determined based on the global maximum attention weight of each first word element in the subtext, the value features of each reference word element, and the attention weight corresponding to each reference word element. This includes: for each reference word element, normalizing the attention weight corresponding to the reference word element based on the global maximum attention weight of the first word element corresponding to the reference word element to obtain the normalized attention weight corresponding to the reference word element; determining the weighted attention weight of the reference word element based on the product between the normalized attention weight and the value features of the reference word element; and determining the response information of the subtext based on the weighted attention weight of each reference word element.

[0014] Secondly, this application also provides a data processing and scheduling device based on dynamic sparse attention computation, comprising:

[0015] The splitting module is used to split the target prompt text into multiple subtexts; each subtext includes multiple words from the target prompt text.

[0016] The first determining module is used to determine, for each subtext, a preset number of target words corresponding to the first word word according to the query features of each first word word and the key features of each second word word in the currently traversed subtext, and to determine the global maximum attention weight of the first word word; wherein, the ratio of the attention weight of the first word word under each corresponding target word word to the sum of the attention weights of the subtext to which the target word word belongs satisfies a preset condition; the sum of attention weights is the sum of the attention weights of the first word word under each second word word in the subtext to which the target word word belongs;

[0017] The second determining module is used to determine the response information of the target prompt text based on the global maximum attention weight of each first word element and the key features and attention weight of each target word element.

[0018] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the various method embodiments of the first aspect described above.

[0019] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the method embodiments of the first aspect described above.

[0020] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps in the method embodiments of the first aspect described above.

[0021] The aforementioned data processing and scheduling method and apparatus based on dynamic sparse attention computation splits the target prompt text into multiple subtexts. Each subtext includes multiple words from the target prompt text. For each subtext, according to the traversal order of different subtexts, a preset number of target words corresponding to the first word are determined based on the query features of each first word in the subtext and the key features of each second word in the currently traversed subtext. The global maximum attention weight of the first word is also determined. The ratio of the attention weight of the first word under each corresponding target word to the sum of the attention weights of the subtext to which the target word belongs satisfies a preset condition. The sum of attention weights is the sum of the attention weights of the first word under each second word in the subtext to which the target word belongs. The response information of the target prompt text is determined based on the global maximum attention weight of each first word, as well as the key features and attention weights of each target word. In this way, since the selection of target lexical units is based solely on the ratio of the attention weight of the second lexical unit corresponding to the first lexical unit to the sum of the attention weights of all second lexical units in the subtext corresponding to the second lexical unit, without relying on the key and value features of all lexical units in the target prompt text, it is possible to filter out second lexical units whose ratio does not meet the preset conditions, thereby achieving local pruning within each subtext, reducing the scale of lexical units participating in subsequent calculations, and reducing unnecessary computational overhead. At the same time, by determining the global maximum attention weight of each first lexical unit in the subtext during the traversal process to determine the response information of the target prompt text, it is not necessary to pre-generate and store the complete global attention matrix, which significantly reduces the storage occupation of intermediate data and the pressure of off-chip data transfer in long text scenarios. Thus, while ensuring the accuracy of attention calculation, it effectively improves the computational efficiency and resource utilization in long text processing scenarios, which is conducive to meeting the high real-time long context processing requirements. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is an application environment diagram of a data processing and scheduling method based on dynamic sparse attention computation in one embodiment.

[0024] Figure 2 This is a flowchart illustrating a data processing and scheduling method based on dynamic sparse attention computation in one embodiment.

[0025] Figure 3This is a flowchart illustrating the process of determining a preset number of target words corresponding to a first word word in one embodiment.

[0026] Figure 4 This is a flowchart illustrating the process of determining the target word corresponding to the first word in the currently traversed subtext in one embodiment.

[0027] Figure 5 This is a flowchart illustrating the process of determining the global maximum attention weight of the first word in one embodiment;

[0028] Figure 6 This is a flowchart illustrating the process of determining the response information for the target prompt text in one embodiment;

[0029] Figure 7 This is a flowchart illustrating the process of determining response information for a subtext in one embodiment;

[0030] Figure 8 This is a flowchart illustrating a data processing and scheduling method based on dynamic sparse attention computation in another embodiment.

[0031] Figure 9 This is a structural block diagram of a data processing and scheduling device based on dynamic sparse attention computation in one embodiment;

[0032] Figure 10 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0034] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0035] The data processing method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located in the cloud or on other network servers. Terminal 102 provides target prompt text, and server 104 determines the query features, key features, and value features of all tokens in the target prompt text based on the target prompt text, and stores these features in the data storage system to determine the response information for the target prompt text based on these features. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, and tablets. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0036] In one exemplary embodiment, such as Figure 2 As shown, a data processing and scheduling method based on dynamic sparse attention computation is provided, which is then applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps: S201~S203. Among them:

[0037] S201, split the target prompt text to obtain multiple sub-texts.

[0038] Each subtext includes multiple lexical units from the target prompt text.

[0039] The target prompt text can be the original text input by the user and to be processed. The word elements in the target prompt text can be the smallest basic unit of text processing, which can include, but are not limited to, characters, words, sub-words or punctuation marks. Furthermore, each word element in the target prompt text can be sorted according to its sequence order in the target prompt text.

[0040] Optionally, the target prompt text can be split according to a preset length or according to semantic boundaries. The length of each subtext is usually limited by the hardware cache capacity to ensure that the subtext can be fully loaded into on-chip storage.

[0041] S202, for each subtext, according to the traversal order of different subtexts, determine the preset number of target words corresponding to the first word word in turn based on the query features of each first word word in the subtext and the key features of each second word word in the currently traversed subtext, and determine the global maximum attention weight of the first word word.

[0042] The ratio of the attention weight of the first word element under each corresponding target word element to the sum of the attention weights of the subtext to which the target word element belongs satisfies a preset condition; the sum of attention weights is the sum of the attention weights of the first word element under each second word element in the subtext to which the target word element belongs.

[0043] The attention weight of the first word element under each second word element in the subtext to which the target word element belongs reflects the degree of attention the first word element pays to that second word element. Generally, the higher the value of this attention weight, the higher the degree of attention the first word element pays to the second word element, that is, the higher the correlation between the first word element and the second word element, and the more significant the influence of the second word element on the attention output of the first word element. Therefore, preset conditions can be set in advance to determine the ratio between the attention weight of the first word element under each second word element and the sum of the attention weights. Based on whether this ratio meets the preset conditions, the target word element can be determined from the currently traversed subtext.

[0044] Optionally, for each word in each subtext, a linear transformation can be performed on the word to obtain the query features, key features, and value features corresponding to that word.

[0045] Optionally, the preset quantity can be flexibly set according to the total length of the target prompt text, the hardware cache capacity of the server 104, or the requirements for calculation accuracy in the actual application scenario. This application embodiment does not limit this. For example, for longer target prompt text, a larger preset quantity can be set to retain more key information; while for situations with limited hardware resources, a smaller preset quantity can be set to reduce calculation and storage overhead.

[0046] Optionally, the process of determining the target word can also be implemented based on a bit-serialization mechanism. Specifically, the feature operation between the query feature of the first word and the key feature of the second word can be split into serial calculations on multiple bits. Starting from the highest bit, the feature operation results are calculated bit by bit. For each bit, the sum of attention weights is determined based on the feature operation results of each traversed bit. Based on the feature operation results of that bit, the attention weight of the first word under the second word is determined. Then, based on the ratio between the attention weight and the sum of attention weights, the satisfaction of the preset condition is determined. If the ratio of the bit does not satisfy the preset condition, the feature operation results of the remaining bits are not calculated, and the second word is directly determined as a non-target word, and the traversal of the next second word begins. If the ratio of the bit satisfies the preset condition, the feature operation results of the remaining bits are calculated, and the attention weight of the first word under the second word is determined based on the feature operation results of all bits.

[0047] When there are multiple preset quantities, each first word element corresponds to multiple target word elements. Therefore, each first word element can have multiple corresponding attention weights. We can first determine a maximum value from each subtext traversed by the first word element, and then select the maximum value from the maximum values ​​corresponding to each traversed subtext to serve as the global maximum attention weight of the first word element.

[0048] The global maximum attention weight of the first word can be used in the normalization process of subsequent attention weights as a benchmark value for normalization, so as to maintain numerical stability without losing any computational accuracy.

[0049] S203. Based on the global maximum attention weight of each first word element, and the key features and attention weight of each target word element, determine the response information of the target prompt text.

[0050] Optionally, the value features of the target word can be obtained based on the index information of the key features of each target word corresponding to each first word. Based on the global maximum attention weight of the corresponding first word, as well as the value features and attention weights of each target word, the attention calculation result of the target prompt text can be determined, and the response information of the target prompt text can be determined based on the attention calculation result.

[0051] Optionally, the response information can be presented in different forms depending on the actual application scenario. For example, the response information can be the attention calculation result, i.e., the attention output feature matrix of the target prompt text. Each element in this attention output feature matrix corresponds to the attention output feature of each word in the target prompt text, which is the new vector representation obtained after the corresponding word has been processed by the attention mechanism and incorporates contextual information. Alternatively, the response information can also be the question-and-answer response, content continuation, translation, classification result, etc., corresponding to the target prompt text.

[0052] Specifically, when the target prompt text is a translation request and the text to be translated entered by the user, the response information can be the translated text of the text to be translated; when the target prompt text is a question entered by the user, the response information can be the answer to the question; when the target prompt text is the beginning of a text, the response information can be the continuation of the text.

[0053] It should be noted that those skilled in the art can flexibly apply the data processing method provided in this application to various text processing scenarios based on attention mechanisms according to actual needs, and the embodiments of this application do not impose any limitations on this.

[0054] The above data processing method splits the target prompt text into multiple subtexts. Each subtext includes multiple words from the target prompt text. For each subtext, according to the traversal order of different subtexts, a preset number of target words corresponding to the first word are determined based on the query features of each first word in the subtext and the key features of each second word in the currently traversed subtext. The global maximum attention weight of the first word is also determined. The ratio of the attention weight of the first word under each corresponding target word to the sum of the attention weights of the subtext to which the target word belongs satisfies a preset condition. The sum of attention weights is the sum of the attention weights of the first word under each second word in the subtext to which the target word belongs. Based on the global maximum attention weight of each first word, as well as the key features and attention weights of each target word, the response information of the target prompt text is determined. In this way, since the selection of target lexical units is based solely on the ratio of the attention weight of the second lexical unit corresponding to the first lexical unit to the sum of the attention weights of all second lexical units in the subtext corresponding to the second lexical unit, without relying on the key and value features of all lexical units in the target prompt text, it is possible to filter out second lexical units whose ratio does not meet the preset conditions, thereby achieving local pruning within each subtext, reducing the scale of lexical units participating in subsequent calculations, and reducing unnecessary computational overhead. At the same time, by determining the global maximum attention weight of each first lexical unit in the subtext during the traversal process to determine the response information of the target prompt text, it is not necessary to pre-generate and store the complete global attention matrix, which significantly reduces the storage occupation of intermediate data and the pressure of off-chip data transfer in long text scenarios. Thus, while ensuring the accuracy of attention calculation, it effectively improves the computational efficiency and resource utilization in long text processing scenarios, which is conducive to meeting the high real-time long context processing requirements.

[0055] Based on the above embodiments, in an exemplary embodiment, such as Figure 3 As shown, in S202 above, for each subtext, according to the traversal order of different subtexts, a preset number of target words corresponding to the first word word are determined sequentially based on the query features of each first word word in the subtext and the key features of each second word word in the currently traversed subtext. This can include the following S301~S303. Wherein:

[0056] S301, for each subtext, take the first subtext in different subtexts as the first subtext to be traversed, and in each of the untraversed subtexts, alternately select the first and last subtexts to be traversed.

[0057] In long text processing applications, the part near the beginning of the target prompt text often mentions the core theme and / or task objective, while the middle part often contains a lot of background information, process descriptions, and detailed elaborations, which are redundant information and therefore have a relatively low relevance to the core theme. The part near the end usually involves the most direct expression of intent. In other words, the earlier and later the subtexts have a more significant impact on the attention calculation results.

[0058] Based on this, the first subtext in different subtexts can be used as the first subtext to be traversed. Among the untraversed subtexts, the first and last subtexts can be selected alternately as the current subtext to be traversed.

[0059] For example, different subtexts can be represented as [B1, B2, B3, ..., Bn] according to their sequence order in the target prompt text. The subtext traversed for the first time can be B1, the subtext traversed for the second time can be Bn, the subtext traversed for the third time can be B2, the subtext traversed for the fourth time can be Bn-1, and so on, until all subtexts have been traversed.

[0060] S302, when the current subtext to be traversed is selected, the target word corresponding to the first word in the current subtext is determined according to the query feature of each first word in the subtext and the key feature of each second word in the current subtext.

[0061] Optionally, for each first word element, the query features of the first word element and the key features of each second word element in the currently traversed subtext can be sequentially subjected to feature operations (e.g., dot product operations) to obtain the attention weight of the first word element under each second word element; further, the sum of the attention weights of the first word element under all second word elements in the currently traversed subtext can be calculated; the attention weight corresponding to each second word element can be divided by the sum of the attention weights to obtain the ratio corresponding to each second word element; the second word element whose ratio satisfies the preset condition is determined as the target word element corresponding to the first word element in the currently traversed subtext.

[0062] S303, determine whether the number of identified target words has reached the preset number. If the number of identified target words has not reached the preset number, return to the above operation S310; if the number of identified target words has reached the preset number, end, that is, stop identifying target words.

[0063] Optionally, if the number of target words currently identified has not reached the preset number, it means that the number of target words selected from the currently traversed subtexts is not sufficient. Therefore, it is necessary to return to S301 to continue selecting the next subtext to be traversed from the untraversed subtexts and continue selecting target words. If the number of target words currently identified has reached the preset number, it means that a sufficient number of target words have been selected, and there is no need to traverse the remaining subtexts. The loop terminates.

[0064] In this embodiment, since the method of alternating between the beginning and end is used to select the current subtext to be traversed from the untraversed subtext, and the traversal is terminated when the number of determined target words reaches a preset number, it can prioritize covering the beginning and end subtext regions with higher semantic relevance in the target prompt text, thereby improving the filtering speed of target words. At the same time, the determination of target words can be completed without traversing all subtexts, which significantly reduces the size of subtexts that need to be loaded and calculated, thereby further reducing the overhead of off-chip data access and the cost of invalid calculation. Thus, while ensuring the accuracy of attention calculation, it further improves the efficiency of attention processing in long text scenarios.

[0065] Based on the above embodiments, in an exemplary embodiment, such as Figure 4 As shown, S302 above, which sequentially determines the target word corresponding to the first word in the currently traversed subtext based on the query features of each first word in the subtext and the key features of each second word in the currently traversed subtext, may include the following S401~S402. Wherein:

[0066] S401, based on the query features of the first word element and the key features of each second word element in the currently traversed subtext, determine the attention weight of the first word element under the second word element.

[0067] Optionally, the attention score of the first word element under the corresponding second word element can be determined based on the dot product of the query features of the first word element and the key features of each second word element in the currently traversed subtext. The attention score can be a real value, reflecting the similarity between the first word element and the corresponding second word element in the vector space. Then, this attention score can be substituted into an exponential function, and the value of the exponential function can be used as the attention weight of the first word element under the second word element. Using an exponential function maps the attention score to a positive value, ensuring that the second word element with higher relevance receives a higher attention weight, avoiding interference from negative or zero values ​​in subsequent ratio calculations.

[0068] S402, based on the ratio of the attention weight of the first word element under each second word element in the currently traversed subtext to the sum of the attention weights corresponding to the current traversed subtext, select the second word element in the currently traversed subtext whose determined ratio satisfies the preset condition, and use it as the target word element corresponding to the first word element.

[0069] The determined ratio reflects the proportion of attention the corresponding second word element occupies in the overall attention of the first word element to the currently traversed subtext. The higher the ratio, the higher the proportion of the second word element in the attention allocation of the first word element.

[0070] Optionally, the preset condition can be that the determined ratio is not less than a preset ratio threshold, or the preset condition can be that the determined ratio is ranked before a preset number of positions in the sorting results of the ratios corresponding to the second word elements in the currently traversed subtext from largest to smallest.

[0071] In this embodiment, since it is not necessary to traverse the key features of all words in the currently traversed subtext, the target word can be selected only based on the key features of each second word in the currently traversed subtext, which saves computing power. After the subtext is traversed, pruning is performed to reduce the data scale of subsequent calculations, thereby improving the security of local pruning and the efficiency of data processing.

[0072] Based on the above embodiments, in an exemplary embodiment, such as Figure 5 As shown, S202 above, determining the global maximum attention weight of the first word element, may include the following S501~S505. Wherein:

[0073] S501, for each first word element in the subtext, select the maximum value from the attention weights of the first word element under each second word element in the currently traversed subtext, and use it as the local undetermined maximum value.

[0074] Optionally, firstly, since the attention weight of the first word element under each second word element in the currently traversed subtext is determined during the process of selecting target word elements from each second word element, the attention weight of the first word element under each second word element in the currently traversed subtext can be recorded synchronously during the process of selecting target word elements, and the maximum value among the attention weights of the first word element under each second word element in the currently traversed subtext can be used as the local undetermined maximum value.

[0075] S502, compare the local undetermined maximum value with the local historical maximum value corresponding to the first traversed word.

[0076] The local historical maximum value corresponding to the first word element can be the maximum value of the attention weights recorded in the subtexts that have been traversed up to the current time.

[0077] Optionally, the values ​​of the local undetermined maximum value and the local historical maximum value can be compared to obtain the comparison results. Then, based on the comparison results, it can be determined whether to update the local historical maximum value.

[0078] S503, if the local undetermined maximum value is greater than the local historical maximum value, determine the local undetermined maximum value as the new local historical maximum value, and traverse the next subtext corresponding to the currently traversed subtext.

[0079] Optionally, if the comparison results show that the local undetermined maximum value is greater than the local historical maximum value, it indicates that the target word corresponding to the current first word element contains an attention weight that is larger than all previously recorded attention weights. Therefore, the local undetermined maximum value needs to be determined as the new local historical maximum value to maintain the accuracy of the local historical maximum value. After the update is completed, the next subtext is traversed. It should be noted that the next subtext here refers to the subtext traversed after the first word element.

[0080] S504: If the local undetermined maximum value is not greater than the local historical maximum value, retain the local historical maximum value and iterate through the next subtext corresponding to the currently traversed subtext.

[0081] Optionally, if the comparison results show that the local undetermined maximum value is not greater than the local historical maximum value, it means that there is no attention weight in the target word corresponding to the current first word element that is greater than all the attention weights recorded previously. Therefore, there is no need to update the local historical maximum value. The local historical maximum value can be retained, and the next first word element in the subtext can be traversed.

[0082] S505: After traversing all the second words in the subtext, the current historical maximum value is determined as the global maximum attention weight of the first word.

[0083] Optionally, after the first word element has traversed all the second resources in all subtexts, the current local historical maximum value has been compared and updated with all local undetermined maximum values. In other words, the local historical maximum value at this time is the maximum value among all attention weights corresponding to the first word element. Therefore, the current local historical maximum value can be used as the global maximum attention weight of the first word element.

[0084] In this embodiment, since the global maximum attention weight is determined by iteratively comparing historical maximum values ​​for each first word element in the subtext, the maximum value can be extracted without temporarily storing the complete attention weight data of all target words, thus reducing the temporary storage overhead of intermediate data. At the same time, the maximum value update process can be executed synchronously with the subtext traversal and target word element filtering process, without the need for an additional global statistics stage in traditional technologies, thereby adapting to the architectural characteristics of block-based streaming computing.

[0085] Based on the above embodiments, in an exemplary embodiment, such as Figure 6 As shown, S203 above, based on the global maximum attention weight of each first word element and the key features and attention weight of each target word element, determines the response information of the target prompt text, which may include the following S601~S603. Wherein:

[0086] S601, for each subtext, obtain the value features of the reference word based on the key features of each reference word corresponding to the subtext.

[0087] In this context, each reference word in the subtext is a target word corresponding to each first word in the subtext.

[0088] Optionally, the value features associated with the key feature can be retrieved from the data storage system based on the index information corresponding to the key feature of each reference word. In this way, the number of reference words is significantly reduced compared to the total number of words in the target prompt text. When obtaining value features, only the value features corresponding to the reference words need to be loaded, instead of loading the value features of all words, thereby reducing the amount of data loaded.

[0089] S602, determine the response information of the subtext based on the global maximum attention weight of each first word element in the subtext, the value features of each reference word element, and the attention weight corresponding to each reference word element.

[0090] The attention weight corresponding to each reference word is the attention weight of the first word in the subtext corresponding to the reference word under the reference word.

[0091] Optionally, the attention weights corresponding to the reference words can be normalized based on the global maximum attention weight of each first word word in the subtext to obtain normalized attention weights. Then, the value features of the corresponding reference words word are multiplied by the corresponding normalized attention weights to obtain the weighted attention weights of each reference word word. Finally, the response information of the subtext can be determined based on the weighted attention weights of each reference word word.

[0092] S603, determine the response information of the target prompt text based on the sum of the response information of each sub-text.

[0093] Optionally, vector addition can be used to accumulate the response information of all sub-texts to obtain the response information of the target prompt text.

[0094] In this embodiment, since the value features corresponding to the selected reference words are obtained only, and the response information is calculated and accumulated in units of subtext to obtain the overall response information of the target prompt text, it is not necessary to load the value features of all words to participate in the calculation, which greatly reduces the data access and computation of value features. At the same time, the block-based calculation and block-by-block accumulation processing method is highly compatible with the subtext splitting architecture, so it can make full use of on-chip cache resources, reduce memory access pressure in long text scenarios, and thus improve the overall execution efficiency of attention calculation.

[0095] Based on the above embodiments, in an exemplary embodiment, such as Figure 7 As shown, S602 above, based on the global maximum attention weight of each first word element in the subtext, the value features of each reference word element, and the attention weight corresponding to each reference word element, determines the response information of the subtext, which may include the following S701~S703. Wherein:

[0096] S701, for each reference word, the attention weight corresponding to the reference word is normalized according to the global maximum attention weight of the first word corresponding to the reference word, so as to obtain the normalized attention weight corresponding to the reference word.

[0097] Since there may be significant numerical differences between the attention weights of different reference words, directly using the original attention weights for subsequent calculations may lead to numerical instability or decreased accuracy. Therefore, normalization can be used to map the attention weights of each reference word to a uniform scale range, thus obtaining the normalized attention weights corresponding to each reference word and avoiding overflow problems caused by excessively large values.

[0098] Specifically, for each reference word, the attention weight corresponding to the reference word can be normalized based on the global maximum attention weight of the first word corresponding to that reference word to obtain the normalized attention weight of that reference word.

[0099] S702, determine the weighted attention weight of the reference word based on the product between the normalized attention weight corresponding to the reference word and the value feature of the reference word.

[0100] Optionally, for each reference word, the product of the normalized attention weight and the value feature corresponding to that reference word represents the degree of contribution of that reference word to the response information. Accordingly, the higher the normalized attention weight and the more significant the value feature, the higher the contribution of that reference word to the response information.

[0101] Specifically, the weighted attention weight of the reference word can be obtained by multiplying the corresponding normalized attention weight with the value feature in vector form through vector multiplication. This weighted attention weight is in vector form, and its dimension is consistent with that of the value feature.

[0102] S703 determines the response information of the subtext based on the weighted attention weights of each reference word.

[0103] Optionally, vector addition can be used to accumulate the weighted attention weights of each reference word to obtain the response information of the subtext. In this case, the response information of the subtext is in vector form, which is a comprehensive representation obtained by weighted fusion of all reference words in the subtext through the attention mechanism, and its dimension is consistent with the dimension of a single value feature.

[0104] In this embodiment, since the attention weights of each reference word are first normalized based on the global maximum attention weight of the first word, and then the response information of the subtext is obtained by weighting the corresponding value features, the numerical stability of the exponential operation can be guaranteed by the global maximum attention weight, avoiding the overflow problem caused by excessively high attention weight values. At the same time, the normalized weights can accurately reflect the relative contribution of each reference word, ensuring the accuracy of attention weighting calculation. Thus, while improving the reliability of the calculation process, the accuracy of the attention output results is guaranteed, thereby taking into account both the stability and effectiveness of long text processing.

[0105] Based on the above embodiments, in an exemplary embodiment, such as Figure 8 As shown, an optional data processing method is provided. Wherein:

[0106] S801, split the target prompt text to obtain multiple sub-texts.

[0107] Each subtext includes multiple lexical units from the target prompt text.

[0108] S802: For each subtext, the first subtext in the different subtexts is taken as the first subtext to be traversed. Among the subtexts that have not been traversed, the first and last subtexts to be traversed are selected alternately.

[0109] S803, when the current subtext to be traversed is selected, the attention weight of the first word under the second word is determined according to the query features of the first word and the key features of each second word in the current subtext.

[0110] S804: Based on the ratio of the attention weight of the first word element under each second word element in the currently traversed subtext to the sum of the attention weights corresponding to the current traversed subtext, select the second word element in the currently traversed subtext whose determined ratio satisfies the preset condition as the target word element corresponding to the first word element. For each first word element in the subtext, select the maximum value from the attention weights of the first word element under each second word element in the currently traversed subtext as the local undetermined maximum value. After traversing all second word elements in all subtexts, determine the global maximum attention weight of the first word element by the current local historical maximum value.

[0111] Specifically, for each subtext traversed by the first word element, if the local undetermined maximum value is greater than the local historical maximum value, the local undetermined maximum value is determined as the new local historical maximum value, and the next subtext corresponding to the currently traversed subtext is traversed; if the local undetermined maximum value is not greater than the historical local maximum value, the local historical maximum value is retained, and the next subtext corresponding to the currently traversed subtext is traversed.

[0112] S805, determine whether the number of identified target words has reached the preset number.

[0113] If the number of identified target words does not reach the preset number, return to the above operation S802; if the number of identified target words reaches the preset number, execute the following S806.

[0114] S806: For each subtext, obtain the value features of the reference lexicon based on the key features of each reference lexicon corresponding to the subtext.

[0115] In this context, each reference word in the subtext is a target word corresponding to each first word in the subtext.

[0116] S807: For each reference word, the attention weight corresponding to the reference word is normalized based on the global maximum attention weight of the first word corresponding to the reference word, so as to obtain the normalized attention weight corresponding to each reference word.

[0117] The attention weight corresponding to each reference word is the attention weight of the first word in the subtext corresponding to the reference word under the reference word.

[0118] S808. The weighted attention weight of the reference word is determined by the product of the normalized attention weight corresponding to the reference word and the value feature of the reference word.

[0119] S809 determines the response information of the subtext based on the weighted attention weight of each reference word.

[0120] S810, determine the response information of the target prompt text based on the cumulative result of the response information of each sub-text.

[0121] It should be noted that the specific implementation methods of S801-S810 are the same as those in the above method embodiments, and will not be repeated here.

[0122] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0123] Based on the same inventive concept, this application also provides a data processing and scheduling apparatus based on dynamic sparse attention computation for implementing the data processing and scheduling method based on dynamic sparse attention computation described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data processing apparatus embodiments provided below can be found in the limitations of the data processing method above, and will not be repeated here.

[0124] In one exemplary embodiment, such as Figure 9 As shown, a data processing and scheduling device based on dynamic sparse attention computation is provided, including: a splitting module 910, a first determining module 920, and a second determining module 930, wherein:

[0125] The splitting module 910 is used to split the target prompt text into multiple subtexts; each subtext includes multiple words from the target prompt text.

[0126] The first determining module 920 is used to determine, for each subtext, a preset number of target words corresponding to the first word word in the subtext according to the query features of each first word word in the subtext and the key features of each second word word in the currently traversed subtext, and to determine the global maximum attention weight of the first word word; wherein, the ratio of the attention weight of the first word word under each corresponding target word word to the sum of the attention weights of the subtext to which the target word word belongs satisfies a preset condition; the sum of attention weights is the sum of the attention weights of the first word word under each second word word in the subtext to which the target word word belongs.

[0127] The second determining module 930 is used to determine the response information of the target prompt text based on the global maximum attention weight of each first word element and the key features and attention weight of each target word element.

[0128] In one embodiment, the first determining module 920 includes:

[0129] The first selection unit is used to select the first subtext in each subtext as the first subtext to be traversed, and to select the current subtext to be traversed alternately from the first and last subtexts in each untraversed subtext.

[0130] The first determining unit is used to determine the target word corresponding to the first word in the currently traversed subtext by sequentially determining the query features of each first word in the subtext and the key features of each second word in the currently traversed subtext when the current subtext to be traversed is selected.

[0131] The second determining unit is used to return to the operation of taking the first subtext in different subtexts as the first traversal and alternately selecting the first and last subtexts to be traversed from each untraversed subtext until the number of determined target words reaches the preset number, when the number of determined target words is less than the preset number.

[0132] In one embodiment, the first determining unit is specifically used to determine the attention weight of the first word under the second word based on the query features of the first word and the key features of each second word in the currently traversed subtext; and to select the second word in the currently traversed subtext whose determined ratio satisfies the preset condition as the target word corresponding to the first word based on the ratio of the attention weight of the first word under each second word in the currently traversed subtext to the sum of the attention weights corresponding to the current traversed subtext.

[0133] In one embodiment, the first determining module 920 further includes:

[0134] The second selection unit is used to select the maximum value from the attention weights of the first word element under each second word element in the currently traversed subtext for each first word element in the subtext, and use it as the local undetermined maximum value.

[0135] The comparison unit is used to compare the local undetermined maximum value with the local historical maximum value corresponding to the first traversed word.

[0136] The traversal unit is used to determine the local undetermined maximum value as the new local historical maximum value when the local undetermined maximum value is greater than the local historical maximum value, and to traverse the next subtext corresponding to the currently traversed subtext; when the local undetermined maximum value is not greater than the historical local maximum value, the local historical maximum value is retained, and the next subtext corresponding to the currently traversed subtext is traversed.

[0137] The third determining unit is used to determine the global maximum attention weight of the first word element by taking the current local historical maximum value after traversing all the second word elements in the subtext.

[0138] In one embodiment, the second determining module 930 includes:

[0139] The acquisition unit is used to acquire the value features of each reference term for each subtext based on the key features of each reference term corresponding to the subtext.

[0140] The fourth determining unit is used to determine the response information of the subtext based on the global maximum attention weight of each first word element in the subtext, the value features of each reference word element, and the attention weight corresponding to each reference word element; wherein, each reference word element corresponding to the subtext is the target word element corresponding to each first word element in the subtext; and the attention weight corresponding to each reference word element is the attention weight of the first word element corresponding to the reference word element in the subtext under the reference word element.

[0141] The fifth determining unit is used to determine the response information of the target prompt text based on the cumulative result of the response information of each sub-text.

[0142] In one embodiment, the fourth determining unit is specifically used to, for each reference word, normalize the attention weight corresponding to the reference word according to the global maximum attention weight of the first word corresponding to the reference word, to obtain the normalized attention weight corresponding to each reference word; determine the weighted attention weight of the reference word according to the product between the normalized attention weight corresponding to the reference word and the value feature of the reference word; and determine the response information of the subtext according to the weighted attention weight of each reference word.

[0143] Each module in the aforementioned data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0144] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 10 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores query features, key features, and value features of all terms. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a data processing method.

[0145] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0146] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0147] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0148] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0149] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0150] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0151] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data processing and scheduling method based on dynamic sparse attention computation, characterized in that, The method includes: The target prompt text is split into multiple subtexts; each subtext includes multiple word elements from the target prompt text. For each subtext, following the traversal order of different subtexts, based on the query features of each first word element in the subtext and the key features of each second word element in the currently traversed subtext, a preset number of target words corresponding to the first word element are determined, and the global maximum attention weight of the first word element is determined; wherein, the ratio of the attention weight of the first word element under each corresponding target word element to the sum of the attention weights of the subtext to which the target word element belongs satisfies a preset condition; the sum of the attention weights is the sum of the attention weights of the first word element under each second word element in the subtext to which the target word element belongs; The response information of the target prompt text is determined based on the global maximum attention weight of each first word and the key features and attention weight of each target word.

2. The method according to claim 1, characterized in that, For each subtext, according to the traversal order of different subtexts, a preset number of target words corresponding to the first word are determined sequentially based on the query features of each first word element in the subtext and the key features of each second word element in the currently traversed subtext, including: For each subtext, the first subtext among different subtexts is selected as the first subtext to be traversed. From the untraversed subtexts, the first and last subtexts to be traversed are selected alternately; and, When the current subtext to be traversed is selected, the target word corresponding to the first word in the current subtext is determined according to the query feature of each first word in the subtext and the key feature of each second word in the current subtext. If the number of determined target words is less than the preset number, return to the operation of taking the first subtext in different subtexts as the first traversal, and alternately selecting the first and last subtexts to be traversed from the untraversed subtexts, until the number of determined target words reaches the preset number.

3. The method according to claim 2, characterized in that, The step of determining the target word corresponding to the first word in the currently traversed subtext based on the query features of each first word in the subtext and the key features of each second word in the currently traversed subtext includes: Based on the query features of the first word element and the key features of each second word element in the currently traversed subtext, determine the attention weight of the first word element under the second word element; Based on the ratio of the attention weight of the first word element under each second word element in the currently traversed subtext to the sum of the attention weights corresponding to the current traversed subtext, a second word element whose determined ratio satisfies a preset condition is selected from the currently traversed subtext as the target word element corresponding to the first word element.

4. The method according to claim 1, characterized in that, Determining the global maximum attention weight of the first word element includes: For each first word element in the subtext, the maximum value is selected from the attention weights of the first word element under each second word element in the currently traversed subtext, and is taken as the local undetermined maximum value. Compare the local undetermined maximum value with the local historical maximum value corresponding to the first traversed word; If the local undetermined maximum value is greater than the local historical maximum value, the local undetermined maximum value is determined as the new local historical maximum value, and the next subtext corresponding to the currently traversed subtext is traversed. If the local undetermined maximum value is not greater than the local historical maximum value, retain the local historical maximum value and iterate through the next subtext corresponding to the currently traversed subtext; After traversing all the second words in the subtext, the current local historical maximum value is determined as the global maximum attention weight of the first word.

5. The method according to claim 1, characterized in that, The step of determining the response information of the target prompt text based on the global maximum attention weight of each first word element and the key features and attention weight of each target word element includes: For each subtext, the value features of the reference lexicon are obtained based on the key features of each reference lexicon corresponding to the subtext; and, The response information of the subtext is determined based on the global maximum attention weight of each first word element in the subtext, the value features of each reference word element, and the attention weight corresponding to each reference word element; wherein, each reference word element corresponding to the subtext is each target word element corresponding to each first word element in the subtext; and the attention weight corresponding to each reference word element is the attention weight of the first word element in the subtext corresponding to the reference word element under the reference word element. The response information of the target prompt text is determined based on the sum of the response information of each sub-text.

6. The method according to claim 5, characterized in that, The step of determining the response information of the subtext based on the global maximum attention weight of each first word element in the subtext, the value features of each reference word element, and the attention weight corresponding to each reference word element includes: For each reference word, the attention weight corresponding to the reference word is normalized according to the global maximum attention weight of the first word corresponding to the reference word, so as to obtain the normalized attention weight corresponding to the reference word. The weighted attention weight of the reference word is determined by multiplying the normalized attention weight corresponding to the reference word with the value feature of the reference word. The response information of the subtext is determined based on the weighted attention weights of each of the reference terms.

7. A data processing and scheduling device based on dynamic sparse attention computation, characterized in that, The device includes: The splitting module is used to split the target prompt text into multiple sub-texts; wherein each sub-text includes multiple word elements from the target prompt text; The first determining module is used to, for each subtext, determine a preset number of target words corresponding to the first word word according to the query features of each first word word in the subtext and the key features of each second word word in the currently traversed subtext, and determine the global maximum attention weight of the first word word; wherein, the ratio of the attention weight of the first word word under each corresponding target word word to the sum of the attention weights of the subtext to which the target word word belongs satisfies a preset condition; the sum of the attention weights is the sum of the attention weights of the first word word under each second word word in the subtext to which the target word word belongs; The second determining module is used to determine the response information of the target prompt text based on the global maximum attention weight of each first word element and the key features and attention weight of each target word element.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images