Large model underlying data processing method and system based on sparse attention

Abstract

The application provides a large model bottom layer data processing method and system based on sparse attention, relates to the technical field of data processing, obtains to-be-processed demand data, sets task feature coding according to the to-be-processed demand data, analyzes feature usefulness and uselessness, obtains a priority coefficient, performs priority scoring, and generates a dynamic sparse mask; according to density information of the dynamic sparse mask, the long text is divided into multiple levels of windows, multiple levels of local windows and window sparse masks thereof are obtained, attention analysis is performed on each local window, window key nodes are obtained, and a multi-window key node connection network is established; update data of the to-be-processed demand data is obtained, update sparse coding is obtained according to the update data, and the dynamic sparse mask is compared, and the application realizes efficient, high-precision and flexible analysis and processing of bottom layer data processing.

Description

Technical Field

[0001] This invention proposes a method and system for processing underlying data of large models based on sparse attention, which relates to the field of data processing technology, specifically to processing underlying data of large models based on sparse attention. Background Technology

[0002] Traditional sparse attention solutions mostly employ static sparse strategies, such as fixed window partitioning or random selection of key tokens, without dynamically adjusting to actual user needs. This leads to a disconnect between sparse regions and task objectives, resulting in the easy omission of key information or ineffective reduction of redundant computation. While some solutions introduce task awareness, they lack multi-level processing mechanisms adapted to the semantic structure of long texts, making it difficult to balance local details with global relationships. Furthermore, when faced with updated user requirements, the entire text needs to be reprocessed, resulting in high update costs, slow response times, and an inability to adapt to multi-turn interaction scenarios. Summary of the Invention

[0003] This invention provides a method and system for processing low-level data of large models based on sparse attention, in order to solve the above-mentioned problems:

[0004] The present invention proposes a method and system for processing low-level data of large models based on sparse attention. The method includes:

[0005] S1. Obtain the demand data to be processed, set the task feature code according to the demand data to be processed, analyze the usefulness and uselessness of features according to the task feature code, obtain the priority coefficient, perform priority scoring, and generate a dynamic sparse mask according to the priority score.

[0006] S2. Based on the density information of the dynamic sparse mask, the long text is divided into multi-level windows to obtain multi-level local windows and their sparse masks. Attention analysis is performed on each local window to obtain key nodes of the window and to establish a multi-window key node connection network.

[0007] S3. Obtain updated data of the data to be processed, obtain updated sparse coding based on the updated data, compare and analyze the differences with the dynamic sparse mask, and then update the multi-window key node connection network to obtain the underlying data processing information.

[0008] Further, S1 includes:

[0009] Extract the requirements features from the requirements data to be processed to obtain requirements feature data;

[0010] The requirement feature data is encoded into task types to obtain initial task type encoded data;

[0011] The semantic topic distribution of the long text is obtained based on the initial task type encoding data, and window feature data is obtained. The initial task type encoding data is then updated based on the window feature data to obtain task feature encoding.

[0012] Priority analysis is performed on the tokens of long texts based on task feature encoding to obtain a priority score for each token;

[0013] The initial task type encoding data is updated based on the priority score to obtain a dynamic sparse mask.

[0014] Furthermore, the step of prioritizing the long text based on task feature encoding to obtain a priority score includes:

[0015] Based on the task feature encoding, the importance of the requirement feature data is analyzed to obtain useful feature data and useless feature data. Based on the useful feature data and the useless feature data, the task feature encoding corresponding to the requirement feature data is used to obtain the task priority coefficient.

[0016] The task feature codes are prioritized according to the priority coefficients to obtain a priority score for the task feature codes.

[0017] Further, the step of performing feature importance analysis on the requirement feature data based on task feature encoding to obtain useful feature data and useless feature data, and obtaining the task priority coefficient based on the task feature encoding corresponding to the requirement feature data according to the useful feature data and the useless feature data, includes:

[0018] Each task feature code is numerically split into multiple dimensions to obtain multi-dimensional task values;

[0019] Each long text and each token is split into multiple dimensions of numerical values ​​to obtain multi-dimensional numerical values ​​of the text;

[0020] Obtain the cosine similarity between the value of each dimension of the feature encoding of each task and the corresponding dimension value of the long text to obtain the dimension similarity data;

[0021] The dimensional similarity data is compared with a preset dimensional similarity threshold to obtain the dimensional similarity comparison result;

[0022] Based on the dimensional similarity comparison results, each dimension is judged to be either useful or useless feature data, thereby obtaining useful feature dimensions and useless feature dimensions;

[0023] Obtain the ratio of the number of useful feature dimensions to the product of the number of useless feature dimensions and a preset minimum value to obtain the dimension priority coefficient;

[0024] The task priority coefficient is obtained by summing the dimensional priority coefficients of multiple dimensions of each task feature encoding.

[0025] Further, S2 includes:

[0026] The long text is divided into multiple levels of local windows based on the dynamic sparse mask, and the window sparse mask of multiple levels of windows is obtained.

[0027] Window attention is calculated for the sparse mask of each local window to obtain the window attention weight matrix;

[0028] The key nodes of the window are determined based on the window attention weight matrix;

[0029] Connect the key nodes of all local windows to obtain a multi-window key node connection network.

[0030] Furthermore, the step of dividing the long text into multi-level local windows based on the dynamic sparse mask to obtain window sparse masks for multiple windows includes:

[0031] The long text is divided into multiple high-level windows by pre-setting the minimum token density based on the dynamic sparse mask density.

[0032] Based on the dynamic sparse mask density, each high-level window is divided into multiple medium-level windows by a preset medium token density.

[0033] Based on the dynamic sparse mask density, each middle-layer window is divided according to the preset maximum token density to obtain multiple bottom-layer windows;

[0034] Obtain the sparse mask of the high-level window, the sparse mask of the middle-level window, and the sparse mask of the low-level window based on the high-level window, the middle-level window, and the low-level window.

[0035] Further, determining the key nodes of the window based on the window attention weight matrix includes:

[0036] Sort the window attention weight matrix of each local window in descending order of weight to obtain the attention node sequence;

[0037] The key nodes of the window are determined based on the attention node sequence;

[0038] Calculate the sparse attention between the key nodes of the window and other key nodes of the local window to obtain key node attention calculation data;

[0039] The key node attention calculation data is compared with a preset key attention threshold to obtain the key node attention comparison result;

[0040] Based on the comparison results of the key nodes, determine whether to update the window key nodes and obtain the updated window key nodes.

[0041] Further, the step of comparing the key node attention calculation data with a preset key attention threshold to obtain a key node attention comparison result includes:

[0042] Obtain preset high threshold and preset low threshold for key attention;

[0043] When the calculated attention data of the key node is greater than the preset high threshold for key attention, the key node of the window will not be updated;

[0044] When the calculated attention data of the key nodes is less than the preset low threshold for key attention, the key nodes of the window are updated;

[0045] When the key node attention calculation data is between the preset key attention high threshold and the preset key attention low threshold, the cosine similarity of the key nodes in the window is updated.

[0046] Further, S3 includes:

[0047] Obtain the demand update data of the demand data to be processed, and obtain the update sparse code of the demand update data;

[0048] The dynamic sparse coding is compared with the updated sparse mask to obtain the updated difference coding;

[0049] The multi-window key node connection network is updated according to the update difference code to obtain the underlying data processing information.

[0050] Furthermore, the system includes:

[0051] The priority analysis module is used to acquire the demand data to be processed, set the task feature code according to the demand data to be processed, analyze the usefulness and uselessness of features according to the task feature code, obtain the priority coefficient, perform priority scoring, and generate a dynamic sparse mask according to the priority score.

[0052] The network establishment module is used to divide the long text into multi-level windows according to the density information of the dynamic sparse mask, obtain multi-level local windows and their window sparse masks, perform attention analysis on each local window, obtain key nodes of the window, and establish a multi-window key node connection network.

[0053] The data update module is used to obtain updated data of the data to be processed, obtain updated sparse encoding based on the updated data, compare and analyze the differences with the dynamic sparse mask, and then update the multi-window key node connection network to obtain the underlying data processing information.

[0054] The beneficial effects of this invention are as follows: By deeply binding requirements with sparse masks, this invention allows attention computation to focus on the core user needs, reducing redundant computation of irrelevant text and significantly improving the efficiency of large models in processing long texts. The multi-level window partitioning and key node network design avoid the semantic fragmentation problem of single-level windows and capture long-distance dependencies through cross-window associations, ensuring the integrity of text processing. The requirement update mechanism enables the system to dynamically adapt, responding to changes in requirements without reprocessing the entire text, greatly reducing repetitive computation costs, and making it suitable for large model applications with multi-round interactions and dynamically adjusted requirements. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the underlying data processing method for large models based on sparse attention. Detailed Implementation

[0056] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0057] In one embodiment of the present invention, a method and system for processing low-level data of large models based on sparse attention is proposed, the method comprising:

[0058] S1. Obtain the demand data to be processed, set the task feature code according to the demand data to be processed, analyze the usefulness and uselessness of features according to the task feature code, obtain the priority coefficient, perform priority scoring, and generate a dynamic sparse mask according to the priority score.

[0059] S2. Based on the density information of the dynamic sparse mask, the long text is divided into multi-level windows to obtain multi-level local windows and their sparse masks. Attention analysis is performed on each local window to obtain key nodes of the window and to establish a multi-window key node connection network.

[0060] S3. Obtain updated data of the data to be processed, acquire updated sparse encoding based on the updated data, compare and analyze the differences with the dynamic sparse mask, and then update the multi-window key node connection network to obtain underlying data processing information, such as... Figure 1 As shown.

[0061] The working principle and technical effects of the above solution are as follows: Starting from the user's original needs, the needs are first transformed into computable task feature codes. By analyzing the correlation between the codes and long text features, useful and useless features are distinguished, priority coefficients are determined, and priority scores are generated. The scores are then transformed into binary dynamic sparse masks, focusing on text regions that require attention calculation. Based on the density distribution of the dynamic sparse mask, the long text is divided into high-level, mid-level, and low-level windows to ensure that high-priority text regions receive more refined window coverage. Attention calculation is performed on each window, key nodes within the window are extracted, and a cross-window key node connection network is constructed to achieve the fusion of local information and global correlation. For scenarios involving updated needs, the updated needs are first transformed into updated sparse codes. Differences are located by comparing them with the original dynamic sparse mask, and then the key node connection network is adjusted based on the difference results to ensure that the processing results always adapt to the latest needs.

[0062] This invention deeply binds requirements with sparse masks, allowing attention computation to focus on core user needs, reducing redundant computation of irrelevant text, and significantly improving the efficiency of large models processing long texts. Multi-level window partitioning and key node network design avoid the semantic fragmentation problem of single-level windows and capture long-distance dependencies through cross-window associations, ensuring the integrity of text processing. The requirement update mechanism enables the system to dynamically adapt, responding to changes in requirements without reprocessing the entire text, greatly reducing repetitive computation costs, and making it suitable for large model applications with multi-round interactions and dynamically adjusted requirements.

[0063] In one embodiment of the present invention, S1 includes:

[0064] Demand feature extraction is performed on the demand data to be processed to obtain demand feature data; the demand data to be processed is the original demand information input by the user (such as generating a summary from a long text).

[0065] The requirement feature data is encoded by task type to obtain initial task type encoded data; the task type encoding includes encodings of task type and text attributes, such as 0.1, 0.2, 0.9, etc.; the task and text attributes are mapped to vectors through an embedding layer;

[0066] The semantic topic distribution of the long text is obtained based on the initial task type encoding data, and window feature data is obtained. The initial task type encoding data is then updated based on the window feature data to obtain task feature encoding. The window feature data includes the number of tokens in the window, the semantic topic of the window, and the matching degree between the window and the task.

[0067] Priority analysis is performed on the tokens of long texts based on task feature encoding to obtain a priority score for each token; the priority score is a coefficient multiplied by a preset score value.

[0068] The initial task type encoding data is updated based on the priority score to obtain a dynamic sparse mask. The dynamic sparse mask is a binary matrix generated based on the priority score (1 indicates that attention needs to be calculated, and 0 indicates that it can be skipped).

[0069] The working principle and technical effect of the above technical solution are as follows: the user input original requirements (such as long text summary generation) are feature extracted, and key requirement feature data such as task type, text attributes, and core demands are selected; these requirement features are mapped into initial task type encoding in vector form through the embedding layer, so that the requirements are computable; the semantic topic distribution of the long text is analyzed in combination with the initial encoding, and window feature data such as the number of tokens and semantic matching degree in the window are obtained, and the initial encoding is optimized to obtain task feature encoding that is more in line with the actual text; the correlation between each token and the requirements of the long text is analyzed based on the task feature encoding, and a priority score for each token is given; a binary dynamic sparse mask is generated according to the score to mark the tokens (1) that need attention calculation and the tokens (0) that can be skipped.

[0070] Through multiple rounds of encoding optimization and feature analysis, precise alignment between requirements and text features is achieved, avoiding the problem of requirements being disconnected from processing priorities. The embedding layer transforms requirements into vector encodings; the optimization of encoding using window feature data makes task features more closely match the semantic structure of long texts, improving the accuracy of priority scoring; the generation of dynamic sparse masks directly defines the scope of attention calculations, reducing invalid computations from the source and improving the efficiency of underlying data processing in large models, which is especially suitable for scenarios with clear requirements and complex text structures.

[0071] In one embodiment of the present invention, the step of performing priority analysis on long texts based on task feature encoding to obtain a priority score includes:

[0072] Based on the task feature coding, the importance of the requirement feature data is analyzed to obtain useful and useless feature data. The task priority coefficient is obtained by coding the task feature corresponding to the requirement feature data based on the useful feature data and the useless feature data. The requirement feature data is key information extracted from the original requirements (such as abstracts, technical documents, and word count information).

[0073] The task feature codes are prioritized according to the priority coefficients to obtain a priority score for the task feature codes.

[0074] The working principle and technical effect of the above technical solution are as follows: Priority coefficients are determined through feature importance analysis, and scoring is set using these coefficients. Based on the task feature encoding, the correlation between each feature and its encoding in the requirement feature data is compared to distinguish between useful feature data that helps complete the task and irrelevant, useless feature data. Combining the quantity and correlation strength of the two types of feature data, a task priority coefficient is calculated. This coefficient quantifies the degree of influence of the requirement features on the task. Based on the priority coefficients, priority is set for each text token corresponding to the task feature encoding. Tokens corresponding to features with high correlation and large coefficients receive higher scores, while those with lower coefficients receive lower scores, forming a priority scoring system covering all tokens in the long text.

[0075] This method makes priority scoring more aligned with the essence of the task, avoiding the shortcomings of traditional scoring that relies solely on text frequency and ignores the relevance of needs. The distinction between useful and useless features ensures that scoring focuses on core needs; the introduction of priority coefficients enables quantitative control of scoring, making the impact of different features on scoring perceptible and adjustable, thereby improving the accuracy of dynamic sparse masks and effectively distinguishing the scoring priorities of different tasks in multi-task, multi-needs scenarios.

[0076] In one embodiment of the present invention, the step of performing feature importance analysis on the requirement feature data based on task feature encoding to obtain useful feature data and useless feature data, and obtaining a task priority coefficient based on the task feature encoding corresponding to the requirement feature data according to the useful feature data and the useless feature data, includes:

[0077] Each task feature encoding is numerically split into multiple dimensions to obtain multi-dimensional numerical values ​​for the task; a vector with dimension d, for example, dimension 1: task type, dimension 2: text domain, dimension 3: key information type, etc.

[0078] Each long text and each token is split into multiple dimensions of numerical values ​​to obtain multi-dimensional numerical values ​​of the text;

[0079] Obtain the cosine similarity between the value of each dimension of the feature encoding of each task and the corresponding dimension value of the long text to obtain the dimension similarity data;

[0080] The dimensional similarity data is compared with a preset dimensional similarity threshold to obtain the dimensional similarity comparison result;

[0081] Based on the dimensional similarity comparison results, each dimension is judged to be either useful or useless feature data, thereby obtaining useful feature dimensions and useless feature dimensions;

[0082] Obtain the ratio of the product of the number of useful feature dimensions and the number of useless feature dimensions and a preset minimum value (to avoid a denominator of 0) to obtain the dimension priority coefficient; obtain the product first, then obtain the ratio.

[0083] The task priority coefficient is obtained by summing the dimensional priority coefficients of multiple dimensions of each task feature encoding.

[0084] The working principle and technical effect of the above technical solution are as follows: This method focuses on the specific calculation process of task priority coefficients, and achieves accurate coefficient solution through multi-dimensional decomposition and similarity analysis. Task feature encoding is decomposed into numerical values ​​of multiple dimensions (such as task type, text domain, and key information type dimensions), and each token of a long text is also decomposed into text values ​​of the same dimension to ensure comparability. The cosine similarity between the numerical values ​​of each dimension of the task and the corresponding numerical values ​​of the text is calculated to quantify the matching degree of each dimension. Then, the similarity data is compared with a preset threshold to determine whether each dimension belongs to a useful feature dimension (similarity meets the standard) or a useless feature dimension (similarity does not meet the standard). The ratio of the number of useful feature dimensions to the number of useless feature dimensions plus a preset minimum value is calculated to obtain the dimension priority coefficient of each dimension. The coefficients of all dimensions are summed to obtain the final task priority coefficient.

[0085] By employing multi-dimensional decomposition, this method achieves a refined comparison of task and text features, avoiding the one-sidedness caused by single-dimensional analysis. The application of cosine similarity makes the quantification of dimensional matching more scientific, while the introduction of a preset minimum value avoids computational anomalies caused by a zero denominator, ensuring process stability. The summation process from dimensional priority coefficients to task priority coefficients comprehensively considers the influence of various dimensions, ensuring that the final coefficients fully reflect the overall correlation between the task and the text. This method significantly improves the accuracy of priority coefficients, providing high-quality input for priority scoring and sparse mask generation, and is suitable for scenarios with complex feature dimensions and highly diverse requirements.

[0086] In one embodiment of the present invention, S2 includes:

[0087] The long text is divided into multi-level local windows based on the dynamic sparse mask to obtain window sparse masks for multiple levels of windows.

[0088] Window attention is calculated for the sparse mask of each local window to obtain the window attention weight matrix;

[0089] The key nodes of the window are determined based on the window attention weight matrix;

[0090] Connect the key nodes of all local windows to obtain a multi-window key node connection network.

[0091] The working principle and technical effect of the above technical solution are as follows: Based on the distribution density of 1 in the dynamic sparse mask, the long text is divided into multi-level local windows. Regions with different densities correspond to different levels of windows, and a window sparse mask is generated for each window. For each local window, attention calculation is performed in combination with its window sparse mask to analyze the correlation strength of each token in the window and obtain the window attention weight matrix. Then, the tokens with high weights and core functions in the semantics of the window are selected from the weight matrix and identified as key nodes of the window. All key nodes of local windows are connected to the window hierarchy according to semantic association to form a multi-window key node connection network covering the entire long text, realizing the interconnection of core information of each window.

[0092] Multi-level window partitioning combined with mask density allows window coverage to better align with the importance distribution of the text, with high-priority regions receiving more refined window splitting to ensure no key information is missed. Window attention calculation and key node extraction focus on the core semantics of the window, reducing interference from redundant information. The multi-window key node connection network breaks down window barriers, enabling the association of global core information in long texts. This not only solves the problem of semantic fragmentation in traditional window partitioning but also reduces network complexity through key nodes, providing support for large models to efficiently capture long-distance dependencies, demonstrating significant advantages in ultra-long text processing scenarios.

[0093] In one embodiment of the present invention, the step of dividing a long text into multi-level local windows based on the dynamic sparse mask to obtain window sparse masks for multiple windows includes:

[0094] The long text is divided into multiple high-level windows based on the preset minimum token density (within a preset range) according to the dynamic sparse mask density; the dynamic sparse mask density is the proportion of tokens marked as 1 within the window.

[0095] Based on the dynamic sparse mask density, each high-level window is divided into multiple medium-level windows by a preset medium token density (within a preset range).

[0096] Based on the dynamic sparse mask density, each middle-layer window is divided into multiple bottom-layer windows by a preset maximum token density (within a preset range);

[0097] The sparse masks of the high-level, mid-level, and low-level windows are obtained based on the high-level, mid-level, and low-level windows. The higher the proportion of priority tokens, the larger the window sparse mask, and the more computational resources are allocated to it.

[0098] The working principle and technical effect of the above solution are as follows: Define the dynamic sparse mask density as the proportion of tokens marked as 1 within a window. Then, divide the area into layers according to a preset density range: the region with the lowest density is divided into high-level windows based on a preset minimum token density standard, covering a larger area; within each high-level window, the region with medium density is divided into mid-level windows based on a preset medium token density, smaller in size than the high-level windows; within each mid-level window, the region with the highest density is divided into low-level windows based on a preset maximum token density, smallest in size. Finally, based on the range and density of each layer of windows, generate corresponding high-level, mid-level, and low-level sparse masks. The higher the density of the window, the greater the proportion of 1s in the mask, and the more computational resources are allocated subsequently.

[0099] By employing a density-layered and progressively subdivided approach, a precise match between window size and text importance is achieved. High-priority, high-density regions receive smaller windows and more computational resources, ensuring precise processing of critical information. Low-priority, low-density regions are covered by larger windows, reducing computational costs and achieving the goal of allocating resources to core areas. The generation of multi-level window sparse masks provides targeted guidance for attention calculations in each window, avoiding resource waste. Simultaneously, progressive subdivision ensures the semantic coherence of long texts, demonstrating a significant effect in balancing processing efficiency and semantic integrity.

[0100] In one embodiment of the present invention, determining the key nodes of the window based on the window attention weight matrix includes:

[0101] Sort the window attention weight matrix of each local window in descending order of weight to obtain the attention node sequence;

[0102] The key nodes of the window are determined based on the attention node sequence;

[0103] Calculate the sparse attention between the key nodes of the window and other key nodes of the local window to obtain key node attention calculation data;

[0104] The key node attention calculation data is compared with a preset key attention threshold to obtain the key node attention comparison result;

[0105] Based on the comparison results of the key nodes, determine whether to update the window key nodes and obtain the updated window key nodes.

[0106] The working principle and technical effect of the above technical solution are as follows: The attention weight matrix of each local window is sorted in descending order of weight value to obtain the attention node sequence, with nodes of higher weight appearing earlier; the node with the highest weight is selected from the front of the sequence and initially identified as the key node of the window; the sparse attention of the key node with other key nodes of the local window is calculated, and the cross-window association strength is analyzed to obtain the key node attention calculation data; the calculated data is compared with two preset high and low key attention thresholds to determine whether the node needs to be updated; finally, the key node is adjusted according to the comparison results to ensure that the key node is always the core token with the closest association with the global context within the window.

[0107] The initial screening of key nodes through weighted sorting ensures their local core status; cross-window attention analysis introduces a global perspective, avoiding the problem of focusing only on local aspects while neglecting global connections; and the dual-threshold comparison update mechanism allows key nodes to dynamically adapt to changes in global connections, preventing connection failures caused by node rigidity. The identified key nodes possess both local semantic coreness and global connection effectiveness, improving the stability and accuracy of the multi-window key node connection network and significantly enhancing the reliability of connection analysis in long text global semantic integration scenarios.

[0108] In one embodiment of the present invention, comparing the key node attention calculation data with a preset key attention threshold to obtain a key node attention comparison result includes:

[0109] Obtain preset high threshold and preset low threshold for key attention;

[0110] When the calculated attention data of the key node is greater than the preset high threshold for key attention, the key node of the window will not be updated;

[0111] When the calculated attention data of the key node is less than the preset low threshold of key attention, the key node of the window is updated; it is directly replaced with the Next-K token in the attention sequence.

[0112] When the key node attention calculation data is between the preset key attention high threshold and the preset key attention low threshold, the cosine similarity of the key nodes in the window is updated.

[0113] The working principle and technical effect of the above technical solution are as follows: two key attention thresholds are preset: a high threshold reflecting strong association and a low threshold reflecting weak association; the attention calculation data of key nodes and other window key nodes are compared with the two thresholds respectively: if the data is greater than the high threshold, it means that the node is extremely strongly associated with the global system and no update is needed; if the data is less than the low threshold, it means that the node is extremely weakly associated with the global system and the node is directly replaced by selecting the next K high-weight tokens from the attention node sequence; if the data is between the two thresholds, it means that the association strength is moderate and the cosine similarity between the node and the task feature encoding needs to be recalculated. The node is retained based on the updated similarity. If the similarity meets the standard, it is retained; if it does not meet the standard, it is replaced.

[0114] The dual-threshold setting allows for more precise judgment of key node updates, avoiding the black-and-white judgment defects of single-threshold. Retaining strongly correlated nodes ensures network stability, while replacing weakly correlated nodes improves network effectiveness. Recalculating the similarity of moderately correlated nodes further verifies the adaptability of nodes to tasks, reducing misjudgments due to moderate global correlation. This mechanism ensures that key nodes always maintain a strong correlation and high adaptability, providing a reliable basis for the dynamic optimization of multi-window key node connection networks and effectively maintaining network performance in scenarios with complex text semantics and variable correlation relationships.

[0115] In one embodiment of the present invention, S3 includes:

[0116] Obtain the demand update data of the demand data to be processed, and obtain the update sparse encoding of the demand update data; the update sparse mask is the update feature encoding corresponding to the demand data update (e.g., the user supplements the summary to include experimental data); the binary vector generated during the update (the structure is consistent with the dynamic sparse mask, 1 indicates the addition of a token that needs attention).

[0117] The dynamic sparse coding is compared with the updated sparse mask to obtain the updated difference coding; the comparison result between the dynamic sparse mask and the updated sparse coding is (1: added, -1: removed, 0: unchanged).

[0118] The multi-window key node connection network is updated according to the update difference code to obtain the underlying data processing information.

[0119] The working principle and technical effect of the above technical solution are as follows: Real-time capture of updated content in the data to be processed (e.g., user-supplemented summaries need to include experimental data), conversion of the update requirements into binary update sparse encoding consistent with the dynamic sparse mask structure, using 1 to mark newly added tokens requiring attention; then comparing the original dynamic sparse encoding and the update sparse encoding bit by bit to generate update difference encoding, using 1 to represent newly added tokens requiring attention, -1 to represent tokens to be removed, and 0 to represent tokens that remain unchanged; adjusting the multi-window key node connection network according to the update difference encoding, adding tokens that meet the key node criteria and adding them to the network, removing tokens that are key nodes and adding new nodes, and finally outputting the updated underlying data processing information.

[0120] This system enables lightweight processing of requirement updates, eliminating the need to recalculate all long text data. It responds to changes in requirements simply through encoding comparison and local network adjustments, significantly reducing update costs and improving system response speed. The design of updating sparse and differential encodings makes requirement changes quantifiable and localizable, ensuring that network adjustments accurately correspond to the updated content. Dynamic adjustments to key node networks guarantee that processing results always match the latest requirements, avoiding the problem of lagging processing results after requirement updates. This makes it suitable for large-scale model applications with multi-round interactions and continuous requirement optimization.

[0121] According to one embodiment of the present invention, the system includes:

[0122] The priority analysis module is used to acquire the demand data to be processed, set the task feature code according to the demand data to be processed, analyze the usefulness and uselessness of features according to the task feature code, obtain the priority coefficient, perform priority scoring, and generate a dynamic sparse mask according to the priority score.

[0123] The network establishment module is used to divide the long text into multi-level windows according to the density information of the dynamic sparse mask, obtain multi-level local windows and their window sparse masks, perform attention analysis on each local window, obtain key nodes of the window, and establish a multi-window key node connection network.

[0124] The data update module is used to obtain updated data of the data to be processed, obtain updated sparse encoding based on the updated data, compare and analyze the differences with the dynamic sparse mask, and then update the multi-window key node connection network to obtain the underlying data processing information.

[0125] The working principle and technical effects of the above-mentioned technical solution are as follows: This invention, through the deep binding of requirements and sparse masks, allows attention computation to focus on the core user needs, reducing redundant computation of irrelevant text and significantly improving the efficiency of large models in processing long texts. Multi-level window partitioning and key node network design not only avoid the problem of semantic fragmentation caused by single-level windows, but also capture long-distance dependencies through cross-window associations, ensuring the integrity of text processing. The requirement update mechanism enables the system to have dynamic adaptation capabilities, responding to changes in requirements without reprocessing the entire text, greatly reducing the cost of repetitive computation, and making it suitable for large model application scenarios with multi-round interactions and dynamic requirement adjustments.

[0126] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for processing the underlying data of a large model based on sparse attention, characterized in that: The method includes: S1. Obtain the demand data to be processed, set the task feature code according to the demand data to be processed, analyze the usefulness and uselessness of features according to the task feature code, obtain the priority coefficient, perform priority scoring, and generate a dynamic sparse mask according to the priority score. S2. Based on the density information of the dynamic sparse mask, the long text is divided into multi-level windows to obtain multi-level local windows and their sparse masks. Attention analysis is performed on each local window to obtain key nodes of the window and to establish a multi-window key node connection network. S3. Obtain updated data of the data to be processed, obtain updated sparse coding based on the updated data, compare and analyze the difference with the dynamic sparse mask, and then update the multi-window key node connection network to obtain the underlying data processing information. Wherein, S1 includes: Extract the requirements features from the requirements data to be processed to obtain requirements feature data; The requirement feature data is encoded into task types to obtain initial task type encoded data; The semantic topic distribution of the long text is obtained based on the initial task type encoding data, and window feature data is obtained. The initial task type encoding data is then updated based on the window feature data to obtain task feature encoding. Priority analysis is performed on the tokens of long texts based on task feature encoding to obtain a priority score for each token; The initial task type encoding data is updated based on the priority score to obtain a dynamic sparse mask; The step of performing priority analysis on long texts based on task feature encoding to obtain priority scores includes: Based on the task feature encoding, the importance of the requirement feature data is analyzed to obtain useful feature data and useless feature data. Based on the useful feature data and the useless feature data, the task feature encoding corresponding to the requirement feature data is used to obtain the task priority coefficient. The task feature codes are prioritized according to the priority coefficients to obtain a priority score for the task feature codes.

2. The method for processing the underlying data of a large model based on sparse attention according to claim 1, characterized in that, The step of analyzing the importance of features in the requirement feature data based on the task feature code to obtain useful and useless feature data, and obtaining the task priority coefficient based on the task feature code corresponding to the requirement feature data by combining the useful and useless feature data, includes: Each task feature code is numerically split into multiple dimensions to obtain multi-dimensional task values; Each long text and each token is split into multiple dimensions of numerical values ​​to obtain multi-dimensional numerical values ​​of the text; Obtain the cosine similarity between the value of each dimension of the feature encoding of each task and the corresponding dimension value of the long text to obtain the dimension similarity data; The dimensional similarity data is compared with a preset dimensional similarity threshold to obtain the dimensional similarity comparison result; Based on the dimensional similarity comparison results, each dimension is judged to be either useful or useless feature data, thereby obtaining useful feature dimensions and useless feature dimensions; Obtain the ratio of the number of useful feature dimensions to the product of the number of useless feature dimensions and a preset minimum value to obtain the dimension priority coefficient; The task priority coefficient is obtained by summing the dimensional priority coefficients of multiple dimensions of each task feature encoding.

3. The method for processing the underlying data of a large model based on sparse attention according to claim 1, characterized in that, S2 includes: The long text is divided into multi-level local windows based on the dynamic sparse mask to obtain window sparse masks for multiple levels of windows. Window attention is calculated for the sparse mask of each local window to obtain the window attention weight matrix; The key nodes of the window are determined based on the window attention weight matrix; Connect the key nodes of all local windows to obtain a multi-window key node connection network.

4. The method for processing the underlying data of a large model based on sparse attention according to claim 3, characterized in that, The step of dividing the long text into multi-level local windows based on the dynamic sparse mask to obtain window sparse masks for multiple windows includes: The long text is divided into multiple high-level windows by pre-setting the minimum token density based on the dynamic sparse mask density. Based on the dynamic sparse mask density, each high-level window is divided into multiple medium-level windows by a preset medium token density. Based on the dynamic sparse mask density, each middle-layer window is divided according to the preset maximum token density to obtain multiple bottom-layer windows; Obtain the sparse mask of the high-level window, the sparse mask of the middle-level window, and the sparse mask of the low-level window based on the high-level window, the middle-level window, and the low-level window.

5. The method for processing the underlying data of a large model based on sparse attention according to claim 3, characterized in that, The step of determining the key nodes of the window based on the window attention weight matrix includes: Sort the window attention weight matrix of each local window in descending order of weight to obtain the attention node sequence; The key nodes of the window are determined based on the attention node sequence; Calculate the sparse attention between the key nodes of the window and other key nodes of the local window to obtain key node attention calculation data; The key node attention calculation data is compared with a preset key attention threshold to obtain the key node attention comparison result; Based on the comparison results of the key nodes, determine whether to update the window key nodes and obtain the updated window key nodes.

6. The method for processing the underlying data of a large model based on sparse attention according to claim 5, characterized in that, The step of comparing the key node attention calculation data with a preset key attention threshold to obtain the key node attention comparison result includes: Obtain preset high threshold and preset low threshold for key attention; When the calculated attention data of the key node is greater than the preset high threshold for key attention, the key node of the window will not be updated; When the calculated attention data of the key nodes is less than the preset low threshold for key attention, the key nodes of the window are updated; When the key node attention calculation data is between the preset key attention high threshold and the preset key attention low threshold, the cosine similarity of the key nodes in the window is updated.

7. The method for processing the underlying data of a large model based on sparse attention according to claim 1, characterized in that, S3 includes: Obtain the demand update data of the demand data to be processed, and obtain the update sparse code of the demand update data; The dynamic sparse mask is compared with the updated sparse code to obtain the updated difference code; The multi-window key node connection network is updated according to the update difference code to obtain the underlying data processing information.

8. A large-scale model underlying data processing system based on sparse attention, characterized in that: The system includes: The priority analysis module is used to acquire the demand data to be processed, set the task feature code according to the demand data to be processed, analyze the usefulness and uselessness of features according to the task feature code, obtain the priority coefficient, perform priority scoring, and generate a dynamic sparse mask according to the priority score. The network establishment module is used to divide the long text into multi-level windows according to the density information of the dynamic sparse mask, obtain multi-level local windows and their window sparse masks, perform attention analysis on each local window, obtain key nodes of the window, and establish a multi-window key node connection network. The data update module is used to obtain updated data of the data to be processed, obtain updated sparse coding based on the updated data, compare and analyze the difference with the dynamic sparse mask, and then update the multi-window key node connection network to obtain the underlying data processing information. The priority analysis module includes: Extract the requirements features from the requirements data to be processed to obtain requirements feature data; The requirement feature data is encoded into task types to obtain initial task type encoded data; The semantic topic distribution of the long text is obtained based on the initial task type encoding data, and window feature data is obtained. The initial task type encoding data is then updated based on the window feature data to obtain task feature encoding. Priority analysis is performed on the tokens of long texts based on task feature encoding to obtain a priority score for each token; The initial task type encoding data is updated based on the priority score to obtain a dynamic sparse mask; The step of performing priority analysis on long texts based on task feature encoding to obtain priority scores includes: Based on the task feature encoding, the importance of the requirement feature data is analyzed to obtain useful feature data and useless feature data. Based on the useful feature data and the useless feature data, the task feature encoding corresponding to the requirement feature data is used to obtain the task priority coefficient. The task feature codes are prioritized according to the priority coefficients to obtain a priority score for the task feature codes.

Images