A long text dynamic compression method and system based on a limited window and a medium

Abstract

A long text dynamic compression method and system based on a limited window, and a medium, include obtaining long text data to be processed and its associated voice data; performing entity recognition on the long text data, and performing confidence enhancement on the recognized entities according to the voice data to obtain an entity list; based on the entity list, dynamically determining a window size and a sliding step of a current text window through a reinforcement learning model, and obtaining a current window text from the long text data according to the window size and the sliding step; performing segment segmentation on the current window text, and calculating the importance weight of each text segment to output a weight matrix; based on the weight matrix and a preset token margin, compressing the current window text to generate compressed text. The application can dynamically and adaptively retain key entity information within a limited token window, and has high compression quality.

Description

Technical Field

[0001] This invention relates to the field of technology, and in particular to a method, system, and medium for dynamic compression of long texts based on a restricted window. Background Technology

[0002] Large language models are limited by the fixed token window capacity when processing long texts, and usually need to compress the input text in advance.

[0003] Existing compression methods often employ static truncation or rule-based summarization, making it difficult to distinguish the importance of text segments and easily resulting in the loss of key entity information located in the middle of the text. Furthermore, existing technologies use fixed sliding window parameters, failing to dynamically adjust according to the entity distribution density of the text content. This leads to computational redundancy in sparse entity regions and exacerbates information loss in dense entity regions. Simultaneously, existing methods rely solely on text features for compression decisions, neglecting the acoustic features such as emotion and stress inherent in the synchronously acquired voice data in real-world application scenarios (e.g., voice customer service systems), which could serve as auxiliary criteria for determining information importance.

[0004] Therefore, existing technologies suffer from the inability to dynamically and adaptively retain key entity information within a limited token window and low compression quality. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a method for dynamic compression of long texts based on a restricted window, comprising the following steps: Acquire the long text data to be processed and its associated speech data; Entity recognition is performed on the long text data, and the confidence of the recognized entities is enhanced based on the speech data to obtain an entity list; Based on the entity list, the window size and sliding step of the current text window are dynamically determined by a reinforcement learning model, and the current window text is extracted from the long text data according to the window size and sliding step. The text in the current window is segmented, and the importance weight of each text segment is calculated, and a weight matrix is ​​output. The importance weight is obtained by fusing the text features of the text segment, the matching degree with the entity list, and the acoustic features of the speech data. Based on the weight matrix and the preset token balance, the text in the current window is compressed to generate compressed text.

[0006] Optionally, entity recognition is performed on the long text data, and confidence enhancement is applied to the recognized entities based on the speech data to obtain an entity list, including: The long text data is used to perform entity recognition by a named entity recognition model to obtain the text entities and their first confidence scores. Mel frequency cepstral coefficient features are extracted from the speech data; The entity enhancement weights are weighted according to the multimodal fusion weight coefficients and the Mel frequency cepstral coefficients, and then superimposed on the first confidence level to obtain an entity list, which includes the text-enhanced entities after multimodal fusion enhancement and the second confidence level.

[0007] Optionally, the window size and sliding step of the current text window can be dynamically determined using a reinforcement learning model, including: Get the initial window size and the average entity retention rate of historical compression interactions; The window size of the current text window is dynamically determined based on the difference between the entity retention rate of the current compression cycle and the average entity retention rate, the initial window size, and the window adjustment coefficient. Calculate the entity density based on the number of text-enhanced entities and the total number of words in the current text window; Obtain an entity density threshold. When the entity density is less than the entity density threshold, calculate the sliding step size based on the current text window size and entity density.

[0008] Optionally, the entity density threshold is dynamically adjusted based on the entity loss rate of the previous compression cycle; The dynamic adjustment steps for the entity density threshold include: obtaining the entity loss rate of the previous compression cycle; and calculating the dynamically adjusted entity density threshold based on the entity loss rate, the penalty learning rate, and the basic decision threshold.

[0009] Optionally, the method further includes: Record log data for the current compression cycle, including entity retention rate and entity loss rate; Update the corresponding parameters of the named entity recognition model based on the log data.

[0010] Optionally, updating the corresponding parameters of the named entity recognition model based on the log data includes: Obtain user feedback data on the compressed text, and determine the entity misclassification rate based on the feedback data. When the entity misclassification rate is greater than a preset misclassification rate threshold, trigger incremental learning. During incremental learning, a total loss function is constructed, which includes a cross-entropy loss term, an entity error constraint term, and an incremental learning loss term. The entity error constraint term is used to minimize the error between the compressed text and the entities in the entity list. The incremental learning loss term calculates the parameter changes of the named entity recognition model based on the log data and feedback data. The total loss function is minimized using the backpropagation algorithm to update the corresponding parameters of the named entity recognition model.

[0011] Optionally, the text in the current window is segmented, and the importance weight of each text segment is calculated, outputting a weight matrix, including: Based on a pre-defined domain corpus, the TF-IDF value of each text segment is calculated; The speech data is time-aligned with the text in the current window, and the speech segment corresponding to each text segment is analyzed based on the VADER model to obtain the corresponding speech emotion score. For each text segment, the importance weight of each text segment is calculated based on the corresponding TF-IDF value, speech emotion score, entity feature weight coefficient, speech emotion feature weight coefficient, and indicator function. Output a weight matrix based on the importance weights of all text segments.

[0012] Optionally, based on the weight matrix and a preset token balance, the text in the current window is compressed to generate compressed text, including: Based on the weight matrix, all text fragments in the current window text are sorted from high to low according to their importance weight; Starting with the text segment with the highest importance weight, the token count is accumulated until the accumulated token count reaches the preset token reserve. The accumulated text segments are retained, and the unretained text segments are input into the summarization model for compression to obtain a compressed summary. The retained fragments are concatenated with the compressed summary to generate compressed text.

[0013] Corresponding to the aforementioned long text dynamic compression method based on a restricted window, the present invention provides a long text dynamic compression system based on a restricted window, comprising: The data acquisition module is used to acquire the long text data to be processed and its associated voice data; An entity recognition module is used to perform entity recognition on the long text data and enhance the confidence of the recognized entities based on the voice data to obtain an entity list; The text extraction module is used to dynamically determine the window size and sliding step of the current text window based on the entity list through a reinforcement learning model, and extract the current window text from the long text data according to the window size and sliding step. The weight calculation module is used to segment the text in the current window, calculate the importance weight of each text segment, and output a weight matrix; the importance weight is obtained by fusing the text features of the text segment, the matching degree with the entity list, and the acoustic features of the speech data. The text compression module is used to compress the text in the current window based on the weight matrix and the preset token balance, and generate compressed text.

[0014] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a long text dynamic compression program based on a restricted window, wherein when the long text dynamic compression program based on a restricted window is executed by a processor, the program implements the steps of the long text dynamic compression method based on a restricted window as described above.

[0015] Compared with the prior art, the present invention has the following beneficial effects: (1) By acquiring speech data associated with long text data, the shortcomings of existing technologies that rely solely on text features for compression decisions are overcome; by enhancing the confidence of entities, the technical problem of losing key entity information in the middle area of ​​the text is solved; by dynamically determining the window size and sliding step size through a reinforcement learning model, the technical problem of fixed sliding window parameters and inability to dynamically adjust according to the entity distribution density of the text content is solved, achieving the technical effect of reducing computational redundancy in sparse entity areas and reducing information loss in dense entity areas; by integrating text features, entity matching degree and speech acoustic features to calculate importance weights, key entity information is dynamically and adaptively retained within a limited token window, significantly improving the compression quality.

[0016] (2) Text entities and their first confidence scores are obtained through the named entity recognition model. The multimodal fusion weight coefficient and Mel frequency cepstral coefficient features are used for weighted superposition, which realizes the synergistic enhancement of text modality and speech modality. This allows the entity confidence score to reflect both text semantic information and speech acoustic information, thereby improving the accuracy and robustness of entity recognition.

[0017] (3) By introducing the average entity retention rate of historical compression interactions, and combining the difference between the entity retention rate of the current compression cycle and the average entity retention rate, the initial window size and the window adjustment coefficient, the window size of the current text window is dynamically determined, thus realizing adaptive window adjustment based on historical performance; by calculating the entity density and dynamically calculating the sliding step when the entity density is lower than the threshold, the window parameters can be adaptively changed according to the entity distribution density of the text content, thus realizing the coordinated optimization configuration of window size and sliding step, which improves compression efficiency while ensuring entity retention rate.

[0018] (4) By introducing the entity loss rate of the previous compression cycle and combining the penalty learning rate with the basic judgment threshold to dynamically adjust the entity density threshold, adaptive iterative optimization of the threshold parameter is realized, so that the entity density threshold can be dynamically corrected according to the historical compression quality, further improving the accuracy of window parameter adjustment and the system's adaptability to different text distributions.

[0019] (5) By recording log data containing entity retention rate and entity loss rate, and updating the corresponding parameters of the named entity recognition model accordingly, closed-loop feedback utilization of compression performance data is realized, enabling the entity recognition model to adaptively optimize according to the performance in the actual compression process, and continuously improve the accuracy of entity recognition and the overall performance of the compression system.

[0020] (6) By introducing user feedback data to determine the entity misclassification rate and trigger incremental learning, a human-machine collaborative model optimization mechanism was realized. By constructing a total loss function that includes cross-entropy loss, entity error constraint and incremental learning loss, the model update takes into account the collaborative constraints of classification accuracy, entity retention consistency and parameter stability. While minimizing the error between compressed text and the original entity list, the change of model parameters is controlled, and the balance between continuous optimization of model performance and stability is achieved.

[0021] (7) The domain importance of text segments is reflected by calculating TF-IDF values ​​based on domain corpus, and speech emotion scores are obtained by time alignment and VADER model analysis to reflect speech emotion features. The entity feature weight coefficient, speech emotion feature weight coefficient and indicator function are combined for fusion calculation, realizing the synergistic fusion of text semantic features, domain features, entity features and speech emotion features, so that the importance weight can fully reflect the multidimensional information value of text segments, and improve the accuracy of weight calculation and the rationality of compression decision.

[0022] (8) By sorting according to importance weight and selectively retaining based on preset token balance, the collaborative processing of prioritizing key information retention and intelligent compression of secondary information is realized; by splicing the retained fragments with the compressed summary, the compressed text contains both high-weight original key information and low-weight semantic summary, thus achieving a balance between original information fidelity and information compression rate optimization, maximizing the retention of key information and maintaining text coherence within a limited token capacity. Attached Figure Description

[0023] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings: Figure 1This is a simplified flowchart of an embodiment of the long text dynamic compression method based on a restricted window according to the present invention; Figure 2 This is a framework diagram of an embodiment of the long text dynamic compression system based on a restricted window according to the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] like Figure 1 As shown, the present invention provides a method for dynamic compression of long texts based on a restricted window, which includes the following steps: Acquire the long text data to be processed and its associated speech data; Entity recognition is performed on long text data, and the confidence of the recognized entities is enhanced based on the speech data to obtain an entity list; Based on the entity list, the window size and sliding step of the current text window are dynamically determined through a reinforcement learning model, and the text of the current window is extracted from the long text data according to the window size and sliding step. The text in the current window is segmented into segments, and the importance weight of each text segment is calculated. The weight matrix is ​​output. The importance weight is obtained by fusing the text features of the text segment, the matching degree with the entity list, and the acoustic features of the speech data. Based on the weight matrix and the preset token balance, the text in the current window is compressed to generate compressed text.

[0026] In this embodiment, the long text data to be processed can be the intelligent customer service dialogue log.

[0027] This invention overcomes the shortcomings of existing technologies that rely solely on text features for compression decisions by acquiring speech data associated with long text data; it solves the technical problem of easily losing key entity information in the middle region of text by enhancing entity confidence; it solves the technical problem of fixed sliding window parameters and inability to dynamically adjust according to the entity distribution density of text content by dynamically determining the window size and sliding step size through a reinforcement learning model, achieving the technical effect of reducing computational redundancy in sparse entity regions and reducing information loss in dense entity regions; and it significantly improves compression quality by fusing text features, entity matching degree, and speech acoustic features to calculate importance weights and dynamically and adaptively retaining key entity information within a limited token window.

[0028] In this embodiment, entity recognition is performed on long text data, and the confidence of the recognized entities is enhanced based on the speech data to obtain an entity list, including: Entity recognition is performed on long text data using a named entity recognition model to obtain text entities and their first confidence scores; preferably, the named entity recognition model is the BERT-NER model, and the initial value of the first confidence score is 1.0. Extract Mel-frequency cepstral coefficient features from speech data; Based on the multimodal fusion weight coefficients and Mel-frequency cepstral coefficients, the entity augmentation weights are weighted and then summed to the first confidence level to obtain an entity list, which includes the text-enhanced entities after multimodal fusion augmentation and a second confidence level; where, ; Represents a list of entities. This represents the text entity and its first confidence score (i.e., the entity recognition result above). This represents the Mel-frequency cepstral coefficient extraction function (e.g., using the Librosa library to extract 13-dimensional features), where A represents the speech data. This represents the multimodal fusion weighting coefficient, preferably 0.3. It is understood that after multimodal fusion enhancement, the number of entities may increase, and the second confidence level may be greater than the first confidence level. This invention obtains text entities and their first confidence scores through a named entity recognition model, and uses multimodal fusion weight coefficients and Mel frequency cepstral coefficient features for weighted superposition to achieve synergistic enhancement of text and speech modalities. This enables the entity confidence score to simultaneously reflect text semantic information and speech acoustic information, thereby improving the accuracy and robustness of entity recognition.

[0029] In this embodiment, the window size and sliding step size of the current text window are dynamically determined through a reinforcement learning model, including: Obtain the initial window size and the average entity retention rate of historical compression interactions; preferably, the initial window size is 4000 (for an 8K Token window). The current text window size is dynamically determined based on the difference between the entity retention rate and the average entity retention rate in the current compression cycle, the initial window size, and the window adjustment coefficient; among which, W represents the current size of the text window. This indicates the initial window size, and R represents the entity retention rate of the current compression cycle (actually referring to the feedback reward generated by the compression task of the previous compression cycle or the most recently completed compression task, initially set to R=). (or R=1.0), and R= , This indicates the number of entities contained in the compressed text in the previous compression cycle. This represents the total number of entities contained in the long text data to be processed. This represents the average entity retention rate, which is obtained by averaging the sum of the entity retention rates of all historical compressed interactions. This represents the window adjustment factor, preferably 0.05; Entity density is calculated based on the number of text-enhanced entities and the total number of words in the current text window; where, ; Represents the density of the substance. Indicates the number of text-enhanced entities. This indicates the total number of words in the current text window; Obtain the entity density threshold. When the entity density is less than the threshold, calculate the sliding step size based on the current text window size and the entity density. S represents the sliding step size.

[0030] This invention dynamically determines the current text window size by introducing the average entity retention rate of historical compression interactions, combining the difference between the entity retention rate of the current compression cycle and the average entity retention rate, the initial window size, and the window adjustment coefficient, thus achieving adaptive window adjustment based on historical performance. By calculating entity density and dynamically calculating the sliding step size when the entity density is below a threshold, the window parameters can adaptively change according to the entity distribution density of the text content, achieving coordinated optimization of window size and sliding step size, thereby improving compression efficiency while ensuring entity retention rate.

[0031] In this embodiment, the entity density threshold is dynamically adjusted based on the entity loss rate of the previous compression cycle; The dynamic adjustment steps for the entity density threshold include: obtaining the entity loss rate from the previous compression cycle; calculating the dynamically adjusted entity density threshold based on the entity loss rate, the penalty learning rate, and the base decision threshold; wherein, ; This represents the dynamically adjusted entity density threshold. This represents the basic threshold for judgment, preferably 0.1. This represents the penalty learning rate, preferably 0.1. This indicates the entity loss rate in the previous compression cycle (e.g., if the entity loss rate in the previous compression cycle was 5%). =0.05), specifically calculated using 1-R', where R' represents the entity retention rate of the previous compression cycle.

[0032] This invention introduces the entity loss rate from the previous compression cycle and dynamically adjusts the entity density threshold by combining the penalty learning rate and the basic judgment threshold. This achieves adaptive iterative optimization of the threshold parameter, enabling the entity density threshold to be dynamically corrected based on historical compression quality. This further improves the accuracy of window parameter adjustment and the system's adaptability to different text distributions.

[0033] In this embodiment, the method further includes: Record log data for the current compression cycle, including entity retention rate and entity loss rate; Update the corresponding parameters of the named entity recognition model based on the log data.

[0034] This invention achieves closed-loop feedback utilization of compression performance data by recording log data containing entity retention rate and entity loss rate, and updating the corresponding parameters of the named entity recognition model accordingly. This enables the entity recognition model to adaptively optimize based on the performance in the actual compression process, continuously improving the accuracy of entity recognition and the overall performance of the compression system.

[0035] In this embodiment, updating the corresponding parameters of the named entity recognition model based on log data includes: Obtain user feedback data on the compressed text and determine the entity misclassification rate based on the feedback data. When the entity misclassification rate is greater than the preset misclassification rate threshold, trigger incremental learning. During incremental learning, a total loss function is constructed, which includes a cross-entropy loss term, an entity error constraint term, and an incremental learning loss term. The entity error constraint term is used to minimize the error between the compressed text and the entities in the entity list. The incremental learning loss term is based on log data and feedback data to calculate the parameter changes of the named entity recognition model. Preferred, ; Represents the total loss function. Represents cross-entropy loss, This represents the feature vector of the compressed text. This represents the entity error constraint (the set of entity feature vectors that must be retained in the long text data to be processed). This indicates that the behavior of missing entities is subject to a quadratic error penalty. Indicates incremental learning loss. This indicates that the entity maintains the penalty term weight, preferably 0.5. This represents the weight of the incremental learning loss, preferably 0.1; In this embodiment, It is used to measure the difference in basic distribution between compressed text and manually annotated (or expected output). The calculation formula is: Where N represents the total number of tokens (or categories) in the current training sample (or within the current text window). This represents the true label of the r-th token (i.e., the true probability distribution of the expected output); This represents the probability value that the current model predicts the r-th token to be a retained entity; In this embodiment, to avoid catastrophic forgetting of existing domain knowledge while learning new feedback logs, the incremental learning loss combines the fitting error of the new data with the penalty for changes in model parameters, and its calculation formula is as follows: Where K represents the total number of log samples invoked when incremental learning is triggered; This represents the true feedback label of the j-th sample obtained based on feedback data (such as secondary supplements and corrections during customer service troubleshooting). This represents the probability that the j-th token in the current model predicts the entity to be retained. This represents the parameter vector of the named entity recognition model that is currently being updated; This represents the historical parameter vector before the update; This is the L2 regularization penalty term for parameter differences; This is the catastrophic forgetting penalty coefficient, used to control the magnitude of model parameter updates, preferably 0.01; The total loss function is minimized using the backpropagation algorithm to update the corresponding parameters of the named entity recognition model.

[0036] This invention achieves a human-machine collaborative model optimization mechanism by introducing user feedback data to determine the entity misclassification rate and trigger incremental learning. By constructing a total loss function that includes a cross-entropy loss term, an entity error constraint term, and an incremental learning loss term, the model update simultaneously considers the collaborative constraints of classification accuracy, entity retention consistency, and parameter stability. While minimizing the error between the compressed text and the original entity list, it controls the magnitude of model parameter changes, achieving a balance between continuous optimization of model performance and stability.

[0037] In this embodiment, the text in the current window is segmented, and the importance weight of each text segment is calculated. A weight matrix is ​​then output, including: Based on a pre-defined domain corpus, the TF-IDF value of each text segment is calculated; The speech data is time-aligned with the text in the current window, and the speech segment corresponding to each text segment is analyzed based on the VADER model to obtain the corresponding speech emotion score. For each text segment, the importance weight of each text segment is calculated based on the corresponding TF-IDF value, speech emotion score, entity feature weight coefficient, speech emotion feature weight coefficient, and indicator function; where, ; This represents the i-th text segment. This represents the importance weight corresponding to the i-th text segment. This represents the TF-IDF value corresponding to the i-th text segment. This represents the indicator function value corresponding to the i-th text fragment (the value is 1 when the i-th text fragment contains any entity from the previously identified entity list, and 0 otherwise). This represents the speech emotion score of the speech segment corresponding to the i-th text segment (preferably ranging from -1 to 1). This represents the entity feature weight coefficient, preferably 2.0. The weighting coefficient for speech emotion features is preferably 0.5; Output a weight matrix based on the importance weights of all text segments.

[0038] In this embodiment, for example, it is based on a domain corpus (intelligent parking dialogue database, 50,000 samples). For the segment "license plate A12345 is present", the TF-IDF value is calculated as follows: TF (Term Frequency): If a word appears once in the current window out of a total of 50 words, TF = 1 / 50 = 0.02.

[0039] IDF (Inverse Document Frequency): In the domain corpus, the number of occurrences of this segment is assumed to be 100 times, and the total number of documents is 50,000. IDF = log(50,000 / 100) ≈ 6.21.

[0040] TF-IDF=TF×IDF=0.02×6.21≈0.1242.

[0041] This invention reflects the domain importance of text segments by calculating TF-IDF values ​​based on a domain corpus, obtains speech emotion scores to reflect speech emotion features through time alignment and VADER model analysis, and combines entity feature weight coefficients, speech emotion feature weight coefficients, and indicator functions for fusion calculation. This achieves the synergistic fusion of text semantic features, domain features, entity features, and speech emotion features, enabling the importance weights to comprehensively reflect the multidimensional information value of text segments, improving the accuracy of weight calculation and the rationality of compression decisions.

[0042] In this embodiment, based on the weight matrix and a preset token balance, the text in the current window is compressed to generate compressed text, including: Based on the weight matrix, sort all text fragments in the current window text according to their importance weight from high to low; Starting with the text segment with the highest importance weight, tokens are accumulated until the accumulated tokens reach a preset token reserve. The accumulated text segments are then retained (the top-k text segments are retained, where k is determined by the preset token reserve; for example, when the preset token reserve is exhausted, the last (selected) text segment corresponds to an importance weight of 2.0, meaning that all top-k text segments with an importance weight ≥ 2.0 are actually extracted). The unretained text segments (i.e., the remaining text segments; for example, if all top-k text segments with an importance weight ≥ 2.0 are actually extracted, then the remaining text segments are all text segments with an importance weight < 2.0) are input into the summarization model for compression to obtain a compressed summary. The retained fragments are concatenated with the compressed summary to generate the compressed text.

[0043] This invention achieves coordinated processing of prioritizing key information retention and intelligently compressing secondary information by sorting according to importance weight and selectively retaining based on a preset token reserve. By splicing the retained fragments with the compressed summary, the compressed text simultaneously contains high-weight original key information and semantic summaries of low-weight content, achieving a balance between original information fidelity and information compression rate optimization. Within a limited token capacity, it maximizes the retention of key information and maintains the coherence of the text.

[0044] like Figure 2 As shown, the present invention also provides a dynamic compression system for long texts based on a restricted window, which includes: The data acquisition module 10 is used to acquire the long text data to be processed and its associated voice data; The entity recognition module 20 is used to perform entity recognition on long text data and enhance the confidence of the recognized entities based on the voice data to obtain an entity list; The text extraction module 30 is used to dynamically determine the window size and sliding step of the current text window based on the entity list through a reinforcement learning model, and extract the current window text from the long text data according to the window size and sliding step. The weight calculation module 40 is used to segment the text in the current window and calculate the importance weight of each text segment, and output a weight matrix. The importance weight is obtained by fusing the text features of the text segment, the matching degree with the entity list, and the acoustic features of the speech data. The text compression module 50 is used to compress the text in the current window based on the weight matrix and the preset token balance, and generate compressed text.

[0045] This invention also provides a computer-readable storage medium, which may be a computer-readable storage medium included in the memory described in the above embodiments; or it may be a standalone computer-readable storage medium not assembled into a device. The computer-readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement... Figure 1 The illustrated method is a dynamic compression method for long text based on a restricted window. The computer-readable storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0046] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments and storage medium embodiments, since they are basically similar to method embodiments, the descriptions are relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0047] Furthermore, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0048] The foregoing description illustrates and describes preferred embodiments of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the inventive concept by means of the foregoing teachings or techniques or knowledge in related fields. Any modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A method for dynamic compression of long text based on a restricted window, characterized in that, Includes the following steps: Acquire the long text data to be processed and its associated speech data; Entity recognition is performed on the long text data, and the confidence of the recognized entities is enhanced based on the speech data to obtain an entity list; Based on the entity list, the window size and sliding step of the current text window are dynamically determined by a reinforcement learning model, and the current window text is extracted from the long text data according to the window size and sliding step. The text in the current window is segmented, and the importance weight of each text segment is calculated, and a weight matrix is ​​output. The importance weight is obtained by fusing the text features of the text segment, the matching degree with the entity list, and the acoustic features of the speech data. Based on the weight matrix and the preset token balance, the text in the current window is compressed to generate compressed text.

2. The method for dynamic compression of long text based on a restricted window according to claim 1, characterized in that, Entity recognition is performed on the long text data, and the confidence of the recognized entities is enhanced based on the speech data to obtain an entity list, including: The long text data is used to perform entity recognition by a named entity recognition model to obtain the text entities and their first confidence scores. Mel frequency cepstral coefficient features are extracted from the speech data; The entity enhancement weights are weighted according to the multimodal fusion weight coefficients and the Mel frequency cepstral coefficients, and then superimposed on the first confidence level to obtain an entity list, which includes the text-enhanced entities after multimodal fusion enhancement and the second confidence level.

3. The method for dynamic compression of long texts based on a restricted window according to claim 2, characterized in that, The window size and sliding step size of the current text window are dynamically determined through a reinforcement learning model, including: Get the initial window size and the average entity retention rate of historical compression interactions; The window size of the current text window is dynamically determined based on the difference between the entity retention rate of the current compression cycle and the average entity retention rate, the initial window size, and the window adjustment coefficient. Calculate the entity density based on the number of text-enhanced entities and the total number of words in the current text window; Obtain an entity density threshold. When the entity density is less than the entity density threshold, calculate the sliding step size based on the current text window size and entity density.

4. The method for dynamic compression of long texts based on a restricted window according to claim 3, characterized in that, The entity density threshold is dynamically adjusted based on the entity loss rate of the previous compression cycle; The dynamic adjustment steps for the entity density threshold include: obtaining the entity loss rate of the previous compression cycle; and calculating the dynamically adjusted entity density threshold based on the entity loss rate, the penalty learning rate, and the basic decision threshold.

5. The method for dynamic compression of long text based on a restricted window according to claim 2, characterized in that, Also includes: Record log data for the current compression cycle, including entity retention rate and entity loss rate; Update the corresponding parameters of the named entity recognition model based on the log data.

6. The method for dynamic compression of long text based on a restricted window according to claim 5, characterized in that, Update the corresponding parameters of the named entity recognition model based on the log data, including: Obtain user feedback data on the compressed text, and determine the entity misclassification rate based on the feedback data. When the entity misclassification rate is greater than a preset misclassification rate threshold, trigger incremental learning. During incremental learning, a total loss function is constructed, which includes a cross-entropy loss term, an entity error constraint term, and an incremental learning loss term. The entity error constraint term is used to minimize the error between the compressed text and the entities in the entity list. The incremental learning loss term calculates the parameter changes of the named entity recognition model based on the log data and feedback data. The total loss function is minimized using the backpropagation algorithm to update the corresponding parameters of the named entity recognition model.

7. The method for dynamic compression of long text based on a restricted window according to claim 1, characterized in that, The text in the current window is segmented into fragments, and the importance weight of each text fragment is calculated. A weight matrix is ​​output, including: Based on a pre-defined domain corpus, the TF-IDF value of each text segment is calculated; The speech data is time-aligned with the text in the current window, and the speech segment corresponding to each text segment is analyzed based on the VADER model to obtain the corresponding speech emotion score. For each text segment, the importance weight of each text segment is calculated based on the corresponding TF-IDF value, speech emotion score, entity feature weight coefficient, speech emotion feature weight coefficient, and indicator function. Output a weight matrix based on the importance weights of all text segments.

8. The method for dynamic compression of long text based on a restricted window according to claim 1, characterized in that, Based on the weight matrix and the preset token balance, the text in the current window is compressed to generate compressed text, including: Based on the weight matrix, all text fragments in the current window text are sorted from high to low according to their importance weight; Starting with the text segment with the highest importance weight, the token count is accumulated until the accumulated token count reaches the preset token reserve. The accumulated text segments are retained, and the unretained text segments are input into the summarization model for compression to obtain a compressed summary. The retained fragments are concatenated with the compressed summary to generate compressed text.

9. A long text dynamic compression system based on a restricted window, characterized in that, include: The data acquisition module is used to acquire the long text data to be processed and its associated voice data; An entity recognition module is used to perform entity recognition on the long text data and enhance the confidence of the recognized entities based on the voice data to obtain an entity list; The text extraction module is used to dynamically determine the window size and sliding step of the current text window based on the entity list through a reinforcement learning model, and extract the current window text from the long text data according to the window size and sliding step. The weight calculation module is used to segment the text in the current window, calculate the importance weight of each text segment, and output a weight matrix; the importance weight is obtained by fusing the text features of the text segment, the matching degree with the entity list, and the acoustic features of the speech data. The text compression module is used to compress the text in the current window based on the weight matrix and the preset token balance, and generate compressed text.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a long text dynamic compression program based on a restricted window, which, when executed by a processor, implements the steps of the long text dynamic compression method based on a restricted window as described in any one of claims 1 to 8.

Images