Text processing method and device, electronic equipment, storage medium and program product

Abstract

The invention discloses a text processing method and device, electronic equipment, a storage medium and a program product. The method comprises the steps that in response to a query request, an original query statement is obtained, a query word set of the original query statement under multiple query dimensions is determined, and a query vector set corresponding to the query word set is determined according to a word vector embedding model; determining initial candidate text blocks in the plurality of original texts according to the query vector set, a hierarchical index tree of the plurality of queriable original texts, a key block set corresponding to the hierarchical index tree and a mapping table; screening out a plurality of target candidate text blocks from the plurality of initial candidate text blocks according to the query word set, generating a text context related to the query word set according to the query word set and the target candidate text blocks, determining a target query result of the original query statement according to the text context, and displaying the target query result, the accuracy of text query is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a text processing method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] In today's information explosion era, efficient and accurate text querying has become an indispensable core function in the face of massive amounts of text data.

[0003] Most related technologies rely on text titles as the basis for queries, which has significant limitations: on the one hand, titles are often too brief or general, failing to fully reflect the main theme of the article; on the other hand, some titles have weak relevance to the actual content or the user's query needs, or even contain discrepancies, resulting in poor semantic matching and retrieval accuracy for title-based queries, easily returning low-relevance results and affecting user experience. Therefore, there is an urgent need for a text query method to improve the accuracy of text retrieval. Summary of the Invention

[0004] This invention provides a text processing method, apparatus, electronic device, storage medium, and program product to solve the problem of low query accuracy caused by relying on text titles for querying in related technologies.

[0005] According to one aspect of the present invention, a text processing method is provided, the method comprising:

[0006] In response to a query request, the original query statement is obtained, the set of query terms for the original query statement under multiple query dimensions is determined, and the set of query vectors corresponding to the set of query terms is determined according to a word vector embedding model, wherein the word vector embedding model is obtained by training a machine learning model based on a set of sample words and their corresponding expected vector sets;

[0007] Initial candidate text blocks from multiple original texts are determined based on the query vector set, a hierarchical index tree of multiple queryable original texts, a key block set corresponding to the hierarchical index tree, and a mapping table. The hierarchical index tree includes a root node, intermediate layer nodes, and leaf nodes. The root node represents a topic summary of the original text, the intermediate layer nodes store semantic summaries of multiple target clusters of the original text, and the leaf nodes store the content of target text blocks within the target clusters. The mapping table stores at least the topic summary of the root node and the semantic summary of the intermediate layer nodes.

[0008] Multiple target candidate text blocks are selected from multiple initial candidate text blocks based on the query term set. A text context related to the query term set is generated based on the query term set and the target candidate text blocks. The target query result of the original query statement is determined based on the text context and the target query result is displayed.

[0009] According to another aspect of the present invention, a text processing apparatus is provided, the apparatus comprising:

[0010] The query vector set determination module is used to respond to a query request, obtain the original query statement, determine the query word set of the original query statement under multiple query dimensions, and determine the query vector set corresponding to the query word set according to the word vector embedding model, wherein the word vector embedding model is obtained by training a machine learning model based on the sample word set and its corresponding expected vector set;

[0011] The initial candidate text block determination module is used to determine initial candidate text blocks from multiple queryable original texts based on the query vector set, a hierarchical index tree of multiple queryable original texts, a key block set corresponding to the hierarchical index tree, and a mapping table. The hierarchical index tree includes a root node, intermediate layer nodes, and leaf nodes. The root node represents a topic summary of the original text, the intermediate layer nodes store semantic summaries of multiple target clusters of the original text, and the leaf nodes store the content of target text blocks within the target clusters. The mapping table stores at least the topic summary of the root node and the semantic summary of the intermediate layer nodes.

[0012] The target query result determination module is used to filter multiple target candidate text blocks from multiple initial candidate text blocks according to the query term set, generate a text context related to the query term set according to the query term set and the target candidate text blocks, determine the target query result of the original query statement according to the text context, and display the target query result.

[0013] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0014] At least one processor; and

[0015] A memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the text processing method according to any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the text processing method described in any embodiment of the present invention.

[0018] According to another aspect of the present invention, embodiments of this disclosure also provide a computer program product, including a computer program that, when executed by a processor, implements the text processing method as described in any of the embodiments of this disclosure.

[0019] The technical solution of this invention firstly involves obtaining the original query statement in response to a query request, determining the set of query terms for the original query statement across multiple query dimensions, and determining the set of query vectors corresponding to the set of query terms based on a word vector embedding model. The word vector embedding model is trained on a machine learning model based on a set of sample words and their corresponding expected vector sets. Multi-dimensional analysis of the original query statement fully explores the potential information needs of the query request, improving the comprehensiveness and robustness of semantic understanding. The generation of query vectors based on the word vector embedding model ensures the accuracy and consistency of semantic representation. Next, initial candidate text blocks in multiple original texts are determined based on the set of query vectors, a hierarchical index tree of multiple queryable original texts, a set of key blocks corresponding to the hierarchical index tree, and a mapping table. The hierarchical index tree includes a root node, intermediate nodes, and leaf nodes. The root node represents the topic summary of the original text, the intermediate nodes store the semantic summaries of multiple target clusters of the original text, and the leaf nodes store the content of the target text blocks in the target clusters. The mapping table stores at least the root node. The topic summary of the points and the semantic summary of the intermediate layer nodes; hierarchical index tree realizes hierarchical organization and efficient navigation of text content, significantly improving retrieval efficiency. Combining semantic summary and position information in the mapping table, it can accurately trace the context source of the initial candidate text blocks while preserving semantic integrity, providing structured support for subsequent accurate filtering; finally, multiple target candidate text blocks are filtered from multiple initial candidate text blocks according to the query term set, and text context related to the query term set is generated according to the query term set and the target candidate text blocks. The target query result of the original query statement is determined according to the text context and the target query result is displayed; a second filtering is performed on the initial candidate text blocks, and fine-grained matching is performed in combination with the query term set to effectively filter irrelevant or weakly related segments, improve the relevance and accuracy of the query results, and use the target candidate text blocks to dynamically generate context information associated with the query term set to improve the accuracy of the query results; dual accurate positioning at the topic level and segment level for massive text content is achieved, significantly improving the accuracy, efficiency and readability of text query.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of a text processing method provided according to Embodiment 1 of the present invention;

[0023] Figure 2a This is a flowchart of a text processing method provided according to Embodiment 2 of the present invention;

[0024] Figure 2b This is a schematic diagram of a text processing method provided according to Embodiment 2 of the present invention;

[0025] Figure 3 This is a schematic diagram of the structure of a text processing device according to Embodiment 3 of the present invention;

[0026] Figure 4 This is a schematic diagram of the structure of an electronic device that implements the text processing method of the present invention. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first," "second," "initial," "target," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0030] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0031] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0032] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0033] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0034] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0035] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0036] Example 1

[0037] Figure 1 This is a flowchart of a text processing method provided in Embodiment 1 of the present invention. This embodiment is applicable to text querying. The method can be executed by a text processing device, which can be implemented in hardware and / or software, optionally through an electronic device, such as a mobile terminal, PC, or server. Figure 1 As shown, the method may specifically include:

[0038] S110. In response to the query request, obtain the original query statement, determine the query term set of the original query statement under multiple query dimensions, and determine the query vector set corresponding to the query term set according to the word vector embedding model, wherein the word vector embedding model is obtained by training a machine learning model based on the sample word set and its corresponding expected vector set.

[0039] In this embodiment of the invention, a query request refers to a request to obtain specific information, which can be presented in the form of natural language. For example, a query request may include various forms such as input questions, keyword combinations, sentences, or paragraphs to reflect the information query needs. The original query statement is the raw text content used when issuing the query request without any processing. Query dimensions can be classification angles from which information is parsed and extracted from the original query statement, such as semantic dimensions, topic dimensions, etc.

[0040] A query term set refers to a collection of words extracted from the original query statement based on multiple query dimensions that reflect the information requirements of the query. Optionally, the query term set for multiple query dimensions can be determined based on the original query statement and the text processing model to improve the efficiency of query term set extraction. The text processing model is trained on a machine learning model based at least on sample query statements and their corresponding expected term sets.

[0041] Optionally, multiple query dimensions can be pre-determined, and several sample query statements can be collected and labeled with the expected word sets corresponding to the multiple query dimensions. Then, the multiple sample query statements are input into the machine learning model, and the loss is calculated by comparing the labeled expected word sets with the model output. The model parameters are updated based on backpropagation. Finally, through iterative optimization and model evaluation, a text processing model with multi-dimensional semantic parsing capabilities is trained to achieve fine-grained semantic understanding of query statements and extraction of query words.

[0042] After obtaining the query term set, a word embedding model can be used to process the query term set, transforming each query term into a vector that represents its semantics, resulting in a query vector set corresponding to the query term set. The query vector set can be understood as a combination of vectors formed after transforming each query term in the query term set through a word embedding model. The query vector set is the semantic representation of the query term set in vector space.

[0043] Optionally, the word embedding model can be trained on a machine learning model based on a set of sample words and their corresponding expected vector sets. The expected vector set corresponds one-to-one with each sample word in the sample word set and is the set of vectors expected to be output by the model. This set can be used as target output data to adjust model parameters during model training. The word embedding model obtained through training can transform query words into vectors that can represent their semantic information, realizing the mapping from words to vector space.

[0044] Alternatively, the word embedding model can be a Bidirectional Encoder Representations from Transformers (BERT) model.

[0045] S120. Based on the query vector set, the hierarchical index tree of multiple queryable original texts, the key block set corresponding to the hierarchical index tree, and the mapping table, initial candidate text blocks in multiple original texts are determined. The hierarchical index tree includes a root node, intermediate layer nodes, and leaf nodes. The root node represents the topic summary of the original text. The intermediate layer nodes store the semantic summaries of multiple target clusters of the original text. The leaf nodes store the content of the target text blocks in the target clusters. The mapping table stores at least the topic summary of the root node and the semantic summary of the intermediate layer nodes.

[0046] The original text that can be queried can be pre-stored text information that can be searched. A hierarchical index tree can be understood as a tree structure used to organize and index original text. A hierarchical index tree can be composed of a root node, intermediate level nodes, and leaf nodes, which are used to store text information at different levels, enabling progressive retrieval from coarse-grained to fine-grained.

[0047] Optionally, the root node is the top-level node of the hierarchical index tree, which can be used to store the topic summary of the original text, highly summarizing the core semantics of the entire text, reflecting the main content and topic direction of the entire text, and providing a basis for preliminary judgment of text relevance. The intermediate-level nodes can be used to store semantic summaries of multiple target clusters of the original text. The semantic summary can be a concise text formed by summarizing and refining the content of the target cluster, which can concisely reflect the core semantics and main content of the target cluster. The leaf nodes are the bottom-level nodes of the hierarchical index tree, which can be used to store the content of the target text blocks in the target cluster. The target cluster can be a set formed by clustering the target text blocks in the original text according to similarity. The target text block can be a text fragment that can express a certain semantics after being segmented and processed from the original text, which can be used as the basic unit to constitute the target cluster. The target text blocks within the same cluster have high correlation, which facilitates targeted retrieval and processing.

[0048] Optionally, the mapping table stores at least the topic summary of the root node and the semantic summary of the intermediate layer nodes.

[0049] The key block set refers to the collection of key text blocks from multiple target clusters in the hierarchical index tree. Key text blocks represent the core content of each target cluster and can be determined by calculating the average similarity within each cluster, thus helping to improve query efficiency and accuracy. Initial candidate text blocks refer to text blocks that are preliminarily selected from multiple original texts based on the query vector set, hierarchical index tree, key block set, and mapping table, and are potentially related to the query information.

[0050] As an optional technical solution in an embodiment of the present invention, optionally, determining initial candidate text blocks from multiple original texts based on the query vector set, a hierarchical index tree of multiple queryable original texts, a key block set corresponding to the hierarchical index tree, and a mapping table includes: for each queryable original text, obtaining the topic summary of the root node of the hierarchical index tree from the mapping table corresponding to the hierarchical index tree of the original text; determining multiple candidate texts from the multiple original texts based on the query vector set and the multiple topic summaries; for each candidate text, obtaining the key block set of the hierarchical index tree from the mapping table corresponding to the hierarchical index tree of the candidate text. The semantic summary of the intermediate layer nodes is used to determine initial candidate clusters from the target clusters of the candidate texts based on the semantic summary of the candidate texts and the query vector set. Multiple target candidate clusters are determined from the multiple initial candidate clusters of the candidate texts based on the intermediate layer nodes of the hierarchical index tree of the multiple candidate texts, the key block set corresponding to the hierarchical index tree, and the mapping table. The key block set is used to store key text blocks of the multiple target clusters. For each target candidate cluster, multiple initial candidate text blocks matching the query vector set are determined from the multiple target text blocks of the target candidate cluster based on the target candidate cluster and its corresponding mapping table.

[0051] Specifically, for each queryable original text, the root node of the hierarchical index tree can be determined from the mapping table corresponding to the hierarchical index tree of the original text, and the topic summary of the original text stored therein can be extracted. Then, based on the query vector set, similarity calculations are performed between the query vector set and the topic summaries corresponding to multiple original texts to determine multiple candidate texts whose topic summary similarity is greater than a first preset threshold. The first preset threshold can be flexibly set based on actual business needs.

[0052] Furthermore, for each candidate text, the intermediate layer nodes of the hierarchical index tree and their stored semantic summaries can be obtained from the mapping table corresponding to the hierarchical index tree of the candidate text. Similarity matching is then performed based on the semantic summaries corresponding to each target cluster in the candidate text and the query vector set to determine multiple initial candidate clusters from the target clusters of the candidate text whose semantic summary similarity is greater than a second preset threshold. The second preset threshold can be flexibly set based on actual business needs.

[0053] Based on the above scheme, the mapping table may also store text embedding vectors of multiple key text blocks of the original text. Optionally, determining multiple target candidate clusters from multiple initial candidate clusters of the candidate texts based on the intermediate layer nodes of the hierarchical index tree of the multiple candidate texts, the key block set corresponding to the hierarchical index tree, and the mapping table includes: for each initial candidate cluster, determining the key text blocks of the initial candidate cluster according to the key block set corresponding to the hierarchical index tree, as the text blocks to be matched; obtaining the text embedding vector of the text blocks to be matched from the mapping table corresponding to the hierarchical index tree, as the embedding vector to be matched; determining the average similarity between multiple query vectors in the query vector set and the embedding vector to be matched; and determining multiple target candidate clusters that match the query vector set from the multiple initial candidate clusters based on the average similarity corresponding to the multiple initial candidate clusters.

[0054] In this context, the text block to be matched refers to the key text block in the initial candidate cluster. The embedding vector to be matched is the text embedding vector corresponding to the text block to be matched, which can represent the semantic information of the text block to be matched. The query vector is any vector in the query vector set, which corresponds one-to-one with the query terms in the query term set, and is the vector representation of the query terms. The average similarity is the average of the similarities between multiple query vectors in the query vector set and the embedding vector to be matched.

[0055] Specifically, for each initial candidate cluster, the key text block in the initial candidate cluster is obtained through the key block set corresponding to its hierarchical index tree, and the key text block is used as the text block to be matched. Then, the text embedding vector corresponding to the text block to be matched is obtained through the mapping table corresponding to the hierarchical index tree, which stores the text embedding vectors of multiple key text blocks of the original text, and is used as the embedding vector to be matched to represent the semantics of the corresponding initial candidate cluster.

[0056] Furthermore, the similarity between each query vector in the query vector set and each embedded vector to be matched can be calculated separately, and the average of these similarities can be taken to obtain the average similarity corresponding to each initial candidate cluster.

[0057] The similarity between each query vector and the embedding vector to be matched corresponding to each initial candidate cluster can be calculated based on the following formula. :

[0058] ;

[0059] in, For the query vector set of the first One query vector; For the first Initial candidate clusters The first included The embedding vectors to be matched corresponding to each key text block.

[0060] Furthermore, each initial candidate cluster can be calculated based on the following formula. Corresponding average similarity :

[0061] ;

[0062] in, To query the size of the vector set; This indicates selecting the query vector from the initial candidate cluster. The embedding vector to be matched corresponding to the key text block with the highest similarity.

[0063] Finally, the average similarity scores of the initial candidate clusters are compared and analyzed. Based on the analysis results, several clusters with the highest matching degree to the query vector set are selected from the initial candidate clusters as target candidate clusters. The number of target candidate clusters can be flexibly set according to actual business needs.

[0064] By using the average similarity between the text embedding vectors of key text blocks in the key block set and the query vector set as the screening criterion, the overall semantic matching degree between the initial candidate clusters and the query information requirements can be effectively measured. By sorting and screening target candidate clusters based on semantic similarity, the relevance and representativeness of the candidate results are improved, interference from low-relevance clusters is avoided, and the retrieval accuracy is further optimized.

[0065] Optionally, the mapping table can store the text positions and text distances of target text blocks in multiple target clusters. After determining the target candidate clusters, based on the text positions and relative distances of target text blocks in multiple target clusters stored in the mapping table, multiple text blocks whose relative distance to the key text block is less than a third preset threshold can be selected from the multiple target text blocks in the target candidate clusters. The selected text blocks and the key text block are then determined together as initial candidate text blocks. The third preset threshold can be flexibly set according to actual business needs.

[0066] By using hierarchical filtering, candidate texts and initial candidate clusters are located step by step from the topic summary of the root node to the semantic summary of the intermediate layer, which significantly narrows the search scope and improves query efficiency. The multi-level structure of the hierarchical index tree is used for top-down matching, which achieves accurate transmission of semantic relevance.

[0067] S130. Select multiple target candidate text blocks from multiple initial candidate text blocks according to the query term set, generate a text context related to the query term set according to the query term set and the target candidate text blocks, determine the target query result of the original query statement according to the text context, and display the target query result.

[0068] In this context, the target candidate text block refers to a text block that is further filtered from the initial candidate text blocks based on the query term set, and is more closely related to the query term set, thus more closely resembling the query requirements. The text context refers to the preceding and following text content associated with the target candidate text block. The target query result can be the final result determined based on the text context analysis, which satisfies the information requirements expressed in the original query statement.

[0069] Optionally, multiple target candidate text blocks can be selected from multiple initial candidate text blocks based on the query term set. This can be achieved through statistical methods such as Term Frequency-Inverse Document Frequency (TF-IDF). First, the importance of words in the text is evaluated, and the weights corresponding to words with high discriminative power in the initial candidate text blocks are increased. Then, the similarity between the vectors corresponding to the increased initial candidate text blocks and the query term set is calculated again, and multiple text blocks with a similarity greater than a fourth preset threshold are selected as target candidate text blocks. The fourth preset threshold can be flexibly set according to actual business needs.

[0070] After obtaining the target candidate text block, a text context related to the query term set can be generated based on the target candidate text block and the remaining initial candidate text blocks. This text context can assist in a comprehensive and accurate understanding of the text's meaning and background.

[0071] Optionally, determining the target query result of the original query statement based on the text context and displaying the target query result includes: inputting the text context and the query term set into a multimodal model to obtain preliminary query results corresponding to the original query statement, and adjusting the preliminary query results based on the query term set to obtain the target query result of the original query statement.

[0072] Specifically, the text context and query term set can be input into a multimodal model. The frequency of words in the text context can be counted by term frequency (TF), and the uniqueness of words in the overall text can be evaluated by inverse document frequency (IDF). This allows for the selection of core words with high discriminative power. Furthermore, semantic rules such as sentence coherence can be used to filter out irrelevant or redundant words. The results are then integrated to form preliminary query results corresponding to the original query statement. This ensures that the data contained in the results has high semantic value, laying the foundation for subsequent optimization.

[0073] Based on this, the preliminary query results can be adjusted in at least one of the following ways to obtain more accurate target query results: Leveraging the structural characteristics of hierarchical index trees, prioritize retrieving key text blocks from the target candidate clusters that are most relevant to the query term set and have the highest similarity. Utilize the representativeness of the key text blocks to the core content of the cluster to enhance the matching degree between the results and the query information. Combine the text location information recorded in the mapping table to accurately locate the specific source of the key text blocks in the original text, ensuring a close correlation between the retrieved text blocks and the query terms in terms of content, and reducing interference from irrelevant information. For specific scenarios, set up corresponding professional terminology weight enhancement mechanisms, that is, assign higher weights to specific terms and concepts involved in the current scenario to improve the recognition accuracy and proportion of scenario-related information in the results.

[0074] After adjustments, target query results that accurately meet the information requirements of the original query statement are generated and displayed.

[0075] By analyzing the text context, content that accurately meets the information requirements expressed in the original query can be extracted and integrated to form the target query results. These results are then displayed in a way that not only meets the core query requirements but also has sufficient contextual support, thereby improving the accuracy and usability of the query results.

[0076] The technical solution of this invention firstly involves obtaining the original query statement in response to a query request, determining the set of query terms for the original query statement across multiple query dimensions, and determining the set of query vectors corresponding to the set of query terms based on a word vector embedding model. The word vector embedding model is trained on a machine learning model based on a set of sample words and their corresponding expected vector sets. Multi-dimensional analysis of the original query statement fully explores the potential information needs of the query request, improving the comprehensiveness and robustness of semantic understanding. The generation of query vectors based on the word vector embedding model ensures the accuracy and consistency of semantic representation. Next, initial candidate text blocks in multiple original texts are determined based on the set of query vectors, a hierarchical index tree of multiple queryable original texts, a set of key blocks corresponding to the hierarchical index tree, and a mapping table. The hierarchical index tree includes a root node, intermediate nodes, and leaf nodes. The root node represents the topic summary of the original text, the intermediate nodes store the semantic summaries of multiple target clusters of the original text, and the leaf nodes store the content of the target text blocks in the target clusters. The mapping table stores at least the root node. The topic summary of the points and the semantic summary of the intermediate layer nodes; hierarchical index tree realizes hierarchical organization and efficient navigation of text content, significantly improving retrieval efficiency. Combining semantic summary and position information in the mapping table, it can accurately trace the context source of the initial candidate text blocks while preserving semantic integrity, providing structured support for subsequent accurate filtering; finally, multiple target candidate text blocks are filtered from multiple initial candidate text blocks according to the query term set, and text context related to the query term set is generated according to the query term set and the target candidate text blocks. The target query result of the original query statement is determined according to the text context and the target query result is displayed; a second filtering is performed on the initial candidate text blocks, and fine-grained matching is performed in combination with the query term set to effectively filter irrelevant or weakly related segments, improve the relevance and accuracy of the query results, and use the target candidate text blocks to dynamically generate context information associated with the query term set to improve the accuracy of the query results; dual accurate positioning at the topic level and segment level for massive text content is achieved, significantly improving the accuracy, efficiency and readability of text query.

[0077] Example 2

[0078] Figure 2 is a flowchart of a text processing method provided in Embodiment 2 of the present invention, which further describes the implementation of constructing a hierarchical index tree and a mapping table. Optionally, before determining the initial candidate text blocks in the multiple original texts based on the query vector set, the hierarchical index tree of multiple queryable original texts, the key block set corresponding to the hierarchical index tree, and the mapping table, the method further includes: obtaining the original text; performing word segmentation on the original text to obtain multiple initial words; determining multiple effective words from the multiple initial words; and dividing the original text into multiple initial text blocks based on the position of the effective words in the original text; wherein each initial text block contains at least one effective word; performing preliminary splitting of the initial text blocks according to text delimiters to obtain multiple clauses; determining target text blocks based on the multiple clauses; determining the text embedding vector of the target text blocks; constructing a feature vector set of the original text based on the text embedding vectors of the multiple target text blocks; recording the text embedding vectors in the feature vector set using a Gaussian mixture model clustering algorithm to obtain multiple target clusters; constructing a hierarchical index tree based on the multiple target clusters; and determining the key block set and mapping table corresponding to the hierarchical index tree. For specific implementation details, please refer to the description of this embodiment. Technical features that are the same as or similar to those in the foregoing embodiments will not be repeated here. As shown in Figure 2, the method may specifically include:

[0079] S210. Obtain the original text, perform word segmentation on the original text to obtain multiple initial words of the original text, determine multiple valid words from the multiple initial words, and divide the original text into multiple initial text blocks according to the position of the valid words in the original text; wherein each initial text block contains at least one of the valid words.

[0080] In this embodiment of the invention, initial words refer to the preliminary set of words obtained after segmenting the original text, which may contain redundant words such as function words or stop words that have no practical meaning. Effective words refer to words that are semantically related to the original text and are selected after removing redundant words from the initial words; these words can reflect the core content of the original text. Initial text blocks can be multiple text segments formed by dividing the original text according to the positional distribution of effective words within the original text. Each initial text block contains at least one effective word.

[0081] Optionally, the original text can be obtained and segmented into multiple independent initial words according to word boundaries, grammatical rules, and other word segmentation methods, providing a basis for subsequent extraction of effective words.

[0082] Based on the above scheme, optionally, determining multiple effective words from multiple initial words includes: for each initial word, determining the word frequency, inverse text frequency, and type of the initial word; if the type is a common word, determining the target weight of the initial word based on the word frequency and the inverse text frequency; and if the type is a specialized word, determining the semantic importance score and a preset enhancement coefficient of the initial word; determining the target weight of the initial word based on the word frequency, the inverse text frequency, the semantic importance score, and the preset enhancement coefficient; determining the average weight of the target weights of multiple initial words; determining keywords from multiple initial words based on the target weights and the average weights of the initial words; determining the semantic vector of each keyword; and determining multiple effective words from multiple keywords based on the semantic vector of the keywords.

[0083] Term frequency (TF) refers to the number of times an initial word appears in the original text, and it can reflect the importance of the word in the text to some extent. Inverse text frequency (IPF) is an indicator that measures the importance of an initial word across multiple texts. IPF is related to the number of texts in which a word appears; the fewer texts in which a word appears, the higher its IPF.

[0084] Optionally, the type of initial words can be a categorization of initial words. The types of initial words can include various categories such as general vocabulary and specialized vocabulary. General vocabulary can be understood as words widely used in everyday language, not limited to a specific professional field; their target weights are mainly determined based on word frequency and inverse text frequency. Specialized vocabulary can be understood as words used within a specific professional field, possessing specific meanings and potentially significant semantic value within that field; their target weights are calculated by considering various factors related to that specific professional field. The target weights can be calculated using the corresponding calculation methods for different types of initial words, and can be used to quantify the numerical importance of initial words in the text.

[0085] The target weights are calculated differently for different types of initial words.

[0086] Optionally, the original text can be scanned to obtain the occurrence count of each initial word in the original text, and the occurrence count can be matched with each initial word to obtain the term frequency (TF) corresponding to each initial word.

[0087] The term frequency (TF) of each initial word can be calculated using the following formula:

[0088] ;

[0089] in, These are initial words; For the original text

[0090] Optionally, a pre-built raw text library can be obtained, and the occurrence count of each initial word in the raw text library can be obtained. The occurrence count is then matched with each initial word to calculate the inverse document frequency (IDF) for each initial word. The raw text library can be built through various methods, such as uploading document text or accessing data sources.

[0091] The inverse document frequency (IDF) of each initial word can be calculated using the following formula:

[0092] .

[0093] In the case where the type is technical terminology, technical terms can be identified and weighted to determine the semantic importance score of the initial words. and preset enhancement coefficient The semantic importance score can be a score that assesses the semantic importance of specialized terms within their respective professional fields. The preset enhancement coefficient can be a pre-set coefficient used to enhance the target weight of specialized terms, highlighting their importance; its magnitude can be set according to actual business needs, for example... .

[0094] Optionally, if the type is ordinary vocabulary, the target weight of the initial word can be determined based on the word frequency and the inverse text frequency.

[0095] The target weights for common words can be calculated using the following formula. :

[0096] ;

[0097] .

[0098] Optionally, when the type is specialized vocabulary, the target weight of the initial word can be determined based on the word frequency, the inverse text frequency, the semantic importance score, and the preset enhancement coefficient.

[0099] The target weights corresponding to specialized terms can be calculated based on the following formula.

[0100] .

[0101] Furthermore, the average weight of the target weights for multiple initial words can be calculated based on the following formula:

[0102] ;

[0103] in, for The average weight of each initial word; The original text was segmented into The target weights corresponding to each initial word are as follows: .

[0104] After obtaining the average weight of the target weights, it can be used as a reference threshold to filter the initial words. Initial words whose target weights are greater than or equal to the average weight are identified as keywords. Keywords refer to words selected based on the target weights and average weights of the initial words, whose corresponding target weights are greater than the average weight, and which have above-average importance in the original text.

[0105] Optionally, for each keyword, the sentence in which it appears in the original text can be extracted, and semantic analysis can be performed on the sentence containing each keyword, such as parsing the grammatical structure of the sentence, understanding the core meaning expressed by the sentence and the contextual relationship, etc., to obtain the semantic analysis results and generate word semantic vectors corresponding to the semantic information of each keyword in a specific context, so as to determine multiple effective words from multiple keywords based on the word semantic vectors of the keywords.

[0106] By employing a differentiated weighting strategy for different types of initial words, the actual semantics of the words can be captured more accurately. This ensures that the selected effective words more accurately reflect the key information of the original text, providing a more reliable basis for subsequent text block segmentation and processing, and improving the accuracy and semantic representativeness of word selection.

[0107] Optionally, determining effective words from a plurality of keywords based on the semantic vectors of the keywords includes at least one of the following: determining effective words from a plurality of keywords based on the vector similarity between the semantic vectors of the words of the keywords; extracting the sentence containing the keyword from the original text, determining the sentence semantic vector of the sentence, and selecting effective words from a plurality of keywords based on the vector similarity between the semantic vectors of the words and the sentence semantic vectors; extracting the context information of the keyword from the original text based on a context window, determining the context semantic vector of the context information, and selecting effective words from a plurality of keywords based on the vector similarity between the semantic vectors of the words and the context semantic vectors. The process involves: selecting effective words from the vocabulary; extracting the target paragraph containing the keywords from the original text; determining the summary information of the target paragraph and the summary semantic vector of the summary information; selecting effective words from multiple keywords based on the vector similarity between the word semantic vector and the summary semantic vector; determining the word embedding vector of each keyword based on a target word embedding model; selecting effective words from multiple keywords based on the vector similarity between the word embedding vector and the word semantic vector, wherein the target word embedding model is trained based on sample words and the expected embedding vectors of the sample words in the target domain; and selecting multiple effective words with a vector similarity greater than a fifth preset threshold. The fifth preset threshold can be flexibly set according to actual business needs.

[0108] After identifying multiple valid words, the original text can be divided into multiple initial text blocks based on the position of the valid words in the original text, and each initial text block contains at least one of the valid words.

[0109] S220. The initial text block is initially split according to the text delimiter to obtain multiple clauses. The target text block is determined according to the multiple clauses, and the text embedding vector of the target text block is determined. The feature vector set of the original text is constructed based on the text embedding vector of the multiple target text blocks.

[0110] Here, a text delimiter refers to a symbol or marker used to segment text. The text delimiter may include at least one of the following: period, semicolon, and newline character. A clause can be a sentence fragment obtained after initially splitting the initial text block using text delimiters. A text embedding vector is a vector obtained by transforming the target text block using a preset model. The text embedding vector can represent the semantic information of the target text block in a vector space, facilitating subsequent processing such as clustering. A feature vector set is a collection composed of the text embedding vectors of multiple target text blocks in the original text.

[0111] Optionally, the initial text block can be initially split using delimiters to obtain multiple clauses.

[0112] Optionally, after obtaining multiple clauses, determining a target text block based on the multiple clauses and determining the text embedding vector of the target text block includes: extracting the clause semantic vector of each clause based on a target language model; determining the cosine similarity between each pair of clauses based on the clause semantic vectors of the multiple clauses; merging the multiple clauses based on the cosine similarity between the clauses, a preset similarity threshold, and the maximum value of a preset text processing unit to obtain at least one target text block; determining the word embedding vector, segment embedding vector, and position embedding vector of the target text block based on a bidirectional encoding model; determining the model input vector based on the word embedding vector, the segment embedding vector, and the position embedding vector; inputting the model input vector into a multilayer encoder to obtain the text embedding vector of the target text block; and constructing a feature vector set of the original text based on the text embedding vectors of the multiple target text blocks. By dynamically merging clauses using cosine similarity, the system can flexibly construct target text blocks of optimal granularity while maintaining semantic coherence, avoiding information fragmentation or over-aggregation. By combining word, segment, and position embedding vectors and generating the final text embedding vector through a multi-layer encoder, it can comprehensively capture local and global semantic features, significantly improving the quality of text representation.

[0113] The target language model refers to a model specifically designed for processing natural language, capable of understanding clause semantics and converting it into vector representations. Clause semantic vectors, reflecting the semantics of clauses, can be obtained through the target language model. Optionally, the target language model can be a BERT model. Clause semantic vectors are vectors obtained after processing each clause using the target language model. Clause semantic vectors can be used to represent the semantic content of clauses in a vector space, facilitating the calculation of similarity between clauses.

[0114] Specifically, the semantic vector of each clause can be extracted based on the target language model, and the cosine similarity between each pair of clauses can be calculated based on the semantic vectors of the multiple clauses to evaluate the semantic relevance between the two clauses.

[0115] Furthermore, multiple clauses with a cosine similarity higher than a preset similarity threshold can be merged into a single paragraph to obtain at least one target text block. If the cosine similarity between a clause and multiple clauses is higher than the preset similarity threshold, it is preferentially merged with the clause that has the highest cosine similarity, i.e., the one that is semantically closest. Simultaneously, during the aggregation process, the number of text processing units within the target text block is monitored. When the number of text processing units exceeds the maximum value, the target text block is further split; when the number of text processing units is lower than the maximum value, adjacent target text blocks can be merged. The preset maximum value for text processing units refers to the maximum number of clauses that each target text block can contain. The maximum value for text processing units can be used to control the size of the target text block, keeping it within a suitable processing range.

[0116] Optionally, changes in the semantic vector of a clause or specific keywords can be detected to identify changes in the core theme of the clause, thereby initiating the merging of the next target text block.

[0117] Based on the above scheme, the indices of the start and end text processing units within each target text block, as well as the semantic summary of the target text block, can be recorded to provide basic information for subsequent processing.

[0118] Furthermore, word embedding, segment embedding, and position embedding can be sequentially performed on the target text block based on a bidirectional encoding model. This allows for a comprehensive understanding of the text semantics, determining the word embedding vector, segment embedding vector, and position embedding vector of the target text block. These vectors are then element-wise summed to obtain the model input vector. The word embedding vector is the vector obtained after processing each word in the target text block using the bidirectional encoding model, representing the semantic information of words in a specific context. The segment embedding vector is used in the bidirectional encoding model to identify different target text blocks, enabling the model to distinguish between them and avoid semantic confusion. The position embedding vector is used in the bidirectional encoding model to represent the position of words in the target text block, allowing the model to understand word position information for better understanding of text syntax and semantic structure. The first text processing unit in the target text block can be used to represent the position of that target text block.

[0119] Specifically, the first can be determined based on the following formula. The model input vector for each target text block:

[0120] ;

[0121] in, The input vector for the model; For the first The first target text block Word embedding vectors of 1 word; For the first The segment embedding vector of the target text block containing each word; For the first Position embedding vectors of each target text block; An index for each target text block.

[0122] Optionally, the model input vector can be input into a multi-layer encoder, where each layer contains a self-attention mechanism algorithm and a feedforward neural network algorithm. By performing different linear transformations on the model input vector, the query, key, and value matrices are obtained, allowing the model to capture multi-dimensional associations in parallel.

[0123] Specifically, the attention weights at different positions can be calculated by first performing a linear transformation based on the following formula using a self-attention mechanism, and then weighted to obtain the output matrix:

[0124] ;

[0125] ;

[0126] ;

[0127] in, This is the weight matrix, used to weight the model input vector. The linear transformation results in a query, key, and value matrix, with dimensions of [dimensions to be filled in]. , Input vectors for the model Dimensions For each attention head dimension, optionally, , To focus on the number of heads, in order to The model input vector of dimension is split into One's attention is on one's head; This represents the combined model input vector, that is... ; These represent the Query, Key, and Value matrices, respectively.

[0128] Furthermore, the query, key, and value matrix can be divided into... There are 1 attention head, and each attention head has 1 dimension. The value matrix is ​​weighted to obtain the output calculated by the self-attention mechanism. .

[0129] The following formula can be used to calculate :

[0130] ;

[0131] in, The dot product of the query matrix and the key matrix represents the attention score between each word pair; This is the scaling factor. It can be set by... It can stabilize training and prevent the dot product result from being too large, which would cause the softmax gradient to vanish; softmax is an activation function that can convert attention scores into a probability distribution, representing the relative importance of different positions.

[0132] Next, the self-attention output is further processed using a feedforward neural network algorithm to capture more complex linear relationships.

[0133] Specifically, the feedforward neural network algorithm can be calculated based on the following formula:

[0134] ;

[0135] Where ReLU is the activation function: This is used to introduce nonlinearity so that multilayer encoder models can learn complex relationships; This is the output of the feedforward neural network;

[0136] The formula for linear layer transformation is: The feedforward neural network consists of two linear transformation layers and an intermediate ReLU activation layer. The first linear transformation layer is... Then it is activated via ReLU. The negative numbers are set to 0, while the integers are retained, introducing non-linearity; finally, a second-level linear transformation is performed. ; This is the weight matrix. This is the weight matrix for the two-layer linear transformation; The input vector is typically the input vector of a feedforward neural network, and usually the output vector of a self-attention mechanism network. ; For bias vectors, This is the bias term for the two-level linear transformation.

[0137] In the multi-layer encoder process, the output of each layer encoder will be used as the input of the next layer encoder. After processing by the multi-layer encoder, the input vector representation of the last layer is obtained. The output vector corresponding to each target text block is extracted according to the vector extraction algorithm and marked as the text embedding vector corresponding to the target text block. The feature vector set of the original text is constructed based on the text embedding vectors of multiple target text blocks.

[0138] Through multi-layer encoder computation, deep feature extraction and semantic encoding can be performed on the model input vector, ultimately generating a text embedding vector that can comprehensively represent the semantics of the target text block.

[0139] S230. The Gaussian mixture model clustering algorithm is used to record the text embedding vectors in the feature vector set to obtain multiple target clusters. A hierarchical index tree is constructed based on the multiple target clusters, and the key block set and mapping table corresponding to the hierarchical index tree are determined.

[0140] Among them, the Gaussian Mixture Model (GMM) clustering algorithm is a probability-based clustering algorithm that assumes that the data comes from a mixture of multiple Gaussian distributions and is used to cluster text embedding vectors in the feature vector set into multiple target clusters with similar features.

[0141] Before obtaining the target cluster, a feature vector set can be extracted. Each text feature vector It is a d-dimensional vector, and the number of clusters M is set, which is the number of Gaussian distributions in the Gaussian mixture model. The Gaussian mixture model parameters and mixing weights are also set. , used to indicate the first The weights of a Gaussian distribution can satisfy... and Mean vector , used to indicate the first Mean of a Gaussian distribution; mean variance matrix , used to indicate the first The covariance matrix of a Gaussian distribution is calculated. Simultaneously, the parameters are initialized with respect to the mean vector. Random selection is possible Using 10 data points as initial values, for the covariance matrix... It can be initialized as an identity matrix or the global covariance matrix of the data; for mixed weights It can be initialized as a uniform analysis, i.e. .

[0142] Optionally, for different numbers of clusters M, the BIC of the Gaussian mixture model can be calculated using the Bayesian Information Criterion (BIC), and the value of M that minimizes the BIC can be selected to balance the complexity and fit of the Gaussian mixture model.

[0143] Optionally, the Expectation-Maximization Algorithm (EM) is used to gradually approximate the optimal solution of the model parameters by alternately executing the expectation step (E step) and the maximization step (M step), thereby estimating the parameters and determining the probability that each data point, i.e., the target text block, belongs to each cluster.

[0144] Specifically, the desired step can be represented by the following formula:

[0145] ;

[0146] in, For each target text block Belongs to each cluster The posterior probability; It is the probability density function of a Gaussian distribution; This is used to enumerate all clusters.

[0147] ;

[0148] The mixture weights, mean vector, and covariance matrix can be updated by maximizing the step:

[0149] ;

[0150] ;

[0151] ;

[0152] in, The number of target text blocks.

[0153] Optionally, the posterior probability of each text embedding vector belonging to each cluster m can be calculated through the above steps. and according to The size of the cluster is used to assign a soft clustering label to each target text block, representing the probability that it belongs to each cluster, thus generating multiple target clusters.

[0154] Optionally, soft clustering can be performed based on probability distribution until the cluster centers reach the convergence condition, resulting in multiple target clusters.

[0155] For example, the difference between old and new parameters assigned to different target text blocks in each cluster can be calculated based on the following formula:

[0156] ;

[0157] ;

[0158] ;

[0159] If the changes in all new and old parameters are less than the sixth preset threshold, convergence is considered achieved. This sixth preset threshold can be flexibly set based on actual business needs; for example, for text data and GMM clustering, the sixth preset threshold can be set to 10. -3 Up to 10 -5 This is to ensure that the algorithm can converge stably in most scenarios while avoiding overfitting.

[0160] After obtaining multiple target clusters from the clustered target text blocks, a hierarchical index tree can be constructed based on these target clusters. Specifically, multiple target clusters corresponding to the original text can be integrated into a hierarchical index tree, with the original text as the root node, storing the topic summary of the original text. Each target cluster serves as a child node of the root node, i.e., an intermediate layer node, storing the semantic summary of the multiple target clusters of the original text. The target text blocks included in the target clusters are used as leaf nodes to construct the hierarchical index tree, which can support progressive queries from coarse-grained to fine-grained.

[0161] The mapping table can also store the text positions and text distances of target text blocks in multiple target clusters. Text position refers to the location of a target text block within the original text, such as a paragraph or line. Relative distance refers to the relative proximity between different target text blocks, which can be quantified to reflect their relative spatial relationship within the original text.

[0162] Based on the above scheme, optionally, determining the key block set and mapping table corresponding to the hierarchical index tree includes: for each target cluster in the hierarchical index tree, determining the index words corresponding to the text embedding vector of each target text block within the target cluster; constructing an index table for the target cluster based on the text embedding vector and its corresponding index words; determining the semantic summary of the target cluster; and determining the key text blocks and related text blocks among multiple target text blocks in the target cluster; and determining the text position and storage position of the key text blocks, the text position and storage position of the related text blocks, and the... The process involves: determining the positional weight and text distance of the associated text block relative to the key text block; constructing a key block set based on the key text blocks of multiple target clusters in the hierarchical index tree; filling the index table with the semantic summary of the target cluster, the text position of the key text block, the text position of the associated text block, the positional weight, and the text distance according to the storage location of the key text block and the storage location of the associated text block; filling the index table with the topic summary of the original text corresponding to the hierarchical index tree to obtain a mapping table; and storing the key block set and the mapping table in association with the hierarchical index tree.

[0163] Here, index terms refer to words that summarize the features of the text embedding vector corresponding to the text embedding vector of each target text block within the target cluster. They serve as markers for the text embedding vectors, used to construct an index table for the target cluster, facilitating rapid lookup. The index table can be a table constructed based on the text embedding vectors within the target cluster and their corresponding index terms.

[0164] Specifically, for each target cluster in the hierarchical index tree, the index terms corresponding to the text embedding vectors of each target text block within the target cluster are determined, and an index table for the target cluster is constructed based on the text embedding vectors and their corresponding index terms. Constructing the index table enables a fast mapping between text embedding vectors and index terms.

[0165] Optionally, a semantic summary of multiple target clusters contained in the original text can be determined, and key text blocks and related text blocks in multiple target text blocks within the target clusters can be identified. The text position and storage location of the key text blocks, the text position and storage location of the related text blocks, and the positional weight and text distance of the related text blocks relative to the key text blocks can be determined respectively. Here, related text blocks refer to other text blocks in the target cluster besides key text blocks. Related text blocks and key text blocks within the same target cluster have semantic relationships and together constitute the target cluster. Storage location refers to the specific location information of the text block stored in the mapping table. Positional weight can be an indicator used to quantify the importance and degree of association between related text blocks and key text blocks. Positional weight can be determined based on the cosine similarity between the two. Text distance refers to the distance between related text blocks and key text blocks in terms of text content. Text distance is usually related to the degree of semantic association; the closer the semantics, the closer the text distance may be.

[0166] As an optional technical solution of the present invention, the step of determining the key text block and associated text block among the multiple target text blocks in the target cluster may include: for each target text block in each target cluster in the hierarchical index tree, determining the cosine similarity between the target text block and the remaining text embedding vectors in the target cluster, and determining the average similarity of the multiple cosine similarities corresponding to the target text block; determining the target text block with the highest average similarity among the multiple target text blocks in the target cluster as the key text block, and determining the target text blocks in the target cluster other than the key text block as associated text blocks.

[0167] Specifically, the cosine similarity between the text embedding vectors corresponding to different target text blocks in each target cluster of the hierarchical index tree can be calculated based on the following formula. :

[0168] ;

[0169] in, These represent the text embedding vector corresponding to the target text block and the text embedding vector corresponding to the other target text blocks in the target cluster where the target text block is located, respectively. Text embedding vector and The dot product; and Text embedding vectors and The model.

[0170] Specifically, the average similarity between each target text block and the remaining text blocks within the target cluster can be calculated based on the following similarity matrix. :

[0171] ;

[0172] in, This represents the number of target text blocks within the target cluster.

[0173] Furthermore, the target text block with the highest average similarity among the multiple target text blocks within the target cluster can be identified as the key text block, and the target text blocks within the target cluster other than the key text block can be identified as associated text blocks.

[0174] By using the average semantic similarity of the target text block with other text blocks within its cluster, the most representative key text block can be accurately identified, ensuring that it is the core semantic center. This avoids the bias caused by manual setting and makes the organization of related text blocks around the key block more logical and consistent, which is conducive to building a high-quality hierarchical structure.

[0175] Optionally, the text position and storage location of the key text block in the original text can be determined based on the semantic content of the key text block or its position in the hierarchical index tree.

[0176] As an optional technical solution of this invention, optionally, determining the storage location of the associated text block, the position weight of the associated text block relative to the key text block, and the text distance includes: determining the cosine similarity between the target text block and the key text block in the target cluster as the position weight; determining the total weight of the position weights of multiple associated text blocks in the target cluster; determining the weight ratio of the associated text block based on the position weight of the associated text block and the total weight; using the position information of the key text block in the original text as a reference point, allocating a storage location for each associated text block according to the weight ratio of the associated text block, wherein the storage location of the associated text block is used to indicate the storage location of the associated text block relative to the key text block; the higher the position weight of the associated text block, the closer its storage location in the mapping table is to the key text block.

[0177] Specifically, the cosine similarity between the target text block and the key text blocks in the target cluster can be determined as the positional weight. After normalizing the positional weights to ensure they are within a reasonable range, the positional weights of all related text blocks in the target cluster are summed to obtain a total weight, which is then used to calculate the weight ratio of each related text block. The weight ratio is the ratio of the positional weight of each related text block to the total weight of all related text blocks in the target cluster, reflecting the relative importance of the related text block in the target cluster, and thus determining the relative position of the related text block in the original text.

[0178] Furthermore, the location information of the key text block in the original text can be used as a reference point, and a storage location can be allocated to each of the associated text blocks according to the weight ratio of the associated text blocks. The storage location of the associated text block is used to indicate the storage location of the associated text block relative to the key text block. The higher the position weight of the associated text block, the closer its storage location in the mapping table is to the key text block.

[0179] During the allocation of storage locations for associated text blocks, it is essential to ensure that the allocated storage locations do not overlap. If overlap occurs, fine-tuning can be performed based on the position weights of the associated text blocks, prioritizing the text blocks with higher position weights that retain their allocated storage locations. After allocation, the text position and storage location of each associated text block, as well as the text distance of the associated text block relative to the key text block, can be recorded.

[0180] By calculating positional weights based on semantic similarity and allocating storage locations accordingly, consistency between semantic similarity and physical storage distance is achieved, facilitating efficient retrieval and local access optimization. By adjusting the storage layout through weight ratios, highly relevant text is placed closer to key text blocks in the mapping table, improving caching efficiency and reading coherence, thus facilitating semantic queries.

[0181] Furthermore, a key block set can be constructed based on the key text blocks of multiple target clusters in the hierarchical index tree. The semantic summary of the target cluster, the text position of the key text block, the text position of the associated text block, the position weight, and the text distance are filled into the index table according to the storage location of the key text block and the storage location of the associated text block. The topic summary of the original text corresponding to the hierarchical index tree is also filled into the index table to obtain a mapping table. The key block set and the mapping table are then stored in association with the hierarchical index tree.

[0182] By constructing a mapping table containing index terms, semantic summaries, key text blocks, and related text blocks and their positional relationships, a high degree of integration of semantic and structural information is achieved. This not only supports rapid location of core content but also records rich contextual information, greatly enhancing the semantic reasoning and contextual reconstruction capabilities during subsequent queries.

[0183] S240. In response to the query request, obtain the original query statement, determine the query term set of the original query statement under multiple query dimensions, and determine the query vector set corresponding to the query term set according to the word vector embedding model, wherein the word vector embedding model is obtained by training a machine learning model based on the sample word set and its corresponding expected vector set.

[0184] S250. Based on the query vector set, the hierarchical index tree of multiple queryable original texts, the key block set corresponding to the hierarchical index tree, and the mapping table, initial candidate text blocks in multiple original texts are determined. The hierarchical index tree includes a root node, intermediate layer nodes, and leaf nodes. The root node represents the topic summary of the original text. The intermediate layer nodes store the semantic summaries of multiple target clusters of the original text. The leaf nodes store the content of the target text blocks in the target clusters. The mapping table stores at least the topic summary of the root node and the semantic summary of the intermediate layer nodes.

[0185] S260. Based on the query term set, select multiple target candidate text blocks from multiple initial candidate text blocks, generate a text context related to the query term set based on the query term set and the target candidate text blocks, determine the target query result of the original query statement based on the text context, and display the target query result.

[0186] The technical solution of this invention first involves acquiring original text, segmenting the original text to obtain multiple initial words, identifying multiple effective words from these initial words, and dividing the original text into multiple initial text blocks based on the positions of the effective words in the original text; each initial text block contains at least one effective word; through word segmentation and effective word identification, key semantic units in the text can be accurately extracted, improving the accuracy and efficiency of subsequent processing; text segmentation based on the positions of effective words ensures that each text block has actual semantic value and avoids interference from meaningless fragments; next, the initial text blocks are initially split according to text delimiters to obtain multiple clauses, target text blocks are determined based on the multiple clauses, and the text embedding vectors of the target text blocks are determined; features of the original text are constructed based on the text embedding vectors of the multiple target text blocks. The initial text block is further split using text delimiters, which helps to obtain clauses with clearer grammatical structures, facilitating refined semantic analysis. Combining these clauses to generate target text blocks and calculating their text embedding vectors effectively captures local semantic features, laying the foundation for constructing a high-quality feature vector set. Finally, the text embedding vectors in the feature vector set are recorded using a Gaussian mixture model clustering algorithm to obtain multiple target clusters. A hierarchical index tree is constructed based on these target clusters, and the corresponding key block set and mapping table are determined. Using a Gaussian mixture model to cluster feature vectors can identify potential topics or semantic clusters in the text, improving the flexibility of handling overlapping categories. Constructing a hierarchical index tree, key block set, and mapping table not only enhances the structural organization of information but also significantly improves the efficiency and scalability of subsequent retrieval and querying. Through step-by-step refined processing, an efficient conversion from raw text to a structured semantic index is achieved.

[0187] Example 3

[0188] Figure 3 This is a schematic diagram of a text processing device according to Embodiment 3 of the present invention. This device is used to execute the text processing method provided in any of the above embodiments. This device and the text processing methods of the above embodiments belong to the same inventive concept. Details not described in detail in the embodiments of the text processing device can be found in the embodiments of the above text processing methods. Figure 3 As shown, the device includes: a query vector set determination module 310, an initial candidate text block determination module 320, and a target query result determination module 330.

[0189] The query vector set determination module 310 is used to respond to a query request, obtain the original query statement, determine the query term set of the original query statement under multiple query dimensions, and determine the query vector set corresponding to the query term set according to a word vector embedding model, wherein the word vector embedding model is obtained by training a machine learning model based on a sample word set and its corresponding expected vector set; the initial candidate text block determination module 320 is used to determine initial candidate text blocks in multiple original texts according to the query vector set, a hierarchical index tree of multiple queryable original texts, a key block set corresponding to the hierarchical index tree, and a mapping table, wherein the hierarchical index tree includes a root node, intermediate level nodes, and leaf nodes. The nodes are as follows: the root node represents the topic summary of the original text; the intermediate layer nodes store the semantic summaries of multiple target clusters of the original text; and the leaf nodes store the content of target text blocks in the target clusters. The mapping table stores at least the topic summary of the root node and the semantic summary of the intermediate layer nodes. The target query result determination module 330 is used to filter multiple target candidate text blocks from multiple initial candidate text blocks according to the query term set, generate a text context related to the query term set according to the query term set and the target candidate text blocks, determine the target query result of the original query statement according to the text context, and display the target query result.

[0190] The technical solution of this invention embodiment firstly involves the query vector set determination module 310 responding to a query request by acquiring the original query statement, determining the query word set of the original query statement under multiple query dimensions, and determining the query vector set corresponding to the query word set based on a word vector embedding model. The word vector embedding model is obtained by training a machine learning model based on a sample word set and its corresponding expected vector set. This multi-dimensional analysis of the original query statement fully explores the potential information needs of the query request, improving the comprehensiveness and robustness of semantic understanding. The generation of query vectors based on the word vector embedding model ensures the accuracy and consistency of semantic representation. Next, the initial candidate text block determination module 320 determines initial candidate text blocks from multiple original texts based on the query vector set, a hierarchical index tree of multiple queryable original texts, a key block set corresponding to the hierarchical index tree, and a mapping table. The hierarchical index tree includes a root node, intermediate nodes, and leaf nodes. The root node represents the topic summary of the original text, the intermediate nodes store the semantic summaries of multiple target clusters of the original text, and the leaf nodes store the content of target text blocks within the target clusters. The mapping table contains at least... The system stores the topic summary of the root node and the semantic summary of the intermediate layer nodes. A hierarchical index tree enables layered organization and efficient navigation of text content, significantly improving retrieval efficiency. Combining the semantic summary with positional information in the mapping table, it can accurately trace the contextual origin of initial candidate text blocks while preserving semantic integrity, providing structured support for subsequent precise filtering. Finally, the target query result determination module 330 filters multiple target candidate text blocks from the initial candidate text blocks based on the query term set. It generates textual context related to the query term set based on the query term set and the target candidate text blocks, determines the target query result of the original query statement based on the textual context, and displays the target query result. A secondary filtering is performed on the initial candidate text blocks, and fine-grained matching is combined with the query term set to effectively filter irrelevant or weakly related segments, improving the relevance and accuracy of the query results. Furthermore, the target candidate text blocks are used to dynamically generate contextual information associated with the query term set, improving the accuracy of the query results. This achieves dual precise positioning of massive text content at both the topic and segment levels, significantly improving the accuracy, efficiency, and readability of text queries.

[0191] Based on the above scheme, optionally, the initial candidate text block determination module 320 includes a candidate text determination submodule, an initial candidate cluster determination submodule, a target candidate cluster determination submodule, and an initial candidate text block determination submodule. The system includes the following modules: a candidate text determination submodule, which, for each queryable original text, retrieves the topic summary of the root node of the hierarchical index tree from the mapping table corresponding to the hierarchical index tree of the original text, and determines multiple candidate texts from the multiple original texts based on the query vector set and the multiple topic summaries; an initial candidate cluster determination submodule, which, for each candidate text, retrieves the semantic summary of the intermediate layer node of the hierarchical index tree from the mapping table corresponding to the hierarchical index tree of the candidate text, and determines an initial candidate cluster from the target clusters of the candidate text based on the semantic summary of the candidate text and the query vector set; a target candidate cluster determination submodule, which determines multiple target candidate clusters from the multiple initial candidate clusters of the candidate text based on the intermediate layer nodes of the hierarchical index tree of the multiple candidate texts, the key block set corresponding to the hierarchical index tree, and the mapping table; wherein the key block set is used to store key text blocks of multiple target clusters; and an initial candidate text block determination submodule, which, for each target candidate cluster, determines multiple initial candidate text blocks matching the query vector set from the multiple target text blocks of the target candidate cluster based on the target candidate cluster and its corresponding mapping table.

[0192] Based on the above scheme, optionally, the mapping table also stores text embedding vectors of multiple key text blocks of the original text; further, the target candidate cluster determination submodule includes an average similarity determination unit and a target candidate cluster determination unit. The average similarity determination unit is used to, for each initial candidate cluster, determine the key text blocks of the initial candidate cluster according to the key block set corresponding to the hierarchical index tree, as the text blocks to be matched, obtain the text embedding vectors of the text blocks to be matched from the mapping table corresponding to the hierarchical index tree, as the embedding vectors to be matched, and determine the average similarity between multiple query vectors in the query vector set and the embedding vectors to be matched; the target candidate cluster determination unit is used to determine multiple target candidate clusters that match the query vector set from the multiple initial candidate clusters based on the average similarity corresponding to the multiple initial candidate clusters.

[0193] Based on the above scheme, the device may optionally further include an initial text block segmentation module, a feature vector set construction module, and a hierarchical index tree construction module. The initial text block segmentation module is used to obtain the original text before determining the initial candidate text blocks in the original text based on the query vector set, the hierarchical index tree of the multiple queryable original texts, the key block set corresponding to the hierarchical index tree, and the mapping table. This process involves: acquiring the original text; segmenting the original text into words to obtain multiple initial words; determining multiple valid words from the multiple initial words; and segmenting the original text into multiple initial text blocks based on the positions of the valid words in the original text. Each initial text block contains at least one valid word. The feature vector set construction module is used to initially split the initial text blocks according to text delimiters to obtain multiple clauses; determine target text blocks based on the multiple clauses; determine the text embedding vectors of the target text blocks; and construct a feature vector set of the original text based on the text embedding vectors of the multiple target text blocks. The hierarchical index tree construction module is used to record the text embedding vectors in the feature vector set using a Gaussian mixture model clustering algorithm to obtain multiple target clusters; construct a hierarchical index tree based on the multiple target clusters; and determine the key block set and mapping table corresponding to the hierarchical index tree.

[0194] Based on the above scheme, optionally, the initial text block segmentation module includes an initial word target weight determination submodule and an effective word determination submodule. The initial word target weight determination submodule is used to determine the word frequency, inverse text frequency, and type of each initial word. If the type is a common word, it determines the target weight of the initial word based on the word frequency and the inverse text frequency. If the type is a specialized word, it determines the semantic importance score and a preset enhancement coefficient of the initial word, and determines the target weight of the initial word based on the word frequency, the inverse text frequency, the semantic importance score, and the preset enhancement coefficient. The effective word determination submodule is used to determine the average weight of the target weights of multiple initial words, determine keywords from the multiple initial words based on the target weights and the average weights, determine the semantic vector of each keyword, and determine multiple effective words from the multiple keywords based on the semantic vectors of the keywords.

[0195] Based on the above scheme, optionally, the feature vector set construction module includes a target text block determination submodule and a feature vector set construction submodule. The target text block determination submodule is used to extract the clause semantic vector of each clause based on the target language model, determine the cosine similarity between each pair of clauses based on the clause semantic vectors of multiple clauses, and merge multiple clauses based on the cosine similarity, a preset similarity threshold, and the maximum value of a preset text processing unit to obtain at least one target text block. The feature vector set construction submodule is used to determine the word embedding vector, segment embedding vector, and position embedding vector of the target text block based on a bidirectional encoding model, determine the model input vector based on the word embedding vector, segment embedding vector, and position embedding vector, input the model input vector into a multi-layer encoder to obtain the text embedding vector of the target text block, and construct the feature vector set of the original text based on the text embedding vectors of multiple target text blocks.

[0196] Based on the above scheme, optionally, the mapping table also stores the text positions and text distances of target text blocks in multiple target clusters; the hierarchical index tree construction module includes a target cluster index table construction submodule, a position determination submodule, and a mapping table construction submodule. Specifically, the target cluster index table construction submodule is used to determine the index terms corresponding to the text embedding vectors of each target text block within each target cluster in the hierarchical index tree, and construct the index table of the target cluster based on the text embedding vectors and their corresponding index terms; the position determination submodule is used to determine the semantic summary of the target cluster, and determine the key text blocks and related text blocks among multiple target text blocks in the target cluster, respectively determining the text position and storage position of the key text blocks, the text position and storage position of the related text blocks, and the position of the related text blocks relative to the key text blocks. The text block position weight and text distance; the mapping table construction submodule is used to construct a key block set based on the key text blocks of multiple target clusters in the hierarchical index tree, and to fill the semantic summary of the target cluster, the text position of the key text block, the text position of the associated text block, the position weight, and the text distance into the index table according to the storage location of the key text block and the storage location of the associated text block, and to fill the topic summary of the original text corresponding to the hierarchical index tree into the index table to obtain the mapping table, and to associate and store the key block set and the mapping table with the hierarchical index tree.

[0197] Based on the above scheme, optionally, the location determination submodule includes a similarity determination unit and a text block type determination unit. Specifically, the similarity determination unit, for each target text block in each target cluster in the hierarchical index tree, determines the cosine similarity between the target text block and the remaining text embedding vectors in the target cluster, and determines the average similarity of the multiple cosine similarities corresponding to the target text block; the text block type determination unit is used to determine the target text block with the highest average similarity among the multiple target text blocks in the target cluster as the key text block, and to determine the target text blocks in the target cluster other than the key text block as associated text blocks.

[0198] Based on the above scheme, optionally, the position determination submodule further includes a position weight determination unit and a storage position determination unit. The position weight determination unit is used to determine the cosine similarity between the target text block and the key text block in the target cluster as a position weight, determine the sum of the position weights of multiple associated text blocks in the target cluster, and determine the weight ratio of the associated text block based on the position weight of the associated text block and the sum of the weights. The storage position determination unit is used to allocate a storage position for each associated text block based on the position information of the key text block in the original text, according to the weight ratio of the associated text blocks, wherein the storage position of the associated text block indicates the storage position of the associated text block relative to the key text block; the higher the position weight of the associated text block, the closer its storage position in the mapping table is to the key text block.

[0199] The text processing apparatus provided in the embodiments of the present invention can execute the text processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0200] Example 4

[0201] Figure 4 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0202] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0203] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0204] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as text processing methods.

[0205] In some embodiments, the text processing method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the text processing method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the text processing method by any other suitable means (e.g., by means of firmware).

[0206] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0207] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0208] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0209] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0210] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0211] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0212] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication unit 19, or installed from storage unit 18, or installed from ROM 12. When the computer program is executed by processor 11, it performs the functions defined in the methods of the embodiments of the present invention.

[0213] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0214] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A text processing method, characterized in that, include: In response to a query request, the original query statement is obtained, the set of query terms for the original query statement under multiple query dimensions is determined, and the set of query vectors corresponding to the set of query terms is determined according to a word vector embedding model, wherein the word vector embedding model is obtained by training a machine learning model based on a set of sample words and their corresponding expected vector sets; Initial candidate text blocks from multiple original texts are determined based on the query vector set, a hierarchical index tree of multiple queryable original texts, a key block set corresponding to the hierarchical index tree, and a mapping table. The hierarchical index tree includes a root node, intermediate layer nodes, and leaf nodes. The root node represents a topic summary of the original text, the intermediate layer nodes store semantic summaries of multiple target clusters of the original text, and the leaf nodes store the content of target text blocks within the target clusters. The mapping table stores at least the topic summary of the root node and the semantic summary of the intermediate layer nodes. Multiple target candidate text blocks are selected from multiple initial candidate text blocks based on the query term set. A text context related to the query term set is generated based on the query term set and the target candidate text blocks. The target query result of the original query statement is determined based on the text context and the target query result is displayed.

2. The text processing method according to claim 1, characterized in that, The step of determining initial candidate text blocks from multiple original texts based on the query vector set, a hierarchical index tree of multiple queryable original texts, a key block set corresponding to the hierarchical index tree, and a mapping table includes: For each queryable raw text, obtain the topic summary of the root node of the hierarchical index tree from the mapping table corresponding to the hierarchical index tree of the raw text, and determine multiple candidate texts from multiple raw texts based on the query vector set and multiple topic summaries; For each candidate text, the semantic summary of the intermediate layer node of the hierarchical index tree is obtained from the mapping table corresponding to the hierarchical index tree of the candidate text, and an initial candidate cluster is determined from the target cluster of the candidate text based on the semantic summary of the candidate text and the query vector set. Based on the intermediate layer nodes of the hierarchical index tree of the candidate texts, the key block set corresponding to the hierarchical index tree, and the mapping table, multiple target candidate clusters are determined from multiple initial candidate clusters of the candidate texts; wherein, the key block set is used to store the key text blocks of the multiple target clusters; For each target candidate cluster, multiple initial candidate text blocks that match the query vector set are determined from multiple target text blocks of the target candidate cluster according to the target candidate cluster and its corresponding mapping table.

3. The text processing method according to claim 2, characterized in that, The mapping table also stores text embedding vectors of multiple key text blocks of the original text; the step of determining multiple target candidate clusters from multiple initial candidate clusters of the candidate texts based on the intermediate layer nodes of the hierarchical index tree of the multiple candidate texts, the key block set corresponding to the hierarchical index tree, and the mapping table includes: For each initial candidate cluster, the key text block of the initial candidate cluster is determined according to the key block set corresponding to the hierarchical index tree, and is used as the text block to be matched. The text embedding vector of the text block to be matched is obtained from the mapping table corresponding to the hierarchical index tree, and is used as the embedding vector to be matched. The average similarity between multiple query vectors in the query vector set and the embedding vector to be matched is determined. Based on the average similarity corresponding to the initial candidate clusters, multiple target candidate clusters that match the query vector set are determined from the multiple initial candidate clusters.

4. The text processing method according to claim 1, characterized in that, Before determining the initial candidate text blocks in the multiple original texts based on the query vector set, the hierarchical index tree of multiple queryable original texts, the key block set corresponding to the hierarchical index tree, and the mapping table, the method further includes: The original text is obtained, and the original text is segmented to obtain multiple initial words. Multiple valid words are determined from the multiple initial words, and the original text is divided into multiple initial text blocks according to the position of the valid words in the original text; wherein each initial text block contains at least one of the valid words. The initial text block is initially split according to the text delimiter to obtain multiple clauses. Target text blocks are determined based on the multiple clauses, and the text embedding vectors of the target text blocks are determined. A feature vector set of the original text is constructed based on the text embedding vectors of the multiple target text blocks. The Gaussian mixture model clustering algorithm is used to record the text embedding vectors in the feature vector set to obtain multiple target clusters. A hierarchical index tree is constructed based on the multiple target clusters, and the key block set and mapping table corresponding to the hierarchical index tree are determined.

5. The text processing method according to claim 4, characterized in that, The determination of multiple valid words from the multiple initial words includes: For each initial word, the word frequency, inverse text frequency, and type of the initial word are determined. If the type is a common word, the target weight of the initial word is determined based on the word frequency and the inverse text frequency. If the type is a specialized word, the semantic importance score and the preset enhancement coefficient of the initial word are determined. The target weight of the initial word is determined based on the word frequency, the inverse text frequency, the semantic importance score, and the preset enhancement coefficient. The average weight of the target weights of the initial words is determined. Keywords are determined from the initial words based on the target weights and the average weights of the initial words. The semantic vector of each keyword is determined. Multiple effective words are determined from the keywords based on the semantic vectors of the keywords.

6. The text processing method according to claim 4, characterized in that, The step of determining the target text block based on the multiple clauses and determining the text embedding vector of the target text block includes: Extract the semantic vector of each clause based on the target language model, determine the cosine similarity between each pair of clauses based on the semantic vectors of the clauses, and merge the clauses based on the cosine similarity, a preset similarity threshold and the maximum value of a preset text processing unit to obtain at least one target text block; The word embedding vector, segment embedding vector, and position embedding vector of the target text block are determined based on the bidirectional encoding model. The model input vector is determined based on the word embedding vector, the segment embedding vector, and the position embedding vector. The model input vector is input into a multi-layer encoder to obtain the text embedding vector of the target text block. The feature vector set of the original text is constructed based on the text embedding vectors of multiple target text blocks.

7. The text processing method according to claim 4, characterized in that, The mapping table also stores the text positions and text distances of target text blocks in multiple target clusters; determining the key block set and mapping table corresponding to the hierarchical index tree includes: For each target cluster in the hierarchical index tree, the index terms corresponding to the text embedding vector of each target text block in the target cluster are determined, and an index table of the target cluster is constructed based on the text embedding vector in the target cluster and its corresponding index terms; Determine the semantic summary of the target cluster, and identify key text blocks and associated text blocks among multiple target text blocks in the target cluster. Determine the text position and storage position of the key text block, the text position and storage position of the associated text block, and the position weight and text distance of the associated text block relative to the key text block, respectively. A key block set is constructed based on the key text blocks of multiple target clusters in the hierarchical index tree. The semantic summary of the target cluster, the text position of the key text block, the text position of the associated text block, the position weight, and the text distance are filled into the index table according to the storage location of the key text block and the storage location of the associated text block. The topic summary of the original text corresponding to the hierarchical index tree is also filled into the index table to obtain a mapping table. The key block set and the mapping table are then stored in association with the hierarchical index tree.

8. The text processing method according to claim 7, characterized in that, The step of determining the key text blocks and associated text blocks among the multiple target text blocks in the target cluster includes: For each target text block in each target cluster in the hierarchical index tree, determine the cosine similarity between the target text block and the other text embedding vectors in the target cluster, and determine the average similarity of the multiple cosine similarities corresponding to the target text block; The target text block with the highest average similarity among the multiple target text blocks in the target cluster is identified as the key text block, and the target text blocks in the target cluster other than the key text block are identified as related text blocks.

9. The text processing method according to claim 7, characterized in that, Determining the storage location of the associated text block, the positional weight of the associated text block relative to the key text block, and the text distance includes: The cosine similarity between the target text block and the key text block in the target cluster is determined as the position weight. The total weight of the position weights of multiple related text blocks in the target cluster is determined. The weight ratio of the related text block is determined based on the position weight of the related text block and the total weight. Using the location information of the key text block in the original text as a reference point, a storage location is assigned to each of the associated text blocks according to the weight ratio of the associated text blocks. The storage location of the associated text block is used to indicate the storage location of the associated text block relative to the key text block. The higher the position weight of the associated text block, the closer its storage location in the mapping table is to the key text block.

10. A text processing device, characterized in that, include: The query vector set determination module is used to respond to a query request, obtain the original query statement, determine the query word set of the original query statement under multiple query dimensions, and determine the query vector set corresponding to the query word set according to the word vector embedding model, wherein the word vector embedding model is obtained by training a machine learning model based on the sample word set and its corresponding expected vector set; The initial candidate text block determination module is used to determine initial candidate text blocks from multiple queryable original texts based on the query vector set, a hierarchical index tree of multiple queryable original texts, a key block set corresponding to the hierarchical index tree, and a mapping table. The hierarchical index tree includes a root node, intermediate layer nodes, and leaf nodes. The root node represents a topic summary of the original text, the intermediate layer nodes store semantic summaries of multiple target clusters of the original text, and the leaf nodes store the content of target text blocks within the target clusters. The mapping table stores at least the topic summary of the root node and the semantic summary of the intermediate layer nodes. The target query result determination module is used to filter multiple target candidate text blocks from multiple initial candidate text blocks according to the query term set, generate a text context related to the query term set according to the query term set and the target candidate text blocks, determine the target query result of the original query statement according to the text context, and display the target query result.

11. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to cause the at least one processor to perform the text processing method as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the text processing method as described in any one of claims 1-9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the text processing method as described in any one of claims 1-9.

Images