Data processing method and apparatus, electronic device, and medium

Abstract

The application provides a data processing method and device, electronic equipment and medium. The method can receive a query statement input by a user; according to the query statement, a plurality of candidate text blocks are retrieved from a saved text block set; wherein each text block in the saved text block set is associated with at least one semantic relationship chain and corresponding semantic abstract information. The semantic relationship chain is used to represent the semantic association relationship between the plurality of text blocks, and the semantic abstract information is used to represent the overall semantic content of the corresponding semantic relationship chain; according to the semantic abstract information associated with the candidate text blocks and the query statement, at least one target text block is selected from the plurality of candidate text blocks; and according to the similarity between the target text block and the query statement, it is determined whether to supplement the target text block based on the semantic relationship chain associated with the target text block to obtain a retrieval result corresponding to the query statement.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus, electronic device and medium. Background Technology

[0002] In text processing fields such as information retrieval, intelligent question answering, and patent retrieval, text block-based retrieval is a common technique. Traditional methods typically divide a document into several text blocks and, based on the user's input query, retrieve candidate text blocks from the set of text blocks that are semantically similar to the query as the retrieval results.

[0003] In practical applications, a single text block often cannot fully express complete semantic information, especially when the document structure is complex and semantics are distributed across blocks. Directly retrieving based on isolated text blocks can easily lead to information loss or semantic fragmentation. To alleviate this problem, related technologies recall retrieved text blocks along with their adjacent text blocks, thus mechanically splicing context information and mitigating the problem of incomplete retrieval to some extent. However, this method has significant drawbacks: firstly, the splicing of context lacks semantic relevance judgment and relies solely on positional proximity, easily introducing text content unrelated to the query; secondly, the fixed context range cannot flexibly adapt to different retrieval needs, easily leading to information overload and decreased retrieval accuracy. Summary of the Invention

[0004] This application provides a data processing method, including: receiving a query statement input by a user; retrieving a plurality of candidate text blocks from a saved set of text blocks according to the query statement; wherein each text block in the saved set of text blocks is associated with at least one semantic relationship chain and corresponding semantic summary information, the semantic relationship chain is used to characterize the semantic relationship between multiple text blocks, and the semantic summary information is used to characterize the overall semantic content of the corresponding semantic relationship chain; selecting at least one target text block from the plurality of candidate text blocks according to the semantic summary information associated with the candidate text block and the query statement; and determining whether to semantically supplement the target text block based on the semantic relationship chain associated with the target text block according to the similarity between the target text block and the query statement, so as to obtain the retrieval results corresponding to the query statement.

[0005] In this embodiment, obtaining the saved text block set includes: dividing the document set to be processed into multiple text blocks; calculating the semantic similarity between adjacent text blocks; iteratively expanding in adjacent directions based on the semantic similarity and a first threshold to construct at least one semantic relationship chain, each semantic relationship chain containing at least two semantically related text blocks; aggregating the content of the text blocks in each semantic relationship chain to generate corresponding semantic summary information; and associating and saving the semantic relationship chain and its corresponding semantic summary information with the corresponding text blocks.

[0006] In this embodiment of the application, iterative diffusion in adjacent directions to construct at least one semantic relationship chain includes: taking the current text block as the anchor text block, recursively determining whether the semantic similarity with other text blocks is not lower than a first threshold in the first direction and the second direction of the anchor text block, respectively; the first direction and the second direction are opposite directions; if it is not lower than the first threshold, the text block is included in the current semantic relationship chain and continues to diffuse in the same direction until it is lower than the first threshold or reaches the boundary of the document to be processed and stops diffusion.

[0007] In this embodiment of the application, the process of aggregating information from the text block content in each semantic relation chain to generate corresponding semantic summary information includes: inputting the text block content in the semantic relation chain as context into the large language model; and guiding the large language model to generate semantic summary information through prompt word engineering.

[0008] In this embodiment of the application, selecting at least one target text block from a plurality of candidate text blocks based on the semantic summary information associated with the candidate text block and the query statement includes: inputting the semantic summary information and query statement corresponding to the candidate text block into a large language model; guiding the large language model to perform a correlation determination between each candidate text block and the query statement one by one through prompt word engineering, and outputting the correlation result between each candidate text block and the query statement; and selecting at least one target text block from a plurality of candidate text blocks based on the correlation result between each candidate text block and the query statement.

[0009] In this embodiment of the application, determining whether to semantically supplement the target text block based on the semantic relationship chain associated with the target text block to obtain the retrieval result corresponding to the query statement, based on the similarity between the target text block and the query statement, includes: if the similarity between the target text block and the query statement is lower than a second threshold, then triggering the semantic supplementation process of the retrieval result to add other text blocks in the semantic relationship chain associated with the target text block to the retrieval result, thereby obtaining the retrieval result corresponding to the query statement.

[0010] In this embodiment of the application, the method further includes: constructing a similarity distribution based on the highest similarity score between the text block retrieved each time in the historical query and the query statement; and determining a second threshold based on a specified quantile of the similarity distribution.

[0011] This application provides a data processing apparatus, including: a receiving module for receiving a query statement input by a user; a retrieval module for retrieving a plurality of candidate text blocks from a saved set of text blocks according to the query statement; wherein each text block in the saved set of text blocks is associated with at least one semantic relationship chain and corresponding semantic summary information, the semantic relationship chain is used to characterize the semantic association between multiple text blocks, and the semantic summary information is used to characterize the overall semantic content of the corresponding semantic relationship chain; a filtering module for filtering at least one target text block from the plurality of candidate text blocks according to the semantic summary information associated with the candidate text block and the query statement; and a semantic supplementation module for determining whether to perform semantic supplementation on the target text block based on the semantic relationship chain associated with the target text block according to the similarity between the target text block and the query statement, so as to obtain the retrieval results corresponding to the query statement.

[0012] This application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a data processing method.

[0013] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described data processing method.

[0014] The beneficial effects of the data processing method provided in this application are as follows: By pre-associating each text block with at least one semantic relationship chain and corresponding semantic summary information, the retrieval process can not only rely on isolated text blocks, but also use semantic summary information to determine the overall relevance of the semantic chain to which the text block belongs to the query statement. Compared with the traditional method of mechanically splicing context based solely on positional proximity, this invention can more accurately filter out target text blocks that are truly relevant to the query statement, avoid introducing irrelevant content, and significantly improve the semantic integrity of the retrieval results. In addition, the traditional method uses a fixed range of context splicing, which is difficult to adapt to diverse retrieval scenarios. After filtering out the target text block, this invention further selects whether to perform semantic supplementation based on its associated semantic relationship chain according to the similarity between the text block and the query statement. This dynamic decision-making mechanism makes context expansion no longer blind, but flexibly triggered according to the actual needs of the query statement, thereby effectively reducing the introduction of redundant information, avoiding information overload, improving retrieval accuracy while controlling the scale of results. Attached Figure Description

[0015] Figure 1 A flowchart illustrating a data processing method according to an embodiment of this application is shown schematically. Figure 2 This illustration shows a flowchart of obtaining a set of saved text blocks according to an embodiment of this application; Figure 3 This illustration schematically shows a flowchart of an embodiment of the present application that illustrates the iterative diffusion in adjacent directions to construct at least one semantic relationship chain; Figure 4 The illustration shows a schematic diagram of the construction of the diffusion semantic relationship chain and information aggregation in an embodiment of this application; Figure 5 The flowchart illustrating step S130 of an embodiment of this application is shown in the schematic diagram. Figure 6 This illustration shows a schematic diagram of semantic supplementation based on a diffusion semantic relationship chain, according to an embodiment of this application. Figure 7 A block diagram of a data processing apparatus according to an embodiment of this application is shown schematically; Figure 8 A schematic block diagram of an electronic device according to an embodiment of this application is shown. Detailed Implementation

[0016] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings to enable those skilled in the art to implement them more readily. Furthermore, for clarity, portions unrelated to the description of exemplary embodiments have been omitted from the drawings. In this disclosure, it should be understood that terms such as "comprising" or "having" are intended to indicate the presence of features, numbers, steps, behaviors, components, portions, or combinations thereof disclosed in this specification, and are not intended to exclude the possibility of the presence or addition of one or more other features, numbers, steps, behaviors, components, portions, or combinations thereof. It should also be noted that, without conflict, embodiments and features in the embodiments of this disclosure can be combined with each other. The present disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0017] In text processing fields such as information retrieval, intelligent question answering, and patent retrieval, documents (such as patent documents, scientific papers, technical manuals, and regulatory documents) are typically divided into several text blocks according to certain rules. A text block can be a paragraph, a sentence, a fixed-length sequence of characters, or a text unit divided based on semantic boundaries (such as headings, line breaks, and punctuation marks). Each text block is used as an independent retrieval unit, facilitating subsequent indexing and similarity calculation.

[0018] In traditional retrieval methods, the system receives a query from the user and calculates the similarity between the query and each text block using vectorized representations (such as TF-IDF, BM25, or deep learning-based semantic embedding models). This allows the system to retrieve the most similar text blocks from the set of text blocks as candidate text blocks. In real-world documents, a complete piece of semantic information (such as the implementation steps of a technical solution or the constituent elements of a legal clause) often spans multiple consecutive or discontinuous text blocks. When text blocks are retrieved in isolation, a single text block may contain only partial semantics, leading to incomplete or difficult-to-understand retrieval results; this phenomenon is called semantic fragmentation.

[0019] To alleviate the semantic fragmentation problem, related technologies employ a context-based concatenation method: when a relevant text block is retrieved, the M preceding and N following text blocks (based on positional order) in the document are returned as search results. While this method supplements the context to some extent, it suffers from the following drawbacks: Lack of semantic relevance judgment: text blocks introduced solely based on positional proximity may be semantically irrelevant to the query statement, introducing noise instead. Fixed extension range: M and N are fixed hyperparameters, unable to adapt to the varying contextual requirements of different queries, easily leading to information overload or insufficient supplementation. To address the aforementioned problems of related technologies, this application proposes a data processing method, apparatus, electronic device, and medium, specifically illustrated in the following figures.

[0020] Figure 1 A flowchart illustrating a data processing method according to an embodiment of this application is shown.

[0021] like Figure 1 As shown, the above data processing method includes steps S110 to S140.

[0022] In step S110, the query statement input by the user is received.

[0023] For example, a user types a natural language query in a text box in a graphical user interface or command-line interface, such as "What are some image segmentation methods based on deep learning?". In this embodiment, the system can also integrate a speech recognition module to receive the user's voice input and convert it into a text query.

[0024] In step S120, according to the query statement, several candidate text blocks are retrieved from the saved text block set; wherein, each text block in the saved text block set is associated with at least one semantic relationship chain and corresponding semantic summary information. The semantic relationship chain is used to characterize the semantic relationship between multiple text blocks, and the semantic summary information is used to characterize the overall semantic content of the corresponding semantic relationship chain.

[0025] For example, after receiving the user's query, a preliminary retrieval stage is initiated. For instance, based on the query, several candidate text blocks semantically related to the query are retrieved from a pre-built and stored set of text blocks. In this embodiment, each recalled candidate text block carries its pre-associated semantic relationship chain and corresponding semantic summary information, thus providing a basis for subsequent filtering and semantic supplementation steps.

[0026] Specifically, semantic relationship chains can be used to represent the semantic associations between multiple text blocks. For example, a semantic relationship chain corresponds to a complete semantic unit, which can span multiple text blocks. These text blocks can be continuous or discontinuous; they can be located in adjacent positions within the same document or in different chapters of the same document. In cross-document retrieval scenarios, they can even be located in different documents.

[0027] Semantic summary information can be used to represent the overall semantic content of the corresponding semantic relationship chain. This semantic summary information can take many forms, including but not limited to: short text summaries, keyword sets, topic tags, or vectorized semantic representations.

[0028] The saved collection of text blocks can be stored using a relational database or a vector database. Each text block record can contain the following fields: a unique identifier for the text block, an identifier for the document it belongs to, the text block content, its position in the original document, and a list field. This list field stores the identifiers of all semantic relationship chains associated with the text block and the semantic summary information corresponding to each semantic relationship chain. The semantic summary information can be stored in text format, vector format, or both.

[0029] In this embodiment, after receiving the preprocessed query statement, the semantic similarity between the query statement and each text block in the text block set is calculated. The similarity calculation method can employ any method from related technologies, and this application is not limited to any particular method. For example, the cosine similarity or Euclidean distance between the query statement vector and each text block vector can be calculated to obtain the corresponding semantic similarity. Based on the calculated semantic similarity, the top K text blocks with the highest semantic similarity are selected as candidate text blocks from the text block set, where K is a preset positive integer. This value can be specified by the user or adaptively determined by the system based on the retrieval scenario. In another implementation, a similarity threshold can also be set, and all text blocks with semantic similarity exceeding this threshold can be selected as candidate text blocks.

[0030] For each candidate text block, the following information can be output: the content of the text block, the position of the text block in the original document, and a list of semantic relationship chains associated with the text block. For each semantic relationship chain in the list, the system outputs a unique identifier for the semantic relationship chain and the corresponding semantic summary information. In a preferred embodiment, a list of identifiers for other text blocks contained in the semantic relationship chain can also be output to quickly locate relevant text blocks in subsequent semantic supplementation steps.

[0031] In step S130, at least one target text block is selected from several candidate text blocks based on the semantic summary information associated with the candidate text block and the query statement.

[0032] For example, after retrieving several candidate text blocks from the text block set, the process enters the candidate text block filtering stage. For instance, based on the semantic summary information associated with each candidate text block, combined with the user's input query, at least one target text block is filtered from the candidate text blocks. This target text block serves as the basis for subsequent judgments regarding whether semantic supplementation is needed and for ultimately generating the search results. Unlike related technologies that directly use all candidate text blocks as search results or mechanically introduce adjacent text blocks, this application introduces semantic summary information as the filtering criterion. Since each candidate text block may be associated with multiple semantic relationship chains, and each semantic relationship chain corresponds to a semantic summary information, this semantic summary information characterizes the overall semantic content of the multiple text blocks covered by that semantic relationship chain. By comparing the query with these semantic summary information, it is possible to determine whether the semantic context of the candidate text block matches the user's query intent, thereby filtering out truly semantically relevant target text blocks and excluding candidate text blocks that, although similar to the query, have an unrelated overall semantic context.

[0033] Specifically, the relevance of the user-input query to each semantic summary associated with each candidate text block is calculated. This relevance calculation can employ various text similarity measurement methods, depending on the storage format of the semantic summary information. For example, based on text-based similarity calculation, the query and the semantic summary text are compared semantically. Based on vector-based similarity calculation, the system does not need to encode the semantic summary text in real time; it can directly read pre-stored semantic summary vectors. This facilitates similarity calculation between the query vector and each semantic summary vector, such as calculating cosine similarity. This method has the advantage of extremely high computational efficiency and is suitable for large-scale retrieval scenarios. Then, based on the calculated relevance scores between the query and each semantic summary, at least one target text block is selected from the candidate text blocks, for example, by setting a preset relevance threshold. For each candidate text block, it is determined whether there exists at least one semantic summary in its associated semantic summary information whose relevance score to the query is greater than or equal to the preset threshold. If it exists, the candidate text block is identified as a target text block; otherwise, it is excluded from the search results.

[0034] In step S140, based on the similarity between the target text block and the query statement, it is determined whether to perform semantic supplementation on the target text block based on the semantic relationship chain associated with the target text block in order to obtain the retrieval results corresponding to the query statement.

[0035] For example, after completing the step of filtering at least one target text block from candidate text blocks, the semantic supplementation decision and retrieval result generation stage is entered. For example, based on the similarity between each target text block and the query statement, it is determined whether to semantically supplement the target text block according to the semantic relationship chain associated with the target text block, and finally obtain the retrieval result corresponding to the query statement.

[0036] Unlike related technologies that mechanically recall all adjacent text blocks of a target text block, this application introduces a dynamic decision-making mechanism. This mechanism uses the similarity between the target text block and the query statement as the decision basis, and the similarity value reflects the degree to which the target text block meets the user's query requirements. For target text blocks with high similarity, they can already respond well to the query statement without the need for additional context. For target text blocks with intermediate similarity, the information they carry may be incomplete, requiring semantic supplementation by recalling other related text blocks in the semantic relationship chain. For target text blocks with low similarity (but which have been retained through semantic summary information filtering), they may only serve as the entry point of the semantic link, requiring reliance on other text blocks in the semantic relationship chain to fully express the semantic information required by the query.

[0037] Specifically, when it is determined that semantic supplementation is needed for a target text block, other text blocks contained in the semantic relationship chain associated with the target text block are retrieved from the text block set, and these text blocks are merged with the current target text block to form the semantically supplemented retrieval result. It should be noted that each target text block may be associated with one or more semantic relationship chains. When semantic supplementation is needed, it is necessary to determine which semantic relationship chain to use for supplementation. In this embodiment, the semantic relationship chain associated with the target text block whose semantic summary information has the highest relevance to the query statement is selected as the supplementation basis. The relevance score of this semantic summary information can be obtained and cached in the aforementioned filtering step. After selecting the semantic relationship chain, all text blocks belonging to that semantic relationship chain are retrieved from the text block set according to the text block identifier list recorded in the semantic relationship chain. The text block identifier list includes the current target text block itself and other associated text blocks. During the retrieval process, the original order of these text blocks can be preserved, for example, arranged according to their position in the original document or according to a preset logical order in the semantic relationship chain.

[0038] In this embodiment, all text blocks in the entire semantic relationship chain are recalled as the result of semantic supplementation. This method is suitable for scenarios with a small semantic relationship chain or high requirements for semantic integrity. To avoid information overload caused by introducing too many text blocks, the scope of semantic supplementation can be limited. Specific limitations include: recalling only a few text blocks in the semantic relationship chain that are located before or after the current target text block; recalling only text blocks in the semantic relationship chain whose similarity to the query statement exceeds a certain threshold; or limiting the total number of text blocks after semantic supplementation to no more than a preset upper limit, such as no more than 10 text blocks.

[0039] After semantic supplementation, a set of text blocks is obtained, including the original target text block and supplementary text blocks recalled from the semantic relationship chain. These text blocks may contain duplicate or highly redundant content. In this embodiment, the supplemented text block set is deduplicated, removing duplicates based on the unique identifier of the text block or the hash value of the text content. After deduplication, the text blocks are sorted. The sorting can be based on the similarity score between each text block and the query statement, or on the logical order of each text block in the semantic relationship chain (e.g., chronological order, causal order, or the original order in the document). In a preferred embodiment, the text blocks are first arranged according to the logical order of the semantic relationship chain to maintain semantic coherence, and then the similarity score of each text block is labeled based on this order for user reference.

[0040] The data processing method of this application can pre-associate at least one semantic relationship chain and corresponding semantic summary information for each text block. During the retrieval process, it can not only rely on isolated text blocks but also use semantic summary information to determine the overall relevance of the semantic chain to which the text block belongs to the query statement. Compared to traditional methods that mechanically splice context based solely on positional proximity, this invention can more accurately filter out target text blocks truly relevant to the query statement, avoiding the introduction of irrelevant content and significantly improving the semantic integrity of the retrieval results. Furthermore, traditional methods use fixed-range context splicing, which is difficult to adapt to diverse retrieval scenarios. After filtering out target text blocks, this invention further selectively decides whether to perform semantic supplementation based on the semantic relationship chain associated with the text block, according to the similarity between the text block and the query statement. This dynamic decision-making mechanism makes context expansion no longer blind but flexibly triggered according to the actual needs of the query statement, thereby effectively reducing the introduction of redundant information, avoiding information overload, improving retrieval accuracy while controlling the scale of results.

[0041] Figure 2 The flowchart illustrating the acquisition of a saved text block set according to an embodiment of this application is shown in the illustration.

[0042] like Figure 2 As shown, the process of obtaining the saved text block set includes steps S210 to S250.

[0043] In step S210, the collection of documents to be processed is divided into multiple text blocks. For example, the collection of documents to be processed includes one or more documents that need to be indexed and retrieved. The source of the documents may be a local file system, a database, web page content obtained by a web crawler, cloud storage service, or any accessible document source. The types of documents include, but are not limited to: patent documents, academic papers, technical reports, product manuals, legal documents, news articles, technical standards, e-books, web page content, and internal enterprise documents.

[0044] In this embodiment, one or more segmentation strategies are selected to segment the document based on the performance requirements of the application scenario, the document type, and the needs of subsequent processing. For example, the segmentation strategies may include, but are not limited to: segmentation strategies based on natural paragraphs, segmentation strategies based on fixed length, segmentation strategies based on semantic boundaries, and segmentation strategies based on sentence boundaries.

[0045] In step S220, the semantic similarity between adjacent text blocks is calculated.

[0046] For example, adjacent text blocks refer to two text blocks that are adjacent in position in the original document, that is, two text blocks with consecutive sequential indices. The semantic similarity is used to quantify the degree of semantic relevance between the two text blocks and is the core judgment basis for constructing semantic relationship chains in subsequent iterative diffusion. By calculating the semantic similarity between adjacent text blocks, it is possible to identify which text blocks in the document are semantically closely related and should belong to the same semantic relationship chain, and which text blocks have semantic breaks and should be used as the dividing point of different semantic relationship chains. The output of this step (i.e., the semantic similarity of each pair of adjacent text blocks) will be stored and used in the subsequent iterative diffusion process.

[0047] Specifically, the semantic similarity between adjacent text blocks is calculated using the following formula: Score(a,b) = (chunk_a, chunk_b).

[0048] Here, chunk_a and chunk_b are two adjacent text blocks to be compared; It is a semantic similarity calculation function; Score is the calculated semantic similarity score, which is usually a value between [0, 1]. The larger the value, the more similar the two people are.

[0049] In this embodiment, the specific semantic similarity calculation function It is divided into three parts. First, the joint input X is constructed: X = [[CLS],q1,q2,…q m [SEP],d1,d2,…d n ,[SEP]。

[0050] Where, q i and d i These refer to all the characters (tokens) corresponding to chunk_a and chunk_b, respectively; "[CLS]chunk_a [SEP]chunk_b [SEP]" is the input sequence, where [CLS] is the classification marker and [SEP] is the separator marker.

[0051] Secondly, there is combined coding: H = Encoder Cross (X).

[0052] Among them, Encoder Cross H is a Transformer-based encoder (such as BERT); H is the sequence of hidden states (context vector sequence) of all tokens output by the encoder.

[0053] Finally, the vector representations h_[CLS] of adjacent text blocks and the semantic similarity score (a,b) of adjacent text blocks are calculated: h_[CLS] = Head s (H) Score(a,b) = W·h_[CLS]+b Among them, Head s is the weight matrix used to extract the output head of a specific vector; h_[CLS] is the vector corresponding to the [CLS] label, which absorbs and integrates the effective information of chunk_a and chunk_b throughout the self-attention process; W and b are trainable linear layer weights and biases used to map the high-dimensional vector h_[CLS] to a scalar score.

[0054] The above formula can be used to quickly calculate the semantic similarity between all adjacent text blocks in the document set to be processed, so as to build a semantic relationship chain based on the semantic similarity between these adjacent text blocks.

[0055] In step S230, based on semantic similarity and a first threshold, the semantic relationship is iteratively diffused in adjacent directions to construct at least one semantic relationship chain, and each semantic relationship chain contains at least two semantically related text blocks.

[0056] For example, the first threshold can be a preset value used to determine whether adjacent text blocks should semantically belong to the same semantic relationship chain. The value range of this threshold is consistent with the similarity score, and is usually set between 0 and 1. In this embodiment, the specific value of the first threshold can be adjusted according to the application scenario. For example, a higher threshold (such as 0.75) produces a shorter semantic relationship chain, aggregating only highly similar text blocks together, which is suitable for scenarios with strict requirements for semantic consistency. A lower threshold (such as 0.55) produces a longer semantic relationship chain, allowing for a certain degree of semantic drift, which is suitable for retrieval scenarios that require a longer context. This threshold can be determined by optimizing the performance of downstream retrieval tasks on the validation set, or it can be configured by the system administrator according to specific needs.

[0057] Specifically, when the semantic similarity of a pair of adjacent text blocks is greater than or equal to a first threshold, the two text blocks are determined to be semantically related and should be included in the same semantic relationship chain. Conversely, when the semantic similarity is less than the first threshold, a semantic break is determined between the two text blocks, and they should not belong to the same semantic relationship chain; this position becomes the natural boundary of the semantic relationship chain. The iterative diffusion in this embodiment is a region growing method. Starting from a starting text block (called the anchor text block), it sequentially checks whether its adjacent text blocks meet the semantic association condition. If they do, the adjacent text block is included in the current semantic relationship chain, and the diffusion continues outward from the newly included text block as a new starting point until it can no longer expand.

[0058] In step S240, information is aggregated from the text block content in each semantic relation chain to generate corresponding semantic summary information.

[0059] For example, after the semantic relationship chain is constructed, each semantic relationship chain contains one or more semantically related text blocks. These text blocks together express a relatively complete semantic unit. In order for this semantic unit to be quickly identified and matched in the subsequent retrieval process, it is necessary to perform information aggregation processing on the text block content in each semantic relationship chain and generate corresponding semantic summary information.

[0060] Information aggregation can refer to the integration, compression, and refinement of semantic information scattered across multiple text blocks in a semantic relationship chain to form a more compact and generalized representation. The semantic summary information is the output form of this aggregation result, used to characterize the overall semantic content of the entire semantic relationship chain.

[0061] In step S250, the semantic relationship chain and its corresponding semantic summary information are associated with and saved with the corresponding text block.

[0062] For example, semantic relationship chains and their corresponding semantic summary information can be associated with the corresponding text blocks and stored in metadata. In order to achieve efficient association between text blocks and semantic relationship chains, metadata can be configured as a multi-dimensional data storage structure, which contains three types of data entities: text block entities, semantic relationship chain entities, and the association mapping between the two.

[0063] Specifically, the data fields of the text block entity: Each text block, as an independent storage unit, contains the following fields: First, the text block identifier, which is a globally unique string or integer used to uniquely identify a text block within the system. Identifiers can be generated in various ways, including: a combination of a document identifier and a text block sequence number, a UUID (Universally Unique Identifier), or an auto-incrementing integer.

[0064] Second, the document identifier, which records the unique identifier of the original document to which the text block belongs, is used to trace the source of the text block.

[0065] Third, the text block content stores the original text content of the text block, which is plain text after segmentation and cleaning.

[0066] Fourth, location information, which records the position of the text block in the original document, including the starting character offset, ending character offset, starting line number, ending line number, page number, and paragraph number. This information is used to locate the specific position in the document when displaying search results.

[0067] Fifth, the sequential index records the sequence number of the text block within the text block sequence of its parent document, used to maintain the original order of the document.

[0068] Sixth, the list of associated semantic relationship chains. This field is a list type and is used to store entries for all semantic relationship chains associated with this text block. Each entry contains the following subfields: semantic relationship chain identifier, semantic summary information (in text or vector form), and optionally, a list of identifiers for other text blocks in this semantic relationship chain.

[0069] Data fields of semantic relation chain entities: Each semantic relation chain, as an independent storage unit, contains the following fields: First, the semantic relationship chain identifier, which is a globally unique string or integer used to uniquely identify a semantic relationship chain in the system.

[0070] Second, it includes a list of text block identifiers, which stores all text block identifiers belonging to the semantic relationship chain in the order of the text blocks in the original document. This list is used to quickly recall all text blocks in the chain when semantic supplementation is needed.

[0071] Third, semantic summary information. This field stores semantic summary information generated for this semantic relationship chain, which may include one or more of the following forms: natural language summary in text form, keyword list, and vectorized semantic representation.

[0072] The association mapping between text blocks and semantic relationship chains: There is a many-to-many relationship between text blocks and semantic relationship chains: a text block can belong to multiple semantic relationship chains, and a semantic relationship chain contains multiple text blocks. In order to achieve fast query of this many-to-many relationship, one of the following two implementation methods can be adopted.

[0073] Method 1: Embedded storage, where related information is directly embedded into the "List of Related Semantic Relationships" field of the text block entity. Each text block record directly stores the identifiers and semantic summary information of its associated semantic relationship chains. The advantage of this method is that when recalling text blocks during retrieval, related information can be obtained without performing additional join queries, resulting in high query efficiency.

[0074] Method 2: Independent Association Table. A separate association table is created to store the mapping between text block identifiers and semantic relationship chain identifiers, as well as copies of the semantic summary information under that association. Each row of this association table contains: a text block identifier, a semantic relationship chain identifier, and the semantic summary information of that semantic relationship chain. When it is necessary to obtain the semantic relationship chain and semantic summary information associated with a specific text block, it is obtained by querying the association table. The advantage of this method is its high degree of storage structure standardization, avoiding data redundancy.

[0075] This application's embodiments utilize iterative diffusion based on semantic similarity to construct semantic relationship chains. This adaptively aggregates semantically related text blocks into coherent semantic units. Compared to contextual definitions based on fixed windows or mechanical positional relationships, this method more accurately reflects the actual semantic structure of a document. Furthermore, by generating semantic summary information, the semantic content of the entire relationship chain is compressed and represented, enabling rapid assessment of the overall relevance of the semantic relationship chain during subsequent retrieval without traversing all text blocks in the chain, significantly improving retrieval efficiency. The semantic relationship chain and semantic summary information are then associated and saved with each text block, achieving rapid mapping from text blocks to semantic units. This provides direct data support for filtering target text blocks based on semantic summary information and for semantic supplementation based on semantic relationship chains.

[0076] Figure 3 The flowchart illustrating the construction of at least one semantic relationship chain by iterative diffusion in adjacent directions according to an embodiment of this application is shown.

[0077] like Figure 3 As shown, the construction of at least one semantic relationship chain by iterative diffusion in adjacent directions can be specifically carried out in steps S310 and S320.

[0078] In step S310, taking the current text block as the anchor text block, the semantic similarity with other text blocks is recursively determined in the first direction and the second direction of the anchor text block, respectively, whether it is not lower than the first threshold; the first direction and the second direction are opposite directions.

[0079] For example, in the iterative diffusion construction process of semantic relationship chains, starting from the current text block, recursively judging in opposite directions to both sides to determine which adjacent text blocks should be included in the same semantic relationship chain. The current text block is called the anchor text block, which is the reference benchmark in the diffusion process. Centered on the anchor text block, the semantic similarity between the anchor text block and other text blocks is recursively judged in the first and second directions to see if it is not lower than a preset first threshold. In this embodiment, the first direction is defined as the document order direction, that is, the direction of diffusion from the current anchor text block to its right (index increasing direction) adjacent text blocks; the second direction is defined as the document reverse order direction, that is, the direction of diffusion from the current anchor text block to its left (index decreasing direction) adjacent text blocks. This setting is only for ease of description, and the interchange of the labels of the first and second directions does not affect the substantive content of the technical solution.

[0080] Specifically, an anchor text block refers to a text block that serves as a judgment benchmark at a certain moment during the diffusion process. During diffusion, using the anchor text block as a reference point, it is checked whether adjacent text blocks in a certain direction have sufficient semantic relevance to the anchor text block. The position of the anchor text block changes dynamically during diffusion. Specifically, when diffusion is performed in the first direction, the rightmost text block in the current semantic relationship chain is considered the anchor text block, used to determine whether text blocks further to the right can be included in the semantic relationship chain. When diffusion is performed in the second direction, the leftmost text block in the current semantic relationship chain is considered the anchor text block, used to determine whether text blocks further to the left can be included in the semantic relationship chain.

[0081] In the initial stage of constructing a new semantic relation chain, a seed text block is selected from an unlabeled text block, which becomes the initial anchor text block. At this point, the semantic relation chain contains only this anchor text block, which is also the leftmost and rightmost text block in the chain. During the diffusion process, whenever an adjacent text block is successfully incorporated into the current semantic relation chain, the newly incorporated text block becomes the new boundary of the semantic relation chain. For the ongoing diffusion direction, the newly incorporated text block becomes the new anchor text block, and based on this new anchor point, the next adjacent text block is determined in the same direction. This dynamic update ensures that the diffusion process can gradually advance along the document sequence until it falls below a first threshold or reaches the boundary of the document to be processed, at which point diffusion stops.

[0082] In step S320, if the text block is not lower than the first threshold, it is included in the current semantic relationship chain and continues to spread in the same direction until it is lower than the first threshold or reaches the boundary of the document to be processed and stops spreading.

[0083] For example, during the iterative diffusion construction of semantic relationship chains, the semantic similarity between the current anchor text block and its adjacent text blocks in a certain direction is judged. When the semantic similarity is not lower than a preset first threshold, the following operations are performed: the adjacent text block is included in the current semantic relationship chain, and diffusion continues in the same direction. The above process is repeated until one of the following two situations is encountered, at which point diffusion in that direction stops: first, the semantic similarity between the current anchor text block and its adjacent text blocks is lower than the first threshold; second, the diffusion direction has reached the boundary of the document to be processed, and there are no more distant adjacent text blocks. In this way, semantically related text blocks can be gradually aggregated into the same semantic relationship chain, and automatically stopped at semantic breaks or document boundaries, thereby achieving accurate definition of the semantic relationship chain boundaries.

[0084] Specifically, after calculating the semantic similarity between a text block and its neighboring text blocks, if the semantic similarity is greater than or equal to a first threshold, iterative diffusion is used to find other neighboring text blocks with the same semantic meaning, thereby automatically extracting the complete semantic chain from the text block sequence. In this example, the semantic chain function is as follows: L(i) = L_left(k)⊕[c i ]⊕L_right(j) L_right(j) =

[0085] L_left(k) =

[0086] Among them, c i It is the anchor text block, the starting point for building semantic chains; L(i) is the text block with c as its anchor. i The final semantic chain constructed for the anchor point; L_left(k) is the sub-chain obtained by expanding to the left, and L_right(j) is the sub-chain obtained by expanding to the right. "⊕" is the list connector, used to connect a new block to the beginning or end of the chain. N is the number of the entire ordered text block column; It is the first threshold, the value of which depends on the semantic similarity function. The output distribution is calculated as follows: =

[0087] For example, during the training phase, a dataset of text blocks with relevant labels... A function representing the semantic similarity between all related text blocks. The arithmetic mean of the scores; This indicates the spaces between all unrelated text blocks. The arithmetic mean of the function scores.

[0088] and The specific calculations are as follows: =

[0089] =

[0090] in, This represents the total number of text pairs labeled "relevant" in the dataset; This represents the total number of text pairs labeled "irrelevant" in the dataset; This indicates that the k-th positive sample pair (consisting of related text blocks) has passed through The calculated score; This indicates that the m-th negative sample pair (consisting of unrelated text blocks) has passed through The calculated score.

[0091] This application's embodiments quantify a document's semantic coherence into a numerical similarity score by calculating the semantic similarity between adjacent text blocks, and automatically identify semantic breakpoints based on an iterative diffusion mechanism. When the semantic similarity falls below a first threshold, the diffusion process terminates, and this location is determined as the natural boundary of the semantic relationship chain. This mechanism allows the boundary of the semantic relationship chain to adaptively fit the actual semantic structure of each document without requiring manual adjustments for different documents or domains, exhibiting good generalization ability. Furthermore, the recursive diffusion rule ensures semantic coherence within the semantic relationship chain. For example, each pair of adjacent text blocks in the semantic relationship chain satisfies the condition that the semantic similarity is not lower than the first threshold. This means that any adjacent text blocks within the chain have high semantic relevance, and the entire chain exhibits a smooth semantic transition or high consistency. This coherence ensures that when the retrieval system supplements along the semantic relationship chain, the supplemented text blocks are semantically naturally connected to the original target text blocks, rather than being awkwardly spliced ​​together.

[0092] In some embodiments, aggregating information from the text block content in each semantic relation chain to generate corresponding semantic summary information includes: inputting the text block content in the semantic relation chain as context into a large language model; and guiding the large language model to generate the semantic summary information through prompt word engineering.

[0093] For example, after constructing the semantic relationship chain, information aggregation can be performed on the text block content in each semantic relationship chain to generate corresponding semantic summary information. For instance, the text block content in the semantic relationship chain can be used as context input to a large language model, and the large language model can be guided to generate semantic summary information for that semantic relationship chain through prompt word engineering. In this embodiment, leveraging the powerful natural language understanding and generation capabilities of the large language model, semantic information scattered across multiple text blocks can be deeply integrated and refined to generate high-quality, highly readable semantic summary information. Prompt word engineering ensures that the summary output by the model conforms to preset format, length, and content requirements, making the generation process controllable and reproducible.

[0094] Specifically, a large language model can refer to a neural network language model that has been pre-trained on a large scale and has a rich number of parameters, possessing powerful natural language understanding and generation capabilities. Optional models include, but are not limited to: commercial closed-source models such as the GPT-4 series developed by OpenAI, open-source models such as the LLaMA series developed by Meta, and various domain-specific fine-tuning models. Prompt engineering involves inputting prompts based on actual needs to guide the large language model to generate output according to the expected format, style, and content. In this application, the goal of prompt engineering is to enable the large language model to generate semantic summary information that accurately summarizes the core content of the semantic relationship chain.

[0095] Figure 4 The illustration shows a schematic diagram of the construction of the diffusion semantic relationship chain and information aggregation in an embodiment of this application.

[0096] like Figure 4 As shown on the left, the document is first divided into several text blocks, labeled as 1, 2, 3, …, n. These text blocks are arranged in the order of their appearance in the original document, forming a linear sequence. Each text block is the basic unit for subsequent semantic similarity calculation and semantic relationship chain construction.

[0097] The central part of the attached diagram illustrates the calculation process for the semantic similarity between adjacent text blocks. The text block sequence is iterated through, and the semantic similarity score between each pair of adjacent text blocks is calculated sequentially. The diagram uses text blocks i-1, i, i+1, and i+2 as an example to demonstrate the specific method of similarity calculation. A cross-encoding model based on the encoder is used to calculate the semantic similarity between two text blocks. The input format is: the classification tag, the first text block, the separator tag, the second text block, and the separator tag are concatenated sequentially and input into the cross-encoder model. This model can jointly encode the two text blocks, capture the deep semantic interaction information between them, and output a semantic similarity score.

[0098] The following similarity calculations for adjacent pairs are clearly marked in the diagram: Score(i-1, i-2): The semantic similarity between text block i-1 and text block i-2; Score(i-1, i): The semantic similarity between text block i-1 and text block i; Score(i, i+1): The semantic similarity between text block i and text block i+1; Score(i+1, i+2): The semantic similarity between text block i+1 and text block i+2; The above calculation process is performed cyclically in the text block sequence until all adjacent pairs have been processed.

[0099] The diagram illustrates the decision-making logic for constructing semantic relationship chains through conditional statements. For example, the calculated semantic similarity score is compared with a first threshold. The specific map shows the comparison judgment between Score(i-1, i) and the first threshold, as well as the comparison judgment between Score(i, i+1) and the first threshold. When the semantic similarity score between adjacent text blocks is greater than the first threshold, it is determined that the two text blocks are semantically related and should belong to the same semantic relationship chain.

[0100] The semantic relationship chain constructed is represented in the form of "chunk_link" in the figure. For example, the figure shows a semantic relationship chain containing text blocks i-1, i, i+1, and i+2. This semantic relationship chain indicates that the semantic similarity threshold conditions are met between text blocks i-1 and i, i and i+1, and i+1 and i+2. Therefore, these four text blocks are aggregated into the same semantic relationship chain.

[0101] The diagram also shows the data structure of the semantic relationship chain, which represents the list of text block identifiers contained in the chain in the form of "[ i-1, i, i+1, i+2 ]" and is labeled with the complete information of "chunk_link", indicating that the semantic relationship chain records the identifiers of all the text blocks it contains and their order relationship.

[0102] As shown on the right side of the figure, for the constructed semantic relationship chain (chunk_link), an information aggregation operation is performed to generate corresponding semantic summary information (link_content_summary). The figure clearly indicates that the input source for information aggregation is the complete information of "chunk_link", that is, the content of all text blocks contained in the semantic relationship chain. The content of these text blocks is used as context information and input into the Large Language Model (LLM). When calling the Large Language Model, cue word engineering and context information aggregation are adopted. The cue words guide the Large Language Model to understand, integrate, and refine the content of the text blocks in the semantic relationship chain, and finally generate semantic summary information. The generated semantic summary information is represented in the form of "link_content_summary" and is used to represent the overall semantic content of the entire semantic relationship chain.

[0103] The last part of the attached diagram illustrates the storage process of the processing results. For example, the constructed semantic relationship chain (chunk_link) and its corresponding semantic summary information (link_content_summary) are associated and saved with the relevant text blocks. The diagram is labeled "Metadata Addition of chunk_link and link_content_summary," indicating that the semantic relationship chain identifier and corresponding semantic summary information are added to the metadata of each text block. This ensures that each text block carries information about its associated semantic relationship chain during storage. Finally, the above data is stored in a vector library that can store the semantic vector representation of text blocks and the vectorized summary of semantic relationship chains, supporting subsequent efficient similarity retrieval operations.

[0104] The method described in this application can calculate the semantic similarity between adjacent text blocks using a cross-encoder model, capturing deep semantic interaction information between two text blocks and achieving higher accuracy than calculating cosine similarity after independent encoding. Through a mechanism of loop traversal and threshold comparison, adaptive construction of semantic relationship chains is achieved without pre-setting the length or window size of semantic units. The complete information of the semantic relationship chain is input into a large language model, and semantic summaries are generated by combining prompt word engineering, ensuring that the summary information accurately summarizes the semantic content of the entire semantic relationship chain. Furthermore, the semantic relationship chain and its semantic summary information are associated with text blocks and stored in a vector library, establishing a fast mapping from text blocks to semantic units. This provides an efficient data access path for subsequent semantic summary-based retrieval and filtering, and semantic supplementation based on semantic relationship chains.

[0105] Figure 5 The flowchart illustrating step S130 of an embodiment of this application is shown in the illustration.

[0106] like Figure 5As shown, the above step S130 specifically includes steps S510 to S530.

[0107] In step S510, the semantic summary information corresponding to the candidate text block and the query statement are input into the large language model.

[0108] For example, in the process of candidate text block filtering based on a large language model, two types of information are used as input to the model: first, the semantic summary information associated with the candidate text block; and second, the query statement entered by the user. For instance, after organizing these two types of information according to specific organizational structures and format requirements, they are passed to the large language model through its application programming interface (API) or local inference interface. Upon receiving the input, the large language model processes the input content based on its internal semantic understanding capabilities and generates corresponding output results for subsequent steps to parse and use.

[0109] In step S520, the large language model is guided by prompt word engineering to determine the relevance between each candidate text block and the query statement, and output the relevance result between each candidate text block and the query statement.

[0110] For example, the role of prompt word engineering is to describe the task of "determining the relevance between candidate text blocks and query statements" in a form that the model can accurately understand, and to embed constraints such as judgment criteria and output format into prompt words, so that the model can generate output results that meet the requirements.

[0111] Specifically, a step-by-step decision-making strategy is adopted. This means that the semantic summary information of only one candidate text block and the query statement are input into the large language model at a time. The large language model then determines the relevance of that single candidate text block and outputs the result. This process is repeated for each candidate text block until all candidate text blocks have been decided. The advantage of this step-by-step decision-making strategy is that each decision task is performed independently, with clear context, no interference between them, and it is not limited by the model's context window length.

[0112] In this embodiment, a batch decision strategy can also be adopted, that is, the semantic summary information of multiple candidate text blocks and the query statement are input into the large language model at one time, requiring the model to make decisions on all candidate text blocks at the same time and output a list containing all decision results. The advantage of this batch decision strategy is that it significantly reduces the number of model calls, reduces latency and cost.

[0113] In step S530, at least one target text block is selected from several candidate text blocks based on the relevance results of each candidate text block to the query statement.

[0114] For example, the relevance results output by the large language model can be binary classification labels, where each candidate text block is judged as "relevant" or "irrelevant". The binary classification results can be filtered based on the labels; for instance, candidate text blocks judged as "relevant" are selected into the target text block set, while candidate text blocks judged as "irrelevant" are excluded. This filtering method is intuitive and efficient, suitable for scenarios where high filtering speed is required and the relevance judgment criteria are relatively clear.

[0115] Traditional methods rely on vector similarity calculations or keyword matching, which struggle to capture deep semantic relationships between queries and text, such as synonyms, implicit logic, or cross-sentence reasoning. This application's large language model, with its powerful semantic understanding capabilities, accurately identifies complex semantic relationships between queries and semantic summaries, significantly improving filtering accuracy. Furthermore, through prompt word engineering, the criteria and granularity for relevance determination can be flexibly defined according to different application scenarios and retrieval needs. For example, the model can be required to output binary classification labels, making this method adaptable to a wide range of applications. Unlike traditional supervised learning methods that require large amounts of labeled data, the large language model can complete relevance determination tasks with no labeled data (zero samples) or only a few examples (few samples), significantly reducing the data preparation costs for system deployment.

[0116] In some embodiments, determining whether to semantically supplement the target text block based on the semantic relationship chain associated with the target text block to obtain the retrieval results corresponding to the query statement, based on the similarity between the target text block and the query statement, includes: if the similarity between the target text block and the query statement is lower than a second threshold, then triggering a semantic supplementation process for the retrieval results to add other text blocks in the semantic relationship chain associated with the target text block to the retrieval results, thereby obtaining the retrieval results corresponding to the query statement.

[0117] For example, after selecting at least one target text block, the process proceeds to the semantic supplementation decision and retrieval result generation stage. For instance, based on the similarity between each target text block and the user's query statement, it is determined whether semantic supplementation is needed for the target text block, thereby obtaining the final retrieval result.

[0118] In this embodiment, a preset second threshold is set. When the similarity between a target text block and the query statement is lower than this second threshold, it is determined that the semantic information contained in the target text block itself is insufficient to fully respond to the user's query needs, and there may be problems of semantic fragmentation or missing information. In this case, a semantic supplementation process for the search results is triggered, incorporating other text blocks in the semantic relationship chain associated with the target text block into the search results, thereby supplementing the semantic information and obtaining more complete and accurate search results. This embodiment automates and conditions the semantic supplementation decision by comparing similarity with the second threshold, triggering supplementation only when the relevance of the target text block is insufficient, avoiding information redundancy caused by unconditional supplementation.

[0119] In some embodiments, the data processing method further includes: constructing a similarity distribution based on the highest similarity score between the text block retrieved in each historical query and the query statement; and determining the second threshold based on a specified quantile of the similarity distribution.

[0120] For example, the second threshold is not determined using a fixed preset value, but is dynamically calculated based on similarity data from historical query records. For instance, the highest similarity score between the text block obtained from each retrieval in historical queries and the query statement is collected, and this score data is used to construct a similarity distribution. Based on this, the specific value of the second threshold is determined according to a specified quantile of this similarity distribution. This method, by analyzing the naturally formed similarity distribution characteristics in historical retrieval behavior, enables the setting of the second threshold to have data-driven and adaptive adjustment capabilities, avoiding the subjectivity of manually set thresholds and the inapplicability of fixed thresholds in different query scenarios.

[0121] Specifically, after obtaining the target text block, in order to balance the retrieval length and semantic enhancement, a second threshold T corresponding to the similarity is set based on the historical query retrieval statistics. Based on T, it is determined whether the retrieval result enhancement is needed (i.e., whether semantic supplementation of the target text is needed). If the similarity between all target text blocks and the query statement is very low, the corresponding semantic relationship chain (chunk_link) is used to enhance the retrieval result. Then, the expanded result is deduplicated, and finally, an enhanced retrieval result set that ensures semantic coherence and avoids information redundancy is output. For example, in the actual authentication field, the above method can improve the retrieval recall rate from 80% to 85%, which is a significant improvement.

[0122] In this embodiment, the formula for calculating the second threshold T is as follows: max( ) = [ , , ] T =

[0123] in, It is the similarity between the i-th text block in the top-n text blocks retrieved and the query statement; It is the score of the highest similarity among the top-n text blocks recalled for the t-th query statement; It corresponds to all n query statements The set is obtained by sorting the similarity from low to high. In this embodiment, ( ) refers to a set of Quantiles, for example express The top 20% quantiles of the set, i.e., the second threshold T, can filter out the bottom 20% of text blocks with poor retrieval for semantic retrieval enhancement.

[0124] Figure 6 The illustration shows a schematic diagram of semantic supplementation based on diffusion semantic relationship chain in an embodiment of this application.

[0125] Figure 6 As shown, Query represents the user-input query, which can be a natural language question, keyword combination, or structured query expression. The vector library in the diagram is a pre-built storage unit containing the vectorized representation of each text block obtained after document segmentation. These vectorized representations are typically encoded using pre-trained language models (such as BERT, Sentence-BERT, etc.) to capture the semantic information of the text blocks. "Text block n" indicates that the vector library contains multiple text blocks, each with a unique identifier and corresponding semantic vector. During the retrieval phase, the query is encoded into a vector, and its similarity is calculated with the vectors of all text blocks in the vector library. The top few text blocks most semantically relevant to the query are retrieved, and these candidate text blocks serve as the basic input for subsequent filtering and enhancement processing.

[0126] The cue word engineering in the diagram guides the large language model to determine the relevance of candidate text blocks. After obtaining candidate text blocks, the semantic summary information associated with each candidate text block and the user's query are input into the large language model. The cue words guide the model to judge the relevance of each candidate text block to the query, thereby filtering out highly relevant text blocks (i.e., target text blocks). In the diagram, "text block i" represents the highly relevant target text block obtained after filtering. This target text block will serve as the basis for subsequent search result enhancement decisions.

[0127] In the search result enhancement phase, first obtain the similarity S between the target text block in the search results of the current query and the query statement max , S max is compared with the second threshold T to form a decision branch: When S max ≥ T (the "No" branch in the figure): it is determined that the current search result already has a high semantic matching degree, no result enhancement is required, the search process directly enters the output link, and the existing search result is taken as the final output.

[0128] When S max < T (the "Yes" branch in the figure): it is determined that the semantic matching degree of the current search result is insufficient, and a search result enhancement process needs to be executed. This judgment logic corresponds to the "on-demand supplementation" logic in the present application: supplementation is triggered only when the search quality is poor, so as to avoid unnecessary calculation and redundant information.

[0129] When it is determined that enhancement is required, the target text block corresponding to the current query (the "text block i" in the figure) is located. Each text block i is pre-associated with at least one semantic relationship chain (chunk_link). The semantic relationship chain characterizes the semantic association relationship between a plurality of text blocks, and includes an identifier list of other text blocks semantically related to the current text block. The figure shows a list of semantic summary information (link_content_summary1 to link_content_summaryn) associated with the semantic relationship chain, where these semantic summary information are used to characterize the overall semantic content of the corresponding semantic relationship chain, and can be used to screen highly relevant text blocks in the search phase. According to the semantic relationship chain (chunk_link) associated with the target text block, other text blocks included in the semantic relationship chain are recalled, and all text blocks in the semantic relationship chain except the current target text block are added to the search result set together to form an expanded result set.

[0130] After the expansion of the search results is completed, a deduplication operation may be performed to avoid introducing duplicate text blocks due to the expansion of the semantic relationship chain (for example, a plurality of different semantic relationship chains may include the same text block, or the expanded text block overlaps with the text block in the original search result). The deduplication operation is performed based on the unique identifier of the text block, to ensure that each text block appears only once in the final result set. After the above processing, the final search result is output. For a query that does not trigger enhancement, the output is the original vector search result. For a query that triggers enhancement, the output is the enhanced search result set that has been expanded through the semantic relationship chain and deduplicated. This result set not only retains semantic coherence (by introducing semantically associated context), but also avoids information redundancy (through on-demand triggering and deduplication processing).

[0131] Through the method of the embodiment of the present application, S can be used tomax Compared with the second threshold T, enhancement is triggered only when the retrieval quality is insufficient, avoiding unconditional context expansion. The enhancement operation is based on semantic relationship chains (chunk_link) rather than simple location proximity relationships, ensuring the semantic relevance between the supplementary content and the original result, and ensuring the simplicity of the output result through deduplication operation to avoid redundancy.

[0132] Figure 7 A block diagram of a data processing apparatus according to an embodiment of this application is shown schematically.

[0133] like Figure 7 As shown, the data processing device 700 includes a receiving module 710, a retrieval module 720, a filtering module 730, and a semantic supplementation module 740.

[0134] Specifically, the receiving module 710 is used to receive the query statement input by the user.

[0135] The retrieval module 720 is used to retrieve a number of candidate text blocks from the saved text block set according to the query statement; wherein each text block in the saved text block set is associated with at least one semantic relationship chain and corresponding semantic summary information. The semantic relationship chain is used to represent the semantic relationship between multiple text blocks, and the semantic summary information is used to represent the overall semantic content of the corresponding semantic relationship chain.

[0136] The filtering module 730 is used to filter at least one target text block from several candidate text blocks based on the semantic summary information associated with the candidate text blocks and the query statement.

[0137] The semantic supplementation module 740 is used to determine whether to perform semantic supplementation on the target text block based on the semantic relationship chain associated with the target text block, based on the similarity between the target text block and the query statement, so as to obtain the retrieval results corresponding to the query statement.

[0138] The data processing device 700 can pre-associate at least one semantic relationship chain and corresponding semantic summary information for each text block. During the retrieval process, it can not only rely on isolated text blocks but also use semantic summary information to determine the overall relevance of the semantic chain containing the text block to the query statement. Compared to traditional methods that mechanically splice context based solely on proximity, this invention can more accurately filter out target text blocks truly relevant to the query statement, avoiding the introduction of irrelevant content and significantly improving the semantic integrity of the retrieval results. Furthermore, traditional methods use fixed-range context splicing, which is difficult to adapt to diverse retrieval scenarios. This invention, after filtering out target text blocks, further selectively decides whether to perform semantic supplementation based on the similarity between the text block and the query statement, using its associated semantic relationship chain. This dynamic decision-making mechanism ensures that context expansion is no longer blind but flexibly triggered according to the actual needs of the query statement, thereby effectively reducing the introduction of redundant information, avoiding information overload, improving retrieval accuracy while controlling the scale of results.

[0139] In some embodiments, the data processing apparatus 700 described above can implement each step of the data processing method described above. The specific implementation process of the steps can be referred to the above method embodiments, and will not be repeated here.

[0140] Figure 8 A schematic block diagram of an electronic device according to an embodiment of this application is shown.

[0141] like Figure 8 As shown, the electronic device 8 of this embodiment includes a processor 801, a memory 802, and a computer program 803 stored in the memory 802 and executable on the processor 801. When the processor 801 executes the computer program 803, it implements the steps in the various method embodiments described above. Alternatively, when the processor 801 executes the computer program 803, it implements the functions of each module in the various device embodiments described above.

[0142] Electronic device 9 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 8 may include, but is not limited to, processor 801 and memory 802. Those skilled in the art will understand that... Figure 8 This is merely an example of electronic device 8 and does not constitute a limitation on electronic device 8. It may include more or fewer components than shown, or different components.

[0143] The processor 801 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0144] The memory 802 can be an internal storage unit of the electronic device 8, such as a hard disk or RAM of the electronic device 8. The memory 802 can also be an external storage device of the electronic device 8, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the electronic device 8. The memory 802 can also include both internal and external storage units of the electronic device 8. The memory 802 is used to store computer programs and other programs and data required by the electronic device.

[0145] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0146] This application provides a computer program product that stores a computer program, which, when executed by a processor, implements the steps in the above method embodiments.

[0147] If the integrated module is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0148] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0149] The units or modules described in the embodiments of this disclosure can be implemented in software or programmable hardware. The described units or modules can also be located in a processor, and the names of these units or modules do not necessarily constitute a limitation on the unit or module itself.

[0150] In another aspect, this disclosure also provides a computer-readable storage medium, which may be a computer-readable storage medium included in the electronic device or computer system described above; or it may be a standalone computer-readable storage medium not assembled into a device. The computer-readable storage medium stores one or more programs, which are used by one or more processors to execute the methods of the embodiments of this disclosure.

[0151] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

Claims

1. A data processing method, comprising: Receive query statements input by the user; According to the query statement, several candidate text blocks are retrieved from the saved text block set; wherein, each text block in the saved text block set is associated with at least one semantic relationship chain and corresponding semantic summary information, the semantic relationship chain is used to characterize the semantic relationship between multiple text blocks, and the semantic summary information is used to characterize the overall semantic content of the corresponding semantic relationship chain. Based on the semantic summary information associated with the candidate text blocks and the query statement, at least one target text block is selected from a plurality of candidate text blocks; Based on the similarity between the target text block and the query statement, determine whether to perform semantic supplementation on the target text block based on the semantic relationship chain associated with the target text block in order to obtain the retrieval results corresponding to the query statement.

2. The data processing method according to claim 1, characterized in that, Obtaining the saved set of text blocks includes: The collection of documents to be processed is divided into multiple text blocks; Calculate the semantic similarity between adjacent text blocks; Based on the semantic similarity and the first threshold, iterative diffusion is performed in adjacent directions to construct at least one semantic relationship chain, and each semantic relationship chain contains at least two semantically related text blocks; Information is aggregated from the text blocks in each semantic relation chain to generate corresponding semantic summary information; The semantic relationship chain and its corresponding semantic summary information are associated with and saved with the corresponding text blocks.

3. The data processing method according to claim 2, characterized in that, The iterative diffusion in adjacent directions to construct at least one semantic relationship chain includes: Using the current text block as the anchor text block, recursively determine whether the semantic similarity with other text blocks is not lower than the first threshold in both the first direction and the second direction of the anchor text block; the first direction and the second direction are opposite directions. If the value is not lower than the first threshold, the text block is included in the current semantic relationship chain and continues to spread in the same direction until it falls below the first threshold or reaches the boundary of the document to be processed and stops spreading.

4. The data processing method according to claim 2, characterized in that, The step of aggregating information from the text blocks in each semantic relation chain to generate corresponding semantic summary information includes: The text block content in the semantic relationship chain is used as context input to the large language model; The semantic summary information is generated by guiding the large language model through prompt word engineering.

5. The data processing method according to claim 1, characterized in that, Selecting at least one target text block from a plurality of candidate text blocks based on the semantic summary information associated with the candidate text blocks and the query statement includes: Input the semantic summary information corresponding to the candidate text block and the query statement into the large language model; The large language model is guided by prompt word engineering to determine the relevance of each candidate text block to the query statement one by one, and output the relevance result of each candidate text block to the query statement. Based on the relevance of each candidate text block to the query statement, at least one target text block is selected from the candidate text blocks.

6. The data processing method according to claim 1, characterized in that, Based on the similarity between the target text block and the query statement, determine whether to semantically supplement the target text block based on the semantic relationship chain associated with the target text block, so as to obtain the retrieval results corresponding to the query statement, including: If the similarity between the target text block and the query statement is lower than a second threshold, a semantic supplementation process for the search results is triggered to add other text blocks in the semantic relationship chain associated with the target text block to the search results, thereby obtaining the search results corresponding to the query statement.

7. The data processing method according to claim 6, characterized in that, Also includes: A similarity distribution is constructed based on the highest similarity score between the retrieved text block and the query statement in each historical query. The second threshold is determined based on the specified quantile of the similarity distribution.

8. A data processing apparatus, comprising: The receiving module is used to receive the query statement input by the user; The retrieval module is used to retrieve a number of candidate text blocks from the saved text block set according to the query statement; wherein each text block in the saved text block set is associated with at least one semantic relationship chain and corresponding semantic summary information, the semantic relationship chain is used to characterize the semantic relationship between multiple text blocks, and the semantic summary information is used to characterize the overall semantic content of the corresponding semantic relationship chain. The filtering module is used to filter at least one target text block from a plurality of candidate text blocks based on the semantic summary information associated with the candidate text block and the query statement; The semantic supplementation module is used to determine whether to perform semantic supplementation on the target text block based on the semantic relationship chain associated with the target text block, based on the similarity between the target text block and the query statement, so as to obtain the retrieval results corresponding to the query statement.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method of any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images