Data processing method and system

Abstract

The embodiment of the invention provides a data processing method and system. In the method, a data processing system performs target rich feature extraction on an original query statement of a user, and constructs a plurality of different retrieval query conditions based on a plurality of target sub-features corresponding to the target rich features and the original query statement of the user. Then, the data processing system obtains a target retrieval result set from the knowledge base through a plurality of retrieval query conditions; further, the data processing system generates a target answer based on the retrieval result set using a large language model.

Description

Technical Field

[0001] This specification relates to the field of information retrieval technology, and in particular to a data processing method and system. Background Technology

[0002] With the rapid development of deep learning technology, retrieval-enhanced generation systems improve the accuracy and timeliness of generated content by combining large language models with external knowledge retrieval. These systems have broad application prospects in various fields such as question-answering, intelligent assistants, and content creation.

[0003] In related technologies, retrieval-enhanced generation systems typically consist of three stages: knowledge base indexing, retrieval, and generation. In the retrieval stage, these systems generally rely on the vector similarity between the user's query and text blocks in the knowledge base to retrieve text blocks. However, in scenarios with large knowledge bases and complex knowledge structures, this retrieval method cannot comprehensively retrieve relevant information from massive corpora.

[0004] Therefore, there is an urgent need for a retrieval and generation scheme to improve the overall performance of the retrieval and generation system.

[0005] The background information is merely information known only to the inventor and does not imply that such information had entered the public domain before the date of this application, nor does it imply that it could be considered prior art in this disclosure. Summary of the Invention

[0006] This manual provides a data processing method and system that is applicable to query scenarios where the accuracy of the user's original query statement is required to a certain extent.

[0007] Firstly, this specification provides a data processing method, comprising: extracting target rich features from a user's original query statement to obtain multiple target sub-features corresponding to the target rich features, wherein the target rich features are features with a hierarchical structure; constructing multiple retrieval query conditions based on the user's original query statement and the multiple target sub-features corresponding to the target rich features; retrieving a knowledge base based on the multiple retrieval query conditions to obtain a target retrieval result set, wherein the target retrieval result set includes multiple corpora; and using a large language model based on the target retrieval result set to determine the target answer corresponding to the user's original query statement.

[0008] In some embodiments, the multiple target sub-features corresponding to the target rich feature include: a first type of sub-feature and a second type of sub-feature; the step of extracting target rich features from the user's original query statement to obtain the multiple target sub-features corresponding to the target rich feature includes: extracting target rich features from the user's original query statement using a large language model to obtain the first type of sub-features corresponding to the target rich feature included in the user's original query statement; and expanding based on the first type of sub-features to obtain the second type of sub-features corresponding to the target rich feature.

[0009] In some embodiments, constructing multiple retrieval query conditions based on the user's original query statement and multiple target sub-features corresponding to the target rich feature includes: concatenating the feature vector of the user's original query statement with the multiple target sub-features corresponding to the target rich feature to obtain retrieval query conditions corresponding to the multiple target sub-features respectively.

[0010] In some embodiments, the step of retrieving the knowledge base based on the multiple search query conditions to obtain a target search result set includes: performing parallel retrieval of the knowledge base based on the multiple search query conditions to obtain an initial search result set, wherein the initial search result set includes multiple corpora; obtaining the actual concentration corresponding to each corpus type in the initial search set based on the corpus type; and truncating the corpora corresponding to each corpus type in the initial search results based on the actual concentration, ideal concentration weight, and concentration adjustment factor corresponding to each corpus type to obtain a target search result set, wherein the sum of the ideal concentration weights corresponding to each corpus type is 1.

[0011] In some embodiments, the step of truncating the corpora corresponding to each corpus type in the initial search results based on the actual concentration, ideal concentration weight, and concentration adjustment factor corresponding to each corpus type to obtain the target search result set includes: truncating the corpora corresponding to each corpus type in the initial search results based on the actual concentration, ideal concentration weight, and concentration adjustment factor corresponding to each corpus type to obtain an intermediate search result set; and pruning the intermediate search result set based on the relevance of each corpus in the intermediate search result set to obtain the target search result set.

[0012] In some embodiments, the step of pruning the intermediate search result set based on the relevance of each corpus in the intermediate search result set to obtain the target search result set includes: obtaining the highest and lowest relevance of the corpus in the intermediate search result set; determining the redundancy of the intermediate search result set based on the highest and lowest relevance; and pruning the intermediate search result set if the redundancy is greater than a preset redundancy threshold to obtain the target search result set.

[0013] In some embodiments, the step of pruning the intermediate retrieval result set to obtain the target retrieval result set when the redundancy exceeds a preset redundancy threshold includes: obtaining a pruning score based on the relevance of each corpus in the intermediate retrieval result set and the pruning adjustment factor when the redundancy exceeds the preset redundancy threshold; obtaining corpus with a relevance greater than the pruning score from the intermediate retrieval result set and adding them to the target retrieval result set until the number of corpus in the target retrieval result set exceeds a maximum threshold or reaches the maximum number of lexical units in the context window of a large language model.

[0014] In some embodiments, obtaining the pruning score based on the relevance of each corpus in the intermediate retrieval result set and the pruning adjustment factor includes: obtaining a preset quantile based on the relevance of each corpus in the intermediate retrieval result set; and obtaining the pruning score based on the preset quantile and the pruning adjustment factor.

[0015] In some embodiments, the step of performing parallel retrieval of the knowledge base based on the plurality of retrieval query conditions to obtain an initial retrieval result set includes: determining the hierarchical knowledge bases corresponding to the plurality of retrieval query conditions based on the plurality of retrieval query conditions; and performing parallel retrieval of the hierarchical knowledge bases corresponding to the plurality of retrieval query conditions based on the plurality of retrieval query conditions to obtain an initial retrieval result set.

[0016] Secondly, this specification also provides a data processing system, including at least one storage medium and at least one processor, wherein the at least one storage medium stores at least one instruction set for data querying; the at least one processor is communicatively connected to the at least one storage medium, wherein the at least one processor reads the at least one instruction set during operation and executes the method described in any of the first aspects above according to the instructions of the at least one instruction set.

[0017] As can be seen from the above technical solutions, the data processing method and system provided in this specification involve the data processing system extracting rich target features from the user's original query statement and constructing multiple distinct retrieval query conditions based on multiple target sub-features corresponding to the rich target features and the user's original query statement. By dynamically planning diverse retrieval query conditions, collaborative retrieval of the knowledge base from different perspectives is achieved, overcoming the problem of key corpora being overwhelmed by single-path retrieval in related technologies, thus improving the accuracy and recall rate of retrieval. The data processing system obtains a target retrieval result set from the knowledge base based on multiple retrieval query conditions. Even in scenarios with complex knowledge structures, it can still efficiently and comprehensively recall relevant corpora, while simultaneously increasing the diversity of the corpora contained in the target retrieval result set, providing richer and more relevant contextual information for the subsequent generation of target answers. The data processing system uses a large language model to generate target answers based on the target retrieval result set. Due to the diversity of the corpora contained in the target retrieval result set, the generated target answers are also more accurate.

[0018] The data processing methods and other functions of the system provided in this specification are partially listed in the following description. The inventive aspects of the data processing methods and systems provided in this specification can be fully explained through practice or use of the methods, apparatus, and combinations described in the detailed examples below. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A schematic diagram illustrating an application scenario of a data processing system provided according to an embodiment of this specification is shown. Figure 2 A schematic diagram of the hardware structure of a computing device provided according to some embodiments of this specification is shown; Figure 3 A schematic flowchart of a data processing method according to an embodiment of this specification is shown; Figure 4 This diagram illustrates a flowchart of rich feature extraction and retrieval query condition generation according to another embodiment of this specification; Figure 5 A flowchart illustrating a process for acquiring relevant corpus according to some embodiments of this specification is shown; and Figure 6A schematic flowchart of a data processing method according to another embodiment of this specification is shown. Detailed Implementation

[0021] The following description provides specific application scenarios and requirements for this specification, intended to enable those skilled in the art to make and use the contents of this specification. Various partial modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments and applications without departing from the spirit and scope of this specification. Therefore, this specification is not limited to the embodiments shown, but rather to the widest scope consistent with the claims.

[0022] The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not restrictive. For example, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” used herein may also include the plural forms. When used in this specification, the terms “comprising,” “including,” and / or “containing” mean that the associated integers, steps, operations, elements, and / or components are present, but do not exclude the presence of one or more other features, integers, steps, operations, elements, components, and / or groups, or that other features, integers, steps, operations, elements, components, and / or groups may be added to the system / method.

[0023] Considering the following description, these and other features of this specification, as well as the operation and function of the related components of the structure, and the economy of assembly and manufacture of the parts, can be significantly improved. All of these form part of this specification with reference to the accompanying drawings. However, it should be clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of this specification. It should also be understood that the drawings are not drawn to scale.

[0024] The flowcharts used in this specification illustrate operations implemented according to some embodiments of this specification. It should be clearly understood that the operations in the flowcharts may not be implemented in a sequential order. Instead, the operations may be implemented in reverse order or simultaneously. Furthermore, one or more additional operations may be added to the flowcharts. One or more operations may be removed from the flowcharts.

[0025] In this specification, "X includes at least one of A, B, or C" means that X includes at least A, or X includes at least B, or X includes at least C. That is, X may include only one of A, B, and C, or any combination of A, B, and C, as well as other possible content / elements. The arbitrary combination of A, B, and C can be A, B, C, AB, AC, BC, or ABC.

[0026] In this specification, unless explicitly stated otherwise, the relationships between structures can be direct or indirect. For example, when describing "A is connected to B," unless it is explicitly stated that A and B are directly connected, it should be understood that A can be directly connected to B or indirectly connected to B. Similarly, when describing "A is on top of B," unless it is explicitly stated that A is directly above B (AB is adjacent and A is above B), it should be understood that A can be directly above B or indirectly above B (AB is separated by other elements, and A is above B). And so on.

[0027] It should be noted that the user data obtained in this manual is authorized by the user and does not involve user privacy.

[0028] For ease of description, the terms that will appear later in this manual will be explained first.

[0029] Retrieval-Augmented Generation (RAG) is a framework or technique that combines information retrieval systems with the functionality of Large Language Models (LLMs). It aims to improve the accuracy, timeliness, and relevance of content generated by LLMs, constrain the model's generation boundaries, and reduce illusions.

[0030] Large language models are typically used in the field of artificial intelligence, particularly in Natural Language Processing (NLP), and especially in large machine learning models with a large number of parameters, computational resources, and complex network structures. They are characterized by their powerful feature representation and feature understanding capabilities, enabling them to better capture patterns and rules in data, thereby understanding and generating natural language.

[0031] Rich features refer to multi-level related information features with clear semantics parsed from user queries described in natural language. Examples include: regional features (such as province-city-district / county), time features (such as year-month-day), intent features, entity features, and attribute features. Each type of rich feature can be further decomposed into multiple levels of sub-features, forming a tree-like or hierarchical feature structure.

[0032] The embodiments described in this specification are applied to the field of information retrieval technology and aim to provide a data processing method that improves the accuracy and comprehensiveness of retrieval results by introducing a hierarchical target feature extraction and a retrieval mechanism with multiple retrieval query conditions, thereby improving the accuracy of the target answer.

[0033] To improve the retrieval performance of data processing systems in complex query scenarios, the method provided in this specification extracts hierarchical target features from the user's original query statement, constructs multiple differentiated retrieval query conditions, and thereby retrieves more high-quality relevant corpora from the knowledge base through these differentiated query conditions, providing richer contextual basis for the subsequent generation of target answers. See the following description for details: In the embodiments described in this specification, the data processing system first extracts target rich features from the user's original query statement. Based on multiple target sub-features corresponding to the target rich features and the user's original query statement, it constructs multiple retrieval query conditions that are different in feature composition. Each retrieval query condition represents a specific retrieval perspective or constraint combination. Then, the data processing system performs a multi-dimensional retrieval of the knowledge base by executing the above multiple retrieval query conditions to obtain a set of retrieval results. In this process, although each retrieval query condition is generated based on the same user's original query statement, the different target sub-features combined create differentiated retrieval query conditions, ensuring the diversity of the retrieval result set. The data processing system uses the retrieval result set as context input to a large language model to obtain the target answer corresponding to the user's original query statement.

[0034] It should be noted that the above description of application scenarios is only one of the many usage scenarios provided in this specification. Those skilled in the art should understand that when the data processing methods and systems provided in this specification are applied to other usage scenarios, their implementation methods and technical effects are similar.

[0035] Figure 1 A schematic diagram illustrating an application scenario of a data processing system provided according to an embodiment of this specification is shown.

[0036] like Figure 1 As shown, scenario 100 may include user terminal 110, knowledge base 120 and data processing system 130.

[0037] User terminal 110 can be a device used by the user, such as a smartphone, tablet, personal computer, smart wearable device, etc. The user can enter the original query statement (e.g., "How do I renew my passport in District B of City A?") through an application, browser, or dialog interface on user terminal 110.

[0038] The knowledge base system 120 serves as an external knowledge source in the retrieval enhancement generation system, storing a large amount of knowledge. The knowledge base system 120 may include one or more of the following databases: vector database, relational database, document database, used to store various types of indexed (e.g., vectorized, keyword indexed, etc.) corpora.

[0039] After receiving the user's original query statement from the user terminal 110, the data processing system 130 first extracts target rich features from the user's original query statement. Then, based on multiple target sub-features corresponding to the target rich features and the user's original query statement, the data processing system 130 generates multiple different search query conditions. These multiple search query conditions aim to retrieve information from the knowledge base from multiple dimensions. In other words, the data processing system 130 searches the knowledge base system 120 in parallel or sequentially based on multiple search query conditions. After obtaining the search result set, the data processing system 130 provides the search result set to the large language model, which then generates the target answer corresponding to the user's original query statement based on the search result set and returns it to the user terminal 110 for display to the user. The data processing system 130 typically possesses certain computing resources and can perform data retrieval, data processing, and run the large language model.

[0040] In some embodiments, the data processing system 130 and the knowledge base system 120 can be deployed on the same physical server or server cluster, or they can be distributed across different network nodes and communicate with each other via a network. The user terminal 110 interacts with the data processing system 130 via a network (such as the Internet or a mobile communication network).

[0041] Figure 2 A schematic diagram of the hardware structure of a computing device 200 according to some embodiments of this specification is shown. This computing device 200 can be used as... Figure 1 The data processing system 130 is described in some embodiments. When the data processing system 130 employs a device cluster, the computing device 200 can be any one of the devices in the data processing system 130.

[0042] like Figure 2 As shown, the computing device 200 includes at least one storage medium 230 and at least one processor 220. In some embodiments, the computing device 200 may further include an internal communication bus 210. In some embodiments, the computing device 200 may further include a communication port 250. In some embodiments, the computing device 200 may further include I / O components 260.

[0043] The internal communication bus 210 can connect different system components, including storage medium 230 and processor 220. I / O component 260 supports input / output between computing device 200 and other components.

[0044] Communication port 250 is used for data communication between computing device 200 and the outside world. For example, computing device 200 can connect to a network through communication port 250.

[0045] Storage medium 230 may include a data storage device. The data storage device may be a non-transitory storage medium or a temporary storage medium. For example, the data storage device may include one or more of a disk 232, a read-only storage medium (ROM) 234, or a random access storage medium (RAM) 236. Storage medium 230 also includes at least one instruction set stored in the data storage device. The instruction set is computer program code, which may include programs, routines, objects, components, data structures, procedures, modules, etc., that execute the data processing methods provided in this specification.

[0046] At least one processor 220 is communicatively connected to at least one storage medium 230 via an internal communication bus 210. The at least one processor 220 is used to execute at least one instruction set. When the data processing system 130 is running, the at least one processor 220 reads at least one instruction set and executes the data processing methods provided in this specification according to the instructions of the at least one instruction set.

[0047] Processor 220 can execute all the steps included in the data processing method. Processor 220 can be in the form of one or more processors. Processor 220 can issue execution instructions. Processor 220 may include one or more hardware processors, such as microcontrollers, microprocessors, reduced instruction set computers (RISC), application-specific integrated circuits (ASICs), application-specific instruction set processors (ASIPs), central processing units (CPUs), graphics processing units (GPUs), physical processing units (PPUs), microcontroller units, digital signal processors (DSPs), field-programmable gate arrays (FPGAs), advanced RISC machines (ARMs), programmable logic devices (PLDs), any circuit or processor capable of performing one or more functions, or any combination thereof.

[0048] For illustrative purposes only, only one processor 220 is shown in the accompanying drawings of the computing device 200. However, it should be noted that the computing device 200 may also include multiple processors. Therefore, the operation and / or method steps disclosed herein may be executed by a single processor or by multiple processors in combination, as described herein. For example, if processor 220 of the computing device 200 in this specification executes steps A and B, it should be understood that steps A and B may also be executed jointly or separately by two different processors 220 (e.g., a first processor executes step A, a second processor executes step B, or the first and second processors jointly execute steps A and B).

[0049] Figure 3A schematic flowchart of a data processing method according to an embodiment of this specification is shown; this data processing method P300 can be executed by a data processing system 130. Figure 3 As shown, the method P300 provided in this specification may include S310-S350, wherein: S310: Extract target rich features from the user's original query statement and obtain multiple target sub-features corresponding to the target rich features. The target rich features are features with a hierarchical structure.

[0050] In some embodiments, the target rich feature can be one of the rich features in a rich feature set, such as a regional feature, a time feature, or an organizational structure feature. The rich feature set includes multiple rich features of different types, and these rich features can be pre-defined; for example, the rich feature set can pre-definedly include various types of rich features such as semantic, entity, time, and regional features. Alternatively, the rich features included in the rich feature set can also be obtained by parsing and extracting the user's original query statement using a large language model. Multiple target sub-features refer to the specific manifestations of a hierarchical target rich feature at different levels. For example, when the target rich feature is a regional feature, the target sub-features may include provincial, municipal, and district / county features. Similarly, when the target rich feature is a time feature, the target sub-features may include year, month, week, and day features.

[0051] The data processing system extracts rich feature sets from the user's original query statement based on a large language model. This allows for the extraction of fine-grained features from the original query statement as rich features. These rich features can include several types, such as: regional features (explicitly mentioned or implicit geographical location information in the query), intent features (the core purpose of the user's query or the action they want to perform), entity features (specific objects or events involved in the query), time features (time points or time periods related to the query), and attribute features (specific attributes describing entities or intents).

[0052] In some embodiments, the data processing system can input the user's original query statement along with a preset rich feature extraction prompt template into the large language model. The prompt template guides the large language model to identify and structurally output multiple rich features from the user's original query statement. For example, if the user's original query statement is "I would like to inquire how individual business owners in District B of City A can apply for a food business license online?", the large language model, based on the user's original query statement and the preset rich feature extraction prompt template, can determine that the current target rich feature is a regional feature. The corresponding target sub-features include: city-level features and district-level features. Therefore, the large language model may output the following structured rich feature set: Regional characteristics: {City-level characteristics: "City A", District-level characteristics: "District B"} Intent characteristic: "Application" Physical characteristics: "Food Business License" Main characteristic: "Individual business owner" Channel characteristics: Online S330: Based on the user's original query statement and the multiple target sub-features corresponding to the target rich features, construct multiple retrieval query conditions.

[0053] In some embodiments, the data processing system can generate retrieval query conditions corresponding to each target sub-feature based on multiple target sub-features. This method of constructing different retrieval query conditions based on multiple target sub-features corresponding to rich target features achieves the transformation of the user's original query statement from a single semantic to a multi-dimensional rich feature set, avoiding the problem of inaccurate retrieval results caused by a single retrieval path.

[0054] Search query conditions define the specific instructions for initiating a query against the knowledge base. A search query condition can be viewed as a combination of the feature vector of the user's original query statement and one or more target sub-features. For example, taking the target feature as a region, the target sub-features include: City A, District B, meaning City A and District B each correspond to a target sub-feature. A search query condition refers to the result of concatenating the feature vector of the user's original query statement with City A; or, the result of concatenating the feature vector of the user's original query statement with District B; or, the result of concatenating the feature vector of the user's original query statement with both City A and District B. Multiple search query conditions are distinct, representing different query perspectives or constraint strengths.

[0055] As an example, knowledge itself has a hierarchical structure in many application scenarios, such as government affairs (e.g., provincial, municipal, district, county levels), legal scenarios, and enterprise management. The user's original query may also implicitly or explicitly indicate a certain basic level. For example, regional features typically include a province-city-district / county-street level, time features may include a year-month-day level, and organizational structure features generally also have corresponding levels.

[0056] Figure 4 This diagram illustrates a flowchart of a target feature extraction and retrieval query condition generation process according to another embodiment of this specification, such as... Figure 4 As shown, after receiving the user's original query statement, the data processing system can extract target rich features from the user's original query statement. The data processing system constructs multiple retrieval query conditions based on multiple target sub-features corresponding to the target rich features and the original query statement.

[0057] As an example, the data processing system can perform multi-path hierarchical retrieval based on multiple search query conditions, thereby retrieving and aggregating corpora from multiple dimensions. This expands the scope of retrieved corpora while maintaining retrieval efficiency, ensuring comprehensive recall of relevant data. In other words, each search query condition retrieves corpora from different angles. The use of multiple search query conditions reduces the risk of missing key data due to a single search path, enabling multi-dimensional and multi-level queries of the knowledge base, thus improving the diversity and comprehensiveness of the retrieved corpora.

[0058] S350: Based on the multiple search query conditions, the knowledge base is searched to obtain a target search result set, which includes multiple related corpora.

[0059] The data processing system 130 can retrieve the knowledge base based on multiple search query conditions in parallel to minimize the overall search latency; or it can be executed serially or scheduled based on different priorities corresponding to different search query conditions.

[0060] As an example, when the data processing system 130 performs a search based on each search query condition, it can determine at least one corpus whose similarity meets preset requirements by calculating the similarity between the user's original query statement and the corpus in the knowledge base. The similarity can be semantic similarity or vector similarity, etc., and the preset requirements can be preset quantity requirements or preset similarity threshold requirements, etc.

[0061] The data processing system 130 can convert the user's original query statement into a feature vector through an embedding model. Subsequently, the data processing system constructs multiple search query conditions based on the feature vector and rich feature set of the user's original query statement. Each search path includes at least one target sub-feature corresponding to the feature vector of the user's original query statement and the target rich feature.

[0062] After multiple search queries are executed, each query returns corresponding search results. These results may be empty or contain at least one corpus. The data processing system 130 merges the corpora returned by all search queries to obtain a set of search results containing a large amount of corpus.

[0063] S370: Using the large language model based on the target retrieval result set, determine the target answer corresponding to the user's original query statement.

[0064] In some embodiments, the large language model can determine the target answer based on the user's original query and the corpus in the target retrieval result set. In this approach, because the corpus in the target retrieval result set is obtained based on different retrieval query conditions, the corpus contained in the target retrieval result set has a higher information density, thereby guiding the large language model to generate a more accurate, richer, and more reliable target answer.

[0065] In summary, the data processing method and system provided in this specification involve the data processing system extracting rich target features from the user's original query statement and constructing multiple distinct retrieval query conditions based on multiple target sub-features corresponding to the rich target features and the user's original query statement. By dynamically planning diverse retrieval query conditions, collaborative retrieval of the knowledge base from different perspectives is achieved, overcoming the problem of key corpora being overwhelmed by single-path retrieval in related technologies, thus improving retrieval accuracy and recall. The data processing system obtains a target retrieval result set from the knowledge base based on multiple retrieval query conditions, enabling efficient and comprehensive recall of relevant corpora even in scenarios with complex knowledge structures. It also enhances the diversity of the corpora contained in the target retrieval result set, providing richer and more relevant contextual information for the subsequent generation of target answers. The data processing system uses a large language model to generate target answers based on the target retrieval result set. Due to the diversity of the corpora contained in the target retrieval result set, the generated target answers are more accurate.

[0066] In some embodiments, the multiple target sub-features corresponding to the target rich features include: a first type of sub-features and a second type of sub-features. The data processing system can use a large language model to extract target rich features from the user's original query statement, obtain the first type of sub-features corresponding to the target rich features included in the user's original query statement, and expand based on the first type of sub-features to obtain the second type of sub-features corresponding to the target rich features.

[0067] Specifically, the data processing system first uses a large language model to extract target rich features from the user's original query statement, obtaining the first type of sub-features corresponding to the target rich features included in the user's original query statement. Subsequently, based on the first type of sub-features, the data processing system expands to obtain at least one second type of sub-features according to a predefined hierarchical structure relationship.

[0068] As an example, the expansion methods may include one or more of the following: upward expansion, downward expansion, or parallel expansion. Upward expansion can be single-level or multi-level upward expansion, which helps to recall corpora with a wider applicability. When the corpus contained in the base level is incomplete, parallel expansion can supplement the corpus. When the base level is high, a subset of the base level can be obtained to expand downward to the corresponding subset of the base level; downward expansion can be single-level or multi-level downward expansion, thereby enriching the recalled corpus.

[0069] As an example, taking the target rich feature as a regional feature, when the first type of sub-feature is region C, the corresponding upward-expanding second type of sub-feature can be one or more of the following: city C corresponding to region C, province C corresponding to city C. The downward-expanding second type of sub-feature may be one or more of the following: county C, county D, or township C under region C, etc. The parallel-expanding second type of sub-feature may include: regions D, E, F, etc., which are adjacent to region C.

[0070] In some embodiments, the data processing system can concatenate the feature vector of the user's original query statement with multiple target sub-features corresponding to the target rich features to obtain retrieval query conditions corresponding to the multiple target sub-features respectively.

[0071] The feature vector of the user's original query can be any rich feature in the rich feature set other than the target rich feature. For example, the types of rich features included in the rich feature set include: region features. Intent characteristics Entity features Time characteristics Attributes and characteristics For example, the data processing system can construct hierarchical features corresponding to different regional levels based on regional features and other rich features. Specifically, provincial features correspond to hierarchical features. = + + + + City-level features and corresponding hierarchical features = + + + + Hierarchical features corresponding to district / county level features Furthermore, the data processing system extends the regional characteristics: for example, by analyzing... Parallel expansion yields features of neighboring regions. Through the The upward expansion yields national characteristics. .

[0072] Subsequently, the data processing system, based on different regional characteristics and combined with the feature vector of the user's original query statement, Construct multiple search query conditions. For example, a data processing system can construct multiple search query conditions according to regional hierarchy. The number of search query conditions... The number of non-empty region features. .

[0073] Among them, multiple search query conditions Each search query condition can include a feature vector. and regional characteristics : =( , (), , (), , (), , (), , ).

[0074] In some embodiments, each search query condition may also include pre-configured search parameters such as the number of searches and search similarity.

[0075] In some embodiments, before extracting the rich feature set, the data processing system can also rewrite and expand the user's original query statement based on the large language model to enrich the user's original query statement from a semantic level, thereby expanding the coverage of semantic recall and avoiding the problem of retrieval omissions due to differences in expression.

[0076] As an example, taking the regional level as the basic layer, if the user's original query contains a first-type sub-feature, the data processing system can extract information from the original query to obtain the first-type sub-feature contained therein. Alternatively, if the user's original query does not contain a valid first-type sub-feature, other user information, such as the user's Internet Protocol (IP) address or location information provided by the Global Positioning System (GPS), can be used as the default first-type sub-feature. Or, if the user's original query does not contain a valid first-type sub-feature, it can be treated as a query problem without regional constraints.

[0077] In some embodiments, since the data processing system retrieves a large amount of corpus based on multiple search query conditions, it may contain duplicate or similar corpus. Therefore, after obtaining an initial search result set based on multiple search query conditions, the data processing system can process the initial search result set to reduce the amount of corpus contained in the initial search result set, thereby reducing the complexity of subsequent large language model processing.

[0078] The methods for processing the initial search result set may include one or more of the following: deduplicating the initial search result set; adjusting the relevant corpora of each type in the initial search result set according to a preset proportion; or pruning the initial search result set when the redundancy of the initial search result set exceeds a preset redundancy threshold.

[0079] In some embodiments, the data processing system performs parallel retrieval of the knowledge base based on multiple retrieval query conditions. After obtaining the initial retrieval result set, it is necessary to further evaluate the proportion of each type of corpus in the initial retrieval set, that is, to evaluate whether the corpus is overly biased to a certain type of knowledge. If the proportion is too high or too low, the preset proportion of the relevant corpus needs to be adaptively adjusted through preset rules.

[0080] Specifically, the data processing system performs parallel searches of the knowledge base based on the multiple search query conditions to obtain an initial search result set. Subsequently, based on the corpus type, the data processing system obtains the actual concentration corresponding to each corpus type in the initial search set, and based on the actual concentration, ideal concentration weight, and concentration adjustment factor corresponding to each corpus type, it truncates the corpus corresponding to each corpus type in the initial search results to obtain a target search result set, wherein the sum of the ideal concentration weights corresponding to each corpus type is 1.

[0081] Among them, the data processing system can solve the problem of imbalance in the types of corpora that may exist in multiple corpora by presetting the preset proportion of different types of corpora and adjusting the different types of corpora according to the preset proportion.

[0082] In some embodiments, the initial search result set Each corpus Each is labeled with its corresponding information type. ∈{ , , ..., For example, product manual corpora may include concept definitions, usage instructions, possible malfunctions, and maintenance instructions; government affairs corpora may include policy interpretations, procedures, processing locations, and frequently asked questions.

[0083] For different information types, the ideal preset weight W = { , , ..., } is used to reflect the proportion of this information type in the target answer. The sum of the preset proportions of each type is 1, that is... =1.

[0084] A certain type of information In the initial search result set The actual proportion for: = Proportion deviation = - .in, The percentage adjustment factor is a preset constant value used to control and adjust the intensity of the percentage.

[0085] Specifically, for types where the actual proportion is greater than the preset proportion, the data processing system can remove some data from the corpus of that type to reduce the actual proportion; and / or, for types where the actual proportion is lower than the preset proportion, all data of that type is retained. In other words, a type where the actual proportion is greater than the preset proportion indicates that the proportion of relevant data for that type is too high, which may lead to a problem of information type homogeneity. This could cause the large language model to over-rely on the excessively high proportion of data when generating the target answer, thus requiring dilution. For types where the actual proportion is lower than the preset proportion, all data of that type is retained to avoid further loss of data.

[0086] In some embodiments, the data processing system can determine the relationship between the actual percentage and the preset percentage based on the percentage deviation. Wherein, in the percentage deviation... When the calculation result is not an integer, it can be rounded up, and the rounded result can be used as the percentage deviation. The final calculation results are then obtained. Subsequently, the data processing system truncates each type of corpus where the actual proportion of the proportion deviation indicator is greater than the preset proportion, ensuring that the actual proportion of the relevant corpus for each type is less than or equal to the preset proportion, thus obtaining a target retrieval result set after proportion adjustment. Through the above adjustments, a target retrieval result set with a more balanced distribution across indicator types can be obtained. The target retrieval result set ensures that the corpus information input into the large language model is diverse, thereby guiding the large language model to generate more comprehensive and richer target answers, avoiding the problem of biased answers due to a single information source.

[0087] In some embodiments, for types where the actual proportion is greater than a preset proportion, the data processing system can remove some less relevant data from the relevant data of that type based on the relevance between each relevant data corpus in that type and the user's original query statement. After adjusting the relevant data of different types by the preset proportion, the data processing system can ensure that the adjusted search result set contains relevant data of different types, and that the proportions of different types of relevant data are balanced, thereby improving the comprehensiveness and accuracy of subsequent target answer determination.

[0088] In some embodiments, the data processing system can truncate the corpora corresponding to each corpus type in the initial search results based on the actual concentration, ideal concentration weight, and concentration adjustment factor corresponding to each corpus type, to obtain an intermediate search result set. Then, based on the relevance of each corpus in the intermediate search result set, the data processing system performs pruning on the intermediate search result set to obtain the target search result set.

[0089] In other words, after obtaining the initial search result set, the data processing system first adjusts the concentration of the initial search result set to obtain an intermediate search result set. Then, based on relevance, it prunes the intermediate search results to obtain the target search result set. This method of performing multiple processes on the initial search result set to obtain the target search result set, through concentration adjustment and pruning, makes the final target search result set more accurate and avoids interference from irrelevant or redundant information.

[0090] In some embodiments, the data processing system obtains the highest and lowest relevance scores of the corpus in the intermediate search result set, and determines the redundancy of the intermediate search result set based on the highest and lowest relevance scores. Then, if the redundancy exceeds a preset redundancy threshold, the data processing system prunes the intermediate search result set to obtain the target search result set. Specifically, pruning the corpus in the intermediate search result set can remove redundant corpus (not duplicate corpus) that has low relevance to the user's original query.

[0091] As an example, this is based on the relevance between each relevant corpus and the user's original query. Let's take the calculation of redundancy in the intermediate retrieval result set as an example. This involves calculating the relevance score for each relevant corpus. It is determined based on the relevance score between the relevant corpus and the user's original query statement, that is... ( )= ( ).

[0092] Furthermore, the data processing system uses the highest relevance score from the intermediate search result set. and lowest correlation score The redundancy H is determined by the ratio of the difference in relevance scores to the highest relevance score. That is, H = .

[0093] As an example, when the redundancy exceeds a preset redundancy threshold, the data processing system can directly remove the less relevant data from the intermediate search result set based on the relevance between each relevant data set and the user's original query statement, until the redundancy is less than or equal to the preset redundancy threshold. Alternatively, the data processing system can remove relevant data from the intermediate search result set whose relevance to the user's original query statement is lower than the pruning score, and add relevant data with a relevance higher than the pruning score to the target search result set to obtain the target search result set.

[0094] The aforementioned method of pruning scores to remove corpora from the intermediate retrieval result set systematically filters out corpora with low relevance to the user's original query through a quantitative standard. This ensures that the target retrieval result set input to the large language model is highly focused, thereby reducing the cognitive load on the large model when processing noisy information. Simultaneously, the target retrieval result set shortens the content length input to the large language model, optimizing computational efficiency while maintaining information integrity.

[0095] In some embodiments, when the redundancy exceeds a preset redundancy threshold, the data processing system can obtain a pruning score based on the relevance and pruning adjustment factor of each corpus in the intermediate retrieval result set. Subsequently, the data processing system obtains corpora with relevance greater than the pruning score from the intermediate retrieval result set and adds them to the target retrieval result set until the number of corpora in the target retrieval result set exceeds a maximum threshold or reaches the maximum number of lexical units in the context window of a large language model.

[0096] The data processing system reduces noise in the target retrieval result set by only acquiring corpora with relevance greater than the pruning score and adding them to the target retrieval result set. This is achieved by removing relevant corpora with relevance lower than the pruning score and adding only corpora with relevance greater than the pruning score.

[0097] In some embodiments, the data processing system may obtain a preset quantile based on the relevance of each corpus in the intermediate retrieval result set, and then obtain the pruning score based on the preset quantile and the pruning adjustment factor.

[0098] As an example, the data processing system can calculate the upper quartile of the relevance of each corpus in the intermediate retrieval result set, and determine the pruning score based on the upper quartile and a preset adjustment factor, wherein the preset adjustment factor is a preset constant value.

[0099] In some embodiments, the data processing system constructs a relevance score set S={ based on the relevance scores between all n corpora in the intermediate retrieval result set and the user's original query statement. }, where n is a positive integer greater than or equal to 1. Subsequently, the data processing system calculates the upper quartiles of this relevance score set S. This is the 75th percentile. Furthermore, the data processing system determines the pruning score T based on the upper quartile Q3 and a preset adjustment factor α, where T = α * Q3, and α is a constant less than 1. This dynamic method of determining the pruning score allows the pruning results to be adaptively adjusted according to the quality distribution of each retrieval result set.

[0100] The above-mentioned method of dynamically determining pruning scores can dynamically identify and remove redundant and low-relevance corpora from the intermediate retrieval result set, thereby ensuring the quality of the final retained corpora, optimizing the performance of the data processing system, and ensuring the generation of high-quality and highly relevant target answers.

[0101] In some embodiments, to remove redundant information from the corpus, the data processing system can perform parallel searches of the knowledge base based on multiple search query conditions to obtain an initial search result set. Before concentration adjustment, the data processing system can deduplicate the corpora returned by different search query conditions based on the corpus's identification information and the relevance between each corpus and the user's original query statement, obtaining the deduplicated initial search result set. Subsequently, the data processing system performs subsequent concentration adjustment, pruning, and other processing based on the initial search result set until the target search result set is obtained. Specifically, by dedupplicating the corpora in the search result set, the data processing system can eliminate duplicate corpora in the search result set.

[0102] Figure 5 A flowchart illustrating a corpus acquisition process according to some embodiments of this specification is shown, such as... Figure 5 As shown, after the data processing system obtains the user's original query statement and multiple target sub-features corresponding to the target rich features, it constructs multiple distinct retrieval query conditions based on the user's original query statement and multiple target sub-features (target sub-feature 1, target sub-feature 2, ..., target sub-feature n) to retrieve the knowledge base according to different retrieval query conditions, thereby ensuring that the information of each target sub-feature is not interfered with.

[0103] In this case, the data processing system may return a large amount of duplicate data based on multiple search queries, and the data corpora lack a hierarchical structure, with all data arranged in a flat list relationship. Therefore, the data processing system needs to perform deduplication to aggregate the data.

[0104] Continue as Figure 5 As shown, the data processing system merges the retrieved corpora corresponding to each search query condition and performs global sorting based on the relevance between the corpora and the user's original query. Then, the data processing system deduplicates the merged and sorted corpora and returns the deduplicated corpora as the initial set of deduplicated search results.

[0105] As an example, deduplication can be based on: the corpus's identifier information and / or the corpus's text information. For instance, when the corpus is stored in the knowledge base, it is assigned identifier information corresponding to the source document (a source document may be split / sliced ​​into multiple corpus blocks, and these multiple corpus blocks have the same identifier information), and deduplication can be performed directly based on the identifier information. Alternatively, hash comparison or direct comparison can be performed on the text content of the corpus to remove corpus parts with identical or highly similar content.

[0106] The deduplication method can be as follows: When the data processing system detects that the corpora retrieved under different search query conditions may be duplicated, for example, corpus A retrieved under search query A and corpus B retrieved under search query B have the same actual identifier information (corpus A and its identifier are the same as the identifier of corpus B), that is, corpus A and corpus B point to the same source document, the data processing system can retain the corpus with higher relevance based on the relevance between the corpus and the user's original query statement. In addition, the data processing system adds source hierarchical identifier information to the retained corpus. This hierarchical identifier information can be tag information, such as [Source Hierarchy: City A].

[0107] In some embodiments, the data processing system can use a re-ranking model to score and sort the corpus in the initial search result set, and determine the target answer based on the re-ranked initial search result set. This method of re-ranking the initial search result set allows for a secondary sorting of the relevant corpus contained within the initial search result set. By re-scoring and re-ranking, the error problems caused by coarse recall methods such as vector retrieval can be reduced.

[0108] As an example, after the data processing system deduplicates the globally sorted corpus, it can select the k most relevant corpora from the deduplicated corpus as the final deduplicated output corpus.

[0109] In some embodiments, the data processing system can determine the hierarchical knowledge bases corresponding to the multiple search query conditions. Then, the data processing system performs parallel searches in the hierarchical knowledge bases corresponding to the multiple search query conditions to obtain an initial set of search results.

[0110] As an example, the aforementioned hierarchical knowledge base can be formed by dividing the overall knowledge according to factors such as knowledge content, source, domain, time, degree of structuring, or access permissions, resulting in multiple logically or physically separated storage units or index partitions. Together, they constitute a multi-level, organized knowledge base system. For instance, a large enterprise's knowledge base can be divided into multiple parallel sub-knowledge bases, such as a product technical document library, a market analysis report library, an internal failure case library, and an external paper library. Each sub-knowledge base may be further subdivided into smaller hierarchical units based on dimensions such as year and product line.

[0111] The above-described approach, which first matches the search query conditions to the corresponding hierarchical knowledge base and then performs parallel retrieval within that knowledge base, reduces the amount of data scanned for each query and lowers retrieval latency by distributing the search query conditions to smaller, more focused hierarchical knowledge bases. Furthermore, this multi-path parallel execution fully utilizes distributed computing or input / output (I / O) resources, thereby improving retrieval efficiency.

[0112] Figure 6 A schematic flowchart of a data processing method according to another embodiment of this specification is shown, such as... Figure 6 As shown, the data processing system searches the knowledge base based on multiple search query conditions and obtains an initial search result set. Then, it deduplicates the initial search result set to obtain a deduplicated initial search result set. Subsequently, the data processing system adjusts the corpora of each type in the deduplicated initial search result set according to a preset proportion to obtain an intermediate search result set. When the redundancy of the intermediate search result set exceeds a preset redundancy threshold, the data processing system prunes the search result set to obtain the target search result set.

[0113] Specifically, after receiving the user's original query statement, the data processing system extracts target rich features from the original query statement, obtaining the first type of sub-features corresponding to the target rich features included in the original query statement. Subsequently, the data processing system expands based on the first type of sub-features to obtain the second type of sub-features corresponding to the target rich features, and then constructs multiple retrieval query conditions based on multiple target sub-features. The data processing system queries the corresponding corpus in the knowledge base in parallel based on multiple retrieval query conditions, and merges the corpora obtained from each retrieval query condition to obtain an initial retrieval result set A. Then, based on the identification information or content of each corpus contained in the initial retrieval result set A, the data processing system deduplicates the corpus in the initial retrieval result set A to remove duplicate corpora, retaining corpora with high relevance to the user's original query statement, and annotating their hierarchical source, thereby obtaining a deduplicated initial retrieval result set B.

[0114] Continue as Figure 6 As shown, the data processing system adjusts the corpora corresponding to each type in the initial deduplicated search result set B based on preset proportions. Specifically, the data processing system calculates the actual proportion of different types of corpora according to their types and compares the actual proportions with the corresponding preset proportions. Subsequently, based on relevance, the data processing system downsamples corpora with actual proportions greater than the preset proportions and retains all corpora with actual proportions less than the preset proportions. Then, the data processing system re-scores the relevance of the corpora after proportion adjustment and re-sorts them based on the relevance scores to obtain the intermediate search result set C.

[0115] Continue as Figure 6 As shown, after the data processing system adjusts the initial deduplicated search result set B based on a preset ratio, the resulting intermediate search result set C may still contain a large number of corpora, exceeding the load of the large language model, or it may contain semantic redundancy or low-quality information. Therefore, the data processing system can calculate the redundancy of the intermediate search result set C based on the highest and lowest relevance of the corpora in the intermediate search result set C. When the redundancy exceeds a preset redundancy threshold, the intermediate search result set C is traversed to perform pruning. Specifically, the data processing system dynamically calculates the pruning score and determines the relationship between each corpus in the search result set and the pruning score. When the relevance is less than the pruning score, the corpus is removed / discarded; when the relevance is greater than the pruning score, the corpus is added to the target search result set. This process yields the pruned target search result set D. The pruning score is calculated based on the relevance of each corpus in the intermediate search result set and the pruning adjustment factor; different search result sets may correspond to different pruning scores.

[0116] Then, the data processing system sends the target retrieval result set D to the large language model, which generates the target answer to the user's original query and returns the target answer to the user. This processing method removes redundant information while preserving the effective information density of the context, ensuring that the retrieval result set received by the large language model is high-quality and low-redundancy, thereby improving the accuracy and effectiveness of the target answer.

[0117] This specification, in another aspect, provides a computer-readable non-transitory storage medium storing at least one set of instructions for performing data query executable instructions. When the at least one set of instructions is executed by a processor, it instructs the processor to implement the steps of the data processing method P300 described herein. In some possible embodiments, various aspects of this specification may also be implemented as a program product comprising program code. When the program product is run on the data processing system 130, the program code causes the data processing system 130 to perform the steps of the method P300 described herein. The program product for implementing the above method may employ a portable compact disc read-only memory (CD-ROM) containing program code and may run on the data processing system 130. However, the program product of this specification is not limited thereto. In this specification, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system. The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable storage medium may also be any readable medium other than a readable storage medium that can send, propagate, or transmit programs for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination of the foregoing. Program code for performing the operations described herein may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar programming languages.

[0118] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0119] In summary, after reading this detailed disclosure, those skilled in the art will understand that the foregoing detailed disclosure may be presented by way of example only and may not be restrictive. Although not explicitly stated herein, those skilled in the art will understand that this specification requires various reasonable changes, improvements, and modifications to the embodiments. These changes, improvements, and modifications are intended to be made by this specification and are within the spirit and scope of the exemplary embodiments described herein.

[0120] Furthermore, certain terms in this specification have been used to describe embodiments of this specification. For example, "an embodiment," "an embodiment," and / or "some embodiments" mean that a particular feature, structure, or characteristic described in connection with that embodiment may be included in at least one embodiment of this specification. Therefore, it is to be emphasized and understood that two or more references to "an embodiment" or "an embodiment" or "alternative embodiment" in various parts of this specification do not necessarily refer to the same embodiment. Moreover, specific features, structures, or characteristics may be suitably combined in one or more embodiments of this specification.

[0121] It should be understood that in the foregoing description of the embodiments in this specification, various features are combined in a single embodiment, drawing, or description for the purpose of simplifying the description and to aid in understanding a feature. However, this does not mean that the combination of these features is necessary, and those skilled in the art, upon reading this specification, may readily identify some of the devices as separate embodiments. That is, the embodiments in this specification can also be understood as an integration of multiple secondary embodiments. And the content of each secondary embodiment is valid even if it contains fewer than all the features of a single foregoing disclosed embodiment.

[0122] Every patent, patent application, publication of a patent application, and other material, such as articles, books, specifications, publications, documents, and literature (excluding any related historical examination documents), cited in this disclosure is incorporated herein for all purposes, including, for example, in the specification and claims of this disclosure. However, in the event of any inconsistency or conflict between the descriptions, definitions, and / or terms used in the foregoing and those used in this disclosure, the descriptions, definitions, and / or terms used in this disclosure shall prevail.

[0123] Finally, it should be understood that the embodiments disclosed herein are illustrative of the principles of the embodiments described in this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are merely examples and not limitations. Those skilled in the art can implement the applications described in this specification using alternative configurations based on the embodiments in this specification. Therefore, the embodiments in this specification are not limited to the embodiments precisely described in the applications.

Claims

1. A data processing method, wherein, The method includes: The user's original query statement is subjected to target rich feature extraction to obtain multiple target sub-features corresponding to the target rich feature, wherein the target rich feature is a feature with a hierarchical structure. Based on the user's original query statement and the multiple target sub-features corresponding to the target rich features, multiple retrieval query conditions are constructed; The knowledge base is searched based on the multiple search query conditions to obtain a target search result set, which includes multiple corpora; and Based on the target retrieval result set, a large language model is used to determine the target answer corresponding to the user's original query statement.

2. The method according to claim 1, wherein, The multiple target sub-features corresponding to the target rich features include: a first type of sub-features and a second type of sub-features; The step of extracting target rich features from the user's original query statement and obtaining multiple target sub-features corresponding to the target rich features includes: The target rich features of the user's original query statement are extracted using a large language model to obtain the first type of sub-features corresponding to the target rich features included in the user's original query statement. Based on the first type of sub-features, the second type of sub-features corresponding to the target rich features are obtained.

3. The method according to claim 2, wherein, The process involves constructing multiple retrieval query conditions based on the user's original query statement and multiple target sub-features corresponding to the target rich features, including: The feature vector of the user's original query statement is concatenated with multiple target sub-features corresponding to the target rich features to obtain the retrieval query conditions corresponding to the multiple target sub-features.

4. The method according to claim 1, wherein, The process of searching the knowledge base based on the multiple search query conditions to obtain a target search result set includes: Parallel retrieval of the knowledge base is performed based on the multiple retrieval query conditions to obtain an initial retrieval result set, which includes multiple corpora; Based on the corpus type, obtain the actual concentration corresponding to each corpus type in the initial retrieval set; Based on the actual concentration, ideal concentration weight, and concentration adjustment factor corresponding to each corpus type, the corpus corresponding to each corpus type in the initial search results is truncated to obtain the target search result set, wherein the sum of the ideal concentration weights corresponding to each corpus type is 1.

5. The method according to claim 4, wherein, The method involves extracting corpora corresponding to each corpus type from the initial search results based on the actual concentration, ideal concentration weight, and concentration adjustment factor for each corpus type, to obtain the target search result set, including: Based on the actual concentration, ideal concentration weight, and concentration adjustment factor corresponding to each corpus type, the corpus corresponding to each corpus type in the initial search results is truncated to obtain an intermediate search result set. Based on the relevance of each corpus in the intermediate search result set, the intermediate search result set is pruned to obtain the target search result set.

6. The method according to claim 5, wherein, The step of pruning the intermediate search result set based on the relevance of each corpus in the intermediate search result set to obtain the target search result set includes: Obtain the highest and lowest relevance scores of the corpus in the intermediate search result set; Based on the highest and lowest relevance scores, the redundancy of the intermediate search result set is determined; If the redundancy exceeds a preset redundancy threshold, the intermediate search result set is pruned to obtain the target search result set.

7. The method according to claim 6, wherein, When the redundancy exceeds a preset redundancy threshold, the intermediate search result set is pruned to obtain the target search result set, including: If the redundancy exceeds a preset redundancy threshold, a pruning score is obtained based on the relevance of each corpus in the intermediate retrieval result set and the pruning adjustment factor. The corpus with a relevance greater than the pruning score is obtained from the intermediate retrieval result set and added to the target retrieval result set until the number of corpus in the target retrieval result set is greater than the maximum threshold or reaches the maximum number of lexical units in the context window of the large language model.

8. The method according to claim 7, wherein, The step of obtaining the pruning score based on the relevance and pruning adjustment factor of each corpus in the intermediate retrieval result set includes: Based on the relevance of each corpus in the intermediate retrieval result set, a preset quantile is obtained; The pruning score is obtained based on the preset quantile and the pruning adjustment factor.

9. The method according to claim 4, wherein, The parallel retrieval of the knowledge base based on the multiple retrieval query conditions yields an initial set of retrieval results, including: Based on the multiple search query conditions, determine the hierarchical knowledge base corresponding to each of the multiple search query conditions; Based on the multiple search query conditions, parallel searches are performed on the hierarchical knowledge bases corresponding to the multiple search query conditions to obtain an initial set of search results.

10. A data processing system, comprising: At least one storage medium storing at least one instruction set for data processing; as well as At least one processor is communicatively connected to the at least one storage medium, wherein the at least one processor reads the at least one instruction set during operation and executes the method according to any one of claims 1-9 based on the instructions of the at least one instruction set.

Images