A query optimization method and system based on detection enhancement generation technology
Through query optimization methods based on detection enhancement generation technology, intelligently process documents in different formats, block processing and deep learning model embedded vector generation, the shortcomings of search enhancement generation technology in text document processing and query preprocessing are solved, and data processing is improved Efficiency and query accuracy improve user experience.
Patent Information
- Application Number
- CN202411179047.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-27
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-08-27
AI Technical Summary
The existing search enhancement generation technology has shortcomings in the processing, chunking, vector mapping and query preprocessing of text documents, resulting in low data processing efficiency and low query accuracy.
Using query optimization method based on detection enhancement generation technology, combining pre-training models with similarity and similarity degree prediction model, optimize the processing flow of documents and queries.
Improve the universality and accuracy of data processing, enhance the accuracy and coverage of query statements, improve the relevance and user satisfaction of search results, solve the problems of low query accuracy and unreasonable result sorting in traditional search systems. .
Smart Images

Figure CN118690001B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of big data technology, and in particular to a query optimization method and system based on detection enhancement generation technology. Background Art
[0002] In recent years, with the rapid development of artificial intelligence technology, especially the widespread application of big data processing, cloud computing and deep learning technology, retrieval augmentation generation (RAG) technology has emerged. RAG technology has achieved significant improvements in database query quality and efficiency by combining retrieval technology and large model generation technology. This technology can not only understand the deep semantics of user queries, but also retrieve content that is highly relevant to user queries from massive data, and convert these contents into coherent and accurate answers through large model generation technology.
[0003] Although RAG technology has made significant progress in the field of information retrieval, it still has many shortcomings in text document processing, segmentation, vector mapping and query preprocessing. Summary of the invention
[0004] The purpose of the present invention is to provide a query optimization method and system based on detection enhancement generation technology to solve the problem of how to optimize the shortcomings of retrieval enhancement generation technology in the prior art in terms of text document processing, segmentation, vector mapping and query preprocessing.
[0005] In a first aspect, the present application provides a query optimization method based on detection enhancement generation technology, comprising:
[0006] Get the first document to be processed;
[0007] Determine a document format type of the first document to be processed, and preprocess the first document to be processed based on a document preprocessing method corresponding to the document format type to obtain a second document to be processed;
[0008] The second document to be processed is processed in blocks using the trained first model to obtain a third document to be processed, wherein the first model is a correlation calculation model;
[0009] Obtaining a first query statement input by a user, cleaning and performing standard processing on the first query statement, and obtaining a second query statement;
[0010] Acquire historical query data of the user, and perform context expansion on the second query statement based on context expansion of the historical query data to obtain a third query statement;
[0011] Based on the third document to be processed, the first query statement, the third query statement, the pre-trained second model, and the pre-trained third model, a first processing result is obtained, wherein the second model generates embedding vectors of documents and query statements based on a deep learning model recurrent neural network, and the third model generates embedding vectors of query statements based on a language model;
[0012] Using a preset word meaning expansion library, a preset synonym database, semantic analysis technology and the first query statement, the first processing result is reviewed to obtain a second processing result, where the second processing result is whether the first processing result needs to be updated;
[0013] The pre-trained fourth model is used to perform recall processing based on the second processing result to obtain the target processing result. The fourth model is a similarity prediction model, which sorts and screens the second processing result to obtain the target processing result.
[0014] In an optional embodiment, preprocessing the first document to be processed based on the document format type to obtain the second document to be processed includes:
[0015] When the document format type of the first document to be processed is a PDF format type, using optical character recognition technology to extract text from the first document to be processed to obtain extracted text, using natural language processing technology to perform entity recognition and syntax analysis on the extracted text to obtain a document structure, and based on the document structure, identifying and removing headers, footers and watermarks in the extracted text to obtain a second document to be processed;
[0016] When the document format type of the first document to be processed is a Word format type, the first document to be processed is extracted using an application programming interface to obtain extracted text and a document structure, and a header, a footer and a watermark in the extracted text are identified and removed based on the document structure to obtain a second document to be processed;
[0017] When the document format type of the first document to be processed is an Excel format type, the first document to be processed is analyzed to determine the table format, the header row, the data column and the data type, and the data integrity of the first document to be processed is checked based on the table format, the header row, the data column and the data type to obtain a check result, and the first document to be processed is corrected and marked based on the check result and a predefined rule to obtain a second document to be processed;
[0018] When the document format type of the first document to be processed is the HTML format type, the web crawler technology is used to capture the target web page data based on the first document to be processed, and the target web page data is analyzed by the analyzer to obtain the web page structure of the first document to be processed, and the advertisements and navigation menus of the first document to be processed are identified and removed based on the web page structure to obtain the second document to be processed.
[0019] In an optional embodiment, the first model is obtained by training a corpus; and the second document to be processed is processed in blocks using the trained first model to obtain a third document to be processed, including:
[0020] Using the first model to extract vectors from each paragraph in the second to-be-processed document, to obtain a paragraph embedding vector for each paragraph;
[0021] Using the first model to calculate similarity based on the paragraph embedding vectors of each paragraph, a similarity set between adjacent paragraphs is obtained;
[0022] The second document to be processed is processed in blocks based on the inter-segment similarity set using the first model to obtain a third document to be processed.
[0023] In an optional embodiment, context expansion is performed on the second query statement based on historical query data to obtain a third query statement, including:
[0024] Using natural language technology to perform part-of-speech tagging, dependency syntax analysis, and keyword extraction on the second query sentence to obtain query relationship data;
[0025] The user's historical query data is obtained, and the second query statement is context-expanded based on the historical query data and the query relationship data to obtain a third query statement.
[0026] In an optional embodiment, based on the third document to be processed, the first query statement, the third query statement, the pre-trained second model, and the pre-trained third model, a first processing result is obtained, the second model is a deep learning model recurrent neural network, and the third model is a language model, including:
[0027] Inputting the first query statement and the third to-be-processed document into a pre-trained second model to generate a first embedding vector;
[0028] Input the third query statement into the pre-trained second model to obtain a second embedding vector;
[0029] Optimizing the second embedding vector using an optimization algorithm;
[0030] A first processing result is obtained based on the first embedding vector and the second embedding vector.
[0031] In an optional embodiment, the first processing result is reviewed using a preset word meaning expansion library, a preset synonym database, a semantic analysis technology and the first query statement to obtain a second processing result, including:
[0032] Using a preset word meaning expansion library to perform keyword extraction and keyword deformation processing on the first query statement to obtain a first review result;
[0033] Using a preset synonym database to perform keyword replacement processing on the first query statement to obtain a second review result;
[0034] Perform semantic analysis on the first query statement using a preset semantic analysis technology to obtain a third review result;
[0035] The first processing result is reviewed based on the first review result, the second review result and the third review result to obtain a second processing result.
[0036] In an optional embodiment, the method further includes: monitoring in real time whether the query statement has changed, and if it is detected that the query statement has changed, recalculating the first embedding vector.
[0037] In a second aspect, the present invention provides a query optimization system based on detection enhancement generation technology, comprising:
[0038] A preprocessing unit is configured to obtain a first document to be processed; and determine a document format type of the first document to be processed, and preprocess the first document to be processed based on a document preprocessing method corresponding to the document format type to obtain a second document to be processed;
[0039] A block division unit is configured to perform block processing on the second document to be processed by using the trained first model to obtain a third document to be processed, wherein the first model is a correlation calculation model;
[0040] a cleaning unit configured to obtain a first query statement input by a user, clean the first query statement and perform standard processing on the first query statement to obtain a second query statement;
[0041] An expansion unit is configured to obtain historical query data of the user, and perform context expansion on the second query statement based on context expansion of the historical query data to obtain a third query statement;
[0042] A processing unit is configured to obtain a first processing result based on a third document to be processed, a first query statement, a third query statement, a pre-trained second model, and a pre-trained third model, wherein the second model generates embedding vectors of documents and query statements based on a deep learning model recurrent neural network, and the third model generates embedding vectors of query statements based on a language model;
[0043] A review unit is configured to review the first processing result by using a preset word meaning expansion library, a preset synonym database, a semantic analysis technology and the first query statement to obtain a second processing result, wherein the second processing result is whether the first processing result needs to be updated;
[0044] The recall unit is configured to use the pre-trained fourth model to perform recall processing based on the second processing result to obtain the target processing result. The fourth model is a similarity prediction model, which sorts and filters the second processing result to obtain the target processing result.
[0045] In a third aspect, the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of any method in the first aspect are implemented.
[0046] In a fourth aspect, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any method in the first aspect are implemented.
[0047] The present invention provides a query optimization method and system based on detection enhancement generation technology. Through intelligent document preprocessing technology, it can automatically identify and process documents of different formats to be processed, effectively solve the processing problems caused by document format differences, and improve the versatility and accuracy of data processing; secondly, through the block processing of the association calculation model, the document content is intelligently divided into logically coherent blocks, which improves the organizational efficiency of the data and makes the subsequent processing more efficient and accurate. Through the query preprocessing and optimization process, including cleaning, standard processing, word meaning expansion, synonym replacement and context expansion, the accuracy and coverage of the query statement are significantly improved, the problem of incomplete understanding of the user's query intention is solved, and the accuracy of the query is enhanced; then, through the deep processing of the language processing model, the document and the query statement are deeply semantically analyzed to extract key information, which further improves the relevance and accuracy of the retrieval results. Through review processing, the initial processing results are carefully reviewed using the word meaning expansion library, synonym database and semantic analysis technology to ensure the quality of the retrieval results and improve user satisfaction; finally, through the result recall and sorting optimization technology and similarity prediction model, the recall results are intelligently sorted, allowing users to quickly obtain the most relevant and valuable results, significantly improving the user experience. Through the implementation of the overall solution, not only the efficiency of document processing and retrieval is improved, but also the problems of low query accuracy and unreasonable result sorting in traditional retrieval systems are solved, thereby solving the problem of how to optimize the shortcomings of retrieval enhancement generation technology in the processing, segmentation, vector mapping and query preprocessing of text documents in the existing technology. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0049] Figure 1 is a flow chart of a query optimization method based on detection enhancement generation technology provided by an embodiment of the present invention;
[0050] Figure 2 is a schematic diagram of a second model structure provided by an embodiment of the present invention;
[0051] Figure 3 It is a structural schematic diagram of a query optimization system based on detection enhancement generation technology provided by an embodiment of the present invention;
[0052] Figure 4 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0053] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.
[0054] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0055] Figure 1 is a flowchart of a query optimization method based on detection enhancement generation technology provided by an embodiment of the present invention. Figure 1 As shown, the method includes:
[0056] Step S101, obtaining a first document to be processed, determining a document format type of the first document to be processed, and preprocessing the first document to be processed based on the document format type to obtain a second document to be processed.
[0057] In some embodiments, step S101 includes:
[0058] Step S1011, when the document format type of the first document to be processed is the PDF format type, optical character recognition technology is used to extract text from the first document to be processed to obtain extracted text, natural language processing technology is used to perform entity recognition and grammatical analysis on the extracted text to obtain the document structure, and based on the document structure, headers, footers and watermarks in the extracted text are identified and removed to obtain the second document to be processed.
[0059] Specifically, the present application performs specialized cleaning and preprocessing operations for different types of document formats to extract important metadata and optimize the structure of text content to ensure the integrity and accuracy of information. For advanced data cleaning and optimization processing, the module is designed to receive document inputs in multiple formats, such as PDF, Word documents, Excel tables, and HTML pages. Each format of document has a specially designed preprocessing process. When a PDF document is used as input, the module first runs a text extraction program that uses optical character recognition (OCR) technology to extract text from scanned PDF documents. Next, the module uses natural language processing (NLP) techniques, such as entity recognition and grammatical analysis, to determine the document structure. This allows the module to identify and remove non-core content such as headers, footers, and watermarks.
[0060] Step S1012, when the document format type of the first document to be processed is the Word format type, the first document to be processed is extracted using the application programming interface to obtain extracted text and document structure, and the header, footer and watermark in the extracted text are identified and removed based on the document structure to obtain the second document to be processed.
[0061] Specifically, for Word documents, we use the API provided by Microsoft Office to directly extract text and format information, and then apply NLP techniques similar to PDF processing to identify the semantic structure of the document and clean the data.
[0062] Step S1013, when the document format type of the first document to be processed is the Excel format type, the first document to be processed is analyzed to determine the table format, header row, data column and data type, and the data integrity of the first document to be processed is checked based on the table format, header row, data column and data type to obtain a check result, and the first document to be processed is corrected and marked based on the check result and predefined rules to obtain a second document to be processed.
[0063] Specifically, when processing Excel files, analyze table formats, identify header rows and data columns, and perform data type inferences. Check data integrity, such as looking for missing values or inconsistent data entries, and apply predefined rules to correct or flag these issues.
[0064] Step S1014, when the document format type of the first document to be processed is the HTML format type, the target web page data is captured based on the first document to be processed using the web crawler technology, the target web page data is analyzed using the analyzer to obtain the web page structure of the first document to be processed, and the advertisements and navigation menus of the first document to be processed are identified and removed based on the web page structure to obtain the second document to be processed.
[0065] Specifically, for web page content, the module first uses web crawler technology to grab HTML data from the website. Then, it runs a DOM analyzer to extract the structure of the web page, identify useful information such as article body and exclude irrelevant content such as advertisements or navigation menus.
[0066] Each step of the cleaning algorithm must not only identify and remove irrelevant content, but also preserve the original intent and content structure of the document to facilitate subsequent information processing and analysis. In addition, this module can configure the cleaning rule engine, allowing users to customize the cleaning logic to adapt to specific data processing needs.
[0067] Step S102, using the trained first model to process the second document to be processed in blocks to obtain a third document to be processed, wherein the first model is a relevance calculation model.
[0068] In some embodiments, step S102 includes:
[0069] Step S1021: Use the first model to extract vectors from each text segment in the second document to be processed to obtain a text segment embedding vector for each text segment.
[0070] Specifically, intelligent data segmentation refers to the use of pre-trained semantic understanding models to automatically identify and segment text data. The model is pre-trained with a corpus that may contain a wide range of topics and fields, allowing the model to learn and understand different semantic relationships and text structures. Through the self-attention mechanism, the model is able to establish an understanding of the connection and contextual relevance between text segments. When processing new text, the model evaluates the relevance between adjacent text segments.
[0071] Step S1022: Calculate similarity based on the text embedding vector of each text segment using the first model to obtain a set of inter-segment similarities of adjacent text segments.
[0072] Specifically, we can calculate the similarity between the text embedding vectors or use other methods to quantify the semantic coherence between them.
[0073] Step S1023: using the first model to perform block processing on the second document to be processed based on the inter-segment similarity set to obtain a third document to be processed.
[0074] Specifically, based on the assessed relevance, the model segments the text into independent chunks or sections. These chunks are determined based on their inherent semantic coherence, so that each chunk contains text segments with strong thematic or conceptual relevance.
[0075] Step S103, obtaining a first query statement input by a user, cleaning and performing standard processing on the first query statement, and obtaining a second query statement.
[0076] Specifically, a first query statement input by a user is received, and preliminary cleaning and standardization processing is performed, such as removing stop words, unifying upper and lower case, etc.
[0077] Step S104: acquiring historical query data of the user, and performing context expansion on the second query statement based on the historical query data to obtain a third query statement.
[0078] In some embodiments, step S104 includes:
[0079] Step S1041 , using natural language technology to perform part-of-speech tagging, dependency syntax analysis and keyword extraction on the second query statement to obtain query relationship data.
[0080] Specifically, natural language processing techniques such as part-of-speech tagging and dependency syntax analysis are used to parse the second query statement, extract keywords and phrases, and understand the relationship between them.
[0081] Among them, part-of-speech tagging is to assign a part-of-speech tag to each word in a sentence, such as noun, verb, adjective, etc., in order to understand the basic attributes and functions of each word in the query.
[0082] Dependency syntactic analysis is to determine the dependency relationship between words in a sentence, such as subject-predicate relationship, verb-object relationship, etc. Through dependency syntactic analysis, the grammatical structure of the query can be constructed, and the key information in the query and the logical relationship between them can be further understood.
[0083] Keyword extraction is the process of automatically identifying and extracting the most important or relevant words or phrases from a text. In the context of a query, keywords usually represent the user's query intent or focus. By extracting these keywords, the user's query needs can be understood more accurately.
[0084] Step S1042: Acquire the user's historical query data, and perform context expansion on the second query statement based on the historical query data and the query relationship data to obtain a third query statement.
[0085] Specifically, the context of the query statement is expanded by combining the user's historical search records, geographic location, time and other information to more comprehensively understand the query intent.
[0086] Obtain and store the user's historical query records, which contain the user's past search behavior, preferences, and interests. Based on the historical query data, you can understand the user's long-term needs and query habits.
[0087] The second query statement can be expanded using the user's historical query data, geographic location, time and other contextual information. For example, if the user often searches for content about "fitness" and the current query is "how to exercise at home", contextual information related to "home fitness" can be automatically added. Context expansion also includes semantic understanding of the query statement, that is, automatically adding or modifying keywords or phrases in the query based on the intent and context of the query to more accurately reflect the user's query needs. After context expansion, a third query statement is generated, which reflects the user's query intent more comprehensively and accurately than the second query statement. The third query statement can be used in search engines, database queries, or other places where it is necessary to understand the user's query intent to provide more accurate and relevant results.
[0088] Step S105, based on the third document to be processed, the first query statement, the third query statement, the pre-trained second model, and the pre-trained third model, obtain the first processing result, the second model is a deep learning model recurrent neural network, and the third model is a language model.
[0089] In some embodiments, step S105 includes:
[0090] S1051, input the first query statement and the third document to be processed into a pre-trained second model to generate a first embedding vector.
[0091] Specifically, the second model is a deep learning model recurrent neural network, the structure of which is as follows Figure 2 As shown in Figure 1, deep learning models, recurrent neural networks, are used to generate embedding vectors for documents and query statements. These models can capture the deep semantic features of document content and query intent. In the output layer, the softmax function is usually used to normalize the data, which limits the data to [0, 1] and the sum is 1.
[0092]
[0093] In the formula, the softmax function is usually used in the output layer of multi-classification problems, compressing a c-dimensional vector Sn containing arbitrary real numbers into another c-dimensional real vector Yn, so that the range of each element is between [0,1] and the sum of all elements is 1.
[0094] In the hidden layer, tanh is often used as the activation function. According to the properties of the recurrent neural network, we can analyze that:
[0095]
[0096] Where U and W are weight matrices that transform the input X into the feature space of the hidden layer. The dimensions of the weight matrix usually include the number of hidden layer units and the number of input features.
[0097] is the input of the RNN at the current time step. When processing sequence data, X is usually a representation of the current time step in the sequence;
[0098] It is a linear combination layer, which linearly transforms the input X through the weight matrix W and adds the bias term b;
[0099] Tanh() is an activation function that compresses the output of the linear combination layer into the (-1,1) interval.
[0100] According to the formula, we can know:
[0101]
[0102] S1052: Input the third query statement into the pre-trained second model to obtain a second embedding vector.
[0103] Specifically, a pre-trained language model, such as GPT-4, is called, and the result of the context analysis is used as input to generate an initial embedding vector. The extended context information is fused into the initial embedding vector, and the dimensions of the vector are adjusted to reflect the specific context of the query, thereby obtaining a second embedding vector.
[0104] Furthermore, the training process of the pre-trained language model in this application is divided into the following steps:
[0105] Data preparation: During the training phase, the model is trained using large-scale text data, which usually includes text data from multiple sources such as books, articles, web pages, etc. Data quality and diversity are ensured through cleaning and annotation.
[0106] Model architecture: GPT-4 is based on the Transformer architecture and captures long-distance dependencies through a multi-layer self-attention mechanism to generate language embeddings. The model generates deep contextual representations.
[0107] Pre-training process: During the pre-training process, the model learns the probability distribution of language through autoregression. Specifically, the model predicts the next word in the sequence given a context. This process uses a large number of training samples, performs multiple iterations, and optimizes the loss function, so that the model gradually grasps the structure and semantics of the language.
[0108] Fine-tuning process: When fine-tuning on a specific task, the model is further trained using data related to the target task to improve its performance in a specific context. The fine-tuning process usually includes adjusting the model's hyperparameters to ensure its effectiveness in a specific scenario.
[0109] S1053: Optimize the second embedding vector using a gradient descent algorithm.
[0110] Specifically, optimization algorithms such as gradient descent are used to fine-tune the embedding vector to ensure that the embedding vector can best represent the query intent.
[0111] S1054: Obtain a first processing result based on the first embedding vector and the second embedding vector.
[0112] Step S106, using a preset word meaning expansion library, a preset synonym database, semantic analysis technology and the first query statement, the first processing result is reviewed to obtain a second processing result.
[0113] In some embodiments, step S106 includes:
[0114] S1061: Use a preset word meaning expansion library to perform keyword extraction and keyword transformation processing on the first query statement to obtain a first review result.
[0115] Specifically, a word sense expansion library is used to identify keywords in a query and find their various variations or related expressions, thereby capturing multiple meanings that may be implied in the query.
[0116] S1062: Perform keyword replacement processing on the first query statement using a preset synonym database to obtain a second review result.
[0117] Specifically, the synonym database enables the module to identify and use words with the same or similar meanings as the keywords in the query, which helps to understand and match the user's true intention when processing natural language queries.
[0118] S1063, using a preset semantic analysis technology to perform semantic analysis on the first query statement to obtain a third review result.
[0119] Specifically, the module performs in-depth semantic analysis of user input to understand the underlying needs behind the query. This may involve natural language understanding techniques such as sentiment analysis, semantic role labeling, or intent recognition.
[0120] S1064, review the first processing result based on the first review result, the second review result and the third review result to obtain a second processing result.
[0121] Specifically, in order to enhance the expressiveness and accuracy of the query, the first processing result is reviewed.
[0122] Step S107, using the pre-trained fourth model to perform recall processing based on the second processing result to obtain a target processing result.
[0123] Specifically, the pre-trained fourth model is used to perform recall processing based on the second processing result to obtain the first recall result, the pre-trained fourth model is used to perform similarity prediction processing on the first recall result to obtain the similarity prediction result, the first recall result is sorted based on the similarity prediction result to obtain the second recall result, the fourth model is a similarity prediction model, and the second recall result is used as the target processing result. RankNet (a machine learning model for learning ranking) is used for result recall and sorting optimization, and the relevance of the results is predicted by training the model to optimize the sorting order. RankNet uses a neural network to score documents. Representation sample The score, if > , then it means Ranked higher than (use -> The sample can be calculated using the function f Compare samples Probability of ranking higher.
[0124]
[0125]
[0126] In the formula, It is a sample Ranked higher than sample The probability of yes A monotonically increasing function of , in RankNet, usually = , that is, directly use As input, yes and A function of the difference in scores between , usually defined as:
[0127]
[0128] Will Substituting into the probability function, we get:
[0129]
[0130] This formula is used to calculate the sample Relative to the sample The probability of ranking higher. > hour, is positive, is greater than 1, so Greater than 0.5, indicating Ranked higher than The probability of is greater; on the contrary, when < hour, is negative, is less than 1, so Less than 0.5, indicating Ranked higher than The probability is small.
[0131] In some optional implementations, the method further includes: monitoring in real time whether the query statement has changed, and if a change in the query statement is detected, recalculating the initial embedding vector.
[0132] Specifically, a monitoring system is established to monitor the updates of the document library and the changes of query statements in real time. Once a change is detected, the recalculation of the embedding vector is triggered immediately. Ensure that the update mechanism can respond at the millisecond level to support real-time search requirements.
[0133] Furthermore, the fourth model is a similarity prediction model, and its training process includes the following steps:
[0134] Data preparation: First, collect data related to the recall task, including query and candidate item pairs; second, clean and preprocess the data, generate similarity labels (such as scores between 0 and 1), and divide the dataset into training set, validation set, and test set.
[0135] Model definition: Choose a suitable model architecture, such as Siamese network, to capture the similarity between input pairs. In the process, define a loss function suitable for similarity learning, such as contrastive loss and optimizer, such as Adam.
[0136] Training process: In the training cycle, the training set is used for forward propagation, the loss is calculated, and the model parameters are updated through backpropagation. At the same time, the model is evaluated on the validation set to monitor the performance of the model and perform hyperparameter tuning.
[0137] Model evaluation: Evaluate model performance on the test set, calculate the accuracy of similarity prediction and other relevant metrics, and visualize learning curves to analyze model performance.
[0138] Model deployment and application: Save the trained model for subsequent use.
[0139] The fourth model is used to predict the similarity of the first recall result, and the prediction result is used for subsequent sorting optimization.
[0140] In some optional implementations, user feedback on search results is collected, such as click behavior, dwell time, conversion rate, etc. These data are analyzed to evaluate the accuracy of search results and user satisfaction. The embedding vector is adjusted based on the feedback results to improve future search results.
[0141] In some optional implementations, the distributed storage and computing framework may be implemented using existing cloud computing platforms, for example, using Hadoop and Spark for big data processing, and using Elasticsearch for efficient full-text search and analysis.
[0142] Compared with the prior art, the embodiments of the present disclosure have the following beneficial effects: first, through the intelligent document preprocessing technology, it is possible to automatically identify and process documents of different formats to be processed, effectively solving the processing difficulties caused by differences in document formats and improving the versatility and accuracy of data processing. Secondly, through the block processing of the relevance calculation model, the document content is intelligently divided into logically coherent blocks, which improves the efficiency of data organization and makes subsequent processing more efficient and accurate. Through the query preprocessing and optimization process, including cleaning, standard processing, word meaning expansion, synonym replacement and context expansion, the accuracy and coverage of the query statement are significantly improved, the problem of incomplete understanding of the user's query intention is solved, and the accuracy of the query is enhanced. Then, through the deep processing of the language processing model, the document and the query statement are deeply semantically analyzed, and key information is extracted, which further improves the relevance and accuracy of the retrieval results. Through the review process, the word meaning expansion library, synonym database and semantic analysis technology are used to conduct a detailed review of the preliminary processing results, ensuring the quality of the retrieval results and improving user satisfaction. Finally, through the result recall and sorting optimization technology and similarity prediction model, the recall results are intelligently sorted, so that users can quickly obtain the most relevant and valuable results, significantly improving the user experience. Through the implementation of the overall solution, not only the efficiency of document processing and retrieval is improved, but also the problems of low query accuracy and unreasonable result sorting in traditional retrieval systems are solved. The method provided by the present disclosure can solve the problem of how to optimize the shortcomings of the retrieval enhancement generation technology in the processing, segmentation, vector mapping and query preprocessing of text documents in the prior art.
[0143] All optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, and will not be described in detail here.
[0144] The following are embodiments of the device disclosed herein, which can be used to execute the method embodiments disclosed herein. For details not disclosed in the device embodiments disclosed herein, please refer to the method embodiments disclosed herein.
[0145] Figure 3 is a structural diagram of a query optimization system based on detection enhancement generation technology provided by an embodiment of the present invention, such as Figure 3 As shown, the system includes: a pre-processing unit 301, a blocking unit 302, a cleaning unit 303, an expansion unit 304, a processing unit 305, a review unit 306 and a recall unit 307.
[0146] Among them, the preprocessing unit 301 is configured to obtain a first document to be processed; and determine the document format type of the first document to be processed, and preprocess the first document to be processed based on the document preprocessing method corresponding to the document format type to obtain a second document to be processed; the block unit 302 is configured to use the trained first model to block the second document to be processed to obtain a third document to be processed, and the first model is a relevance calculation model; the cleaning unit 303 is configured to obtain a first query statement input by a user, and clean and standardize the first query statement to obtain a second query statement; the expansion unit 304 is configured to obtain the user's historical query data, and contextually expand the second query statement based on the context expansion of the historical query data to obtain a third query statement; the processing unit 305 is configured to perform block processing on the second document to be processed based on the third query statement. Process the document, the first query statement, the third query statement, the pre-trained second model, and the pre-trained third model to obtain a first processing result, wherein the second model generates embedding vectors of the document and the query statement based on a deep learning model recurrent neural network, and the third model generates embedding vectors of the query statement based on a language model; the review unit 306 is configured to review the first processing result using a preset word meaning expansion library, a preset synonym database, semantic analysis technology, and the first query statement to obtain a second processing result, wherein the second processing result is whether the first processing result needs to be updated; the recall unit 307 is configured to perform recall processing based on the second processing result using a pre-trained fourth model to obtain a target processing result, wherein the fourth model is a similarity prediction model, and the second processing result is sorted and screened to obtain a target processing result.
[0147] In a possible embodiment, the preprocessing unit 301 is further configured to: when the document format type of the first document to be processed is a PDF format type, use optical character recognition technology to extract text from the first document to be processed to obtain extracted text, use natural language processing technology to perform entity recognition and grammatical analysis on the extracted text to obtain a document structure, and identify and remove headers, footers and watermarks in the extracted text based on the document structure to obtain a second document to be processed; when the document format type of the first document to be processed is a Word format type, use an application programming interface to extract the first document to be processed to obtain extracted text and document structure, and identify and remove headers, footers and watermarks in the extracted text based on the document structure to obtain a second document to be processed; when the first document to be processed is a Word format type, use an application programming interface to extract the first document to be processed to obtain extracted text and document structure, and identify and remove headers, footers and watermarks in the extracted text based on the document structure to obtain a second document to be processed; When the document format type of the document is an Excel format type, the first document to be processed is analyzed to determine the table format, title row, data column and data type, and the data integrity of the first document to be processed is checked based on the table format, title row, data column and data type to obtain a check result, and the first document to be processed is corrected and marked based on the check result and predefined rules to obtain a second document to be processed; when the document format type of the first document to be processed is an HTML format type, the target web page data is captured based on the first document to be processed using a web crawler technology, and the target web page data is analyzed using an analyzer to obtain the web page structure of the first document to be processed, and the advertisements and navigation menus of the first document to be processed are identified and removed based on the web page structure to obtain the second document to be processed.
[0148] In a possible embodiment, the blocking unit 302 of the system is further configured to: use the first model to extract vectors for each paragraph in the second document to be processed to obtain a paragraph embedding vector for each paragraph; use the first model to calculate similarity based on the paragraph embedding vector of each paragraph to obtain a set of inter-segment similarities of adjacent paragraphs; use the first model to block the second document to be processed based on the inter-segment similarity set to obtain a third document to be processed.
[0149] In a possible embodiment, the expansion unit 304 of the system is further configured to: use natural language technology to perform part-of-speech tagging, dependency syntax analysis and keyword extraction on the second query statement to obtain query relationship data; obtain the user's historical query data, and perform context expansion on the second query statement based on the historical query data and query relationship data to obtain a third query statement.
[0150] In a possible embodiment, the processing unit 305 of the system is further configured to: input the first query statement and the third document to be processed into a pre-trained second model to generate a first embedding vector; input the third query statement into the pre-trained second model to obtain a second embedding vector; optimize the second embedding vector using a gradient descent algorithm; and obtain a first processing result based on the first embedding vector and the second embedding vector.
[0151] In a possible embodiment, the review unit 306 of the system is further configured to: use a preset word meaning expansion library to perform keyword extraction and keyword deformation processing on the first query statement to obtain a first review result; use a preset synonym database to perform keyword replacement processing on the first query statement to obtain a second review result; use a preset semantic analysis technology to perform semantic analysis processing on the first query statement to obtain a third review result; and review the first processing result based on the first review result, the second review result and the third review result to obtain a second processing result.
[0152] In a possible embodiment, the system is further configured to: monitor in real time whether the query statement has changed, and if a change in the query statement is detected, recalculate the initial embedding vector.
[0153] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, Figure 4 The server shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0154] like Figure 4 As shown, the electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 are also stored. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.
[0155] Typically, the following devices may be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 408 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 409. The communication device 409 may allow the electronic device 400 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The electronic device 400 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 4 Each block shown in the figure may represent one device, or may represent multiple devices as required.
[0156] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 409, or installed from the storage device 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the functions defined in the method of some embodiments of the present disclosure are executed.
[0157] It should be noted that the computer-readable medium of some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.
[0158] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.
[0159] The computer-readable medium may be included in the device; or may exist separately without being incorporated into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains a first document to be processed, determines the document format type of the first document to be processed, and pre-processes the first document to be processed based on the document format type to obtain a second document to be processed; uses a trained first model to block the second document to be processed to obtain a third document to be processed, and the first model is a relevance calculation model; obtains a first query statement input by a user, cleans and standardizes the first query statement to obtain a second query statement; obtains the user's historical query data, and contextually expands the second query statement based on the historical query data to obtain a third query statement; obtains a first processing result based on the third document to be processed, the first query statement, the third query statement, the pre-trained second model, and the pre-trained third model, and the second model is a deep learning model recurrent neural network, and the third model is a language model; uses a preset word meaning expansion library, a preset synonym database, semantic analysis technology and the first query statement to review the first processing result to obtain a second processing result; uses a pre-trained fourth model to recall the second processing result to obtain a target processing result.
[0160] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0161] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0162] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be arranged in a processor, for example, may be described as: a processor including a preprocessing unit, a blocking unit, a cleaning unit, an expansion unit, a processing unit, a review unit and a recall unit. Among them, the names of these units do not constitute a limitation on the unit itself under certain circumstances. For example, the preprocessing unit may also be described as "obtaining a first document to be processed, determining a document format type of the first document to be processed, preprocessing the first document to be processed based on the document format type, and obtaining a second document to be processed".
[0163] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.
[0164] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of technical features, but should also cover other technical solutions formed by any combination of technical features or their equivalent features without departing from the inventive concept. For example, a technical solution formed by replacing a feature with a technical feature with similar functions disclosed in the embodiments of the present disclosure (but not limited to).
[0165] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.
[0166] In the description of the present invention, it should be noted that the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.
[0167] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A query optimization method based on detection enhancement generation technology, characterized in that: include: Get the first document to be processed; Determining a document format type of the first document to be processed, and preprocessing the first document to be processed based on a document preprocessing method corresponding to the document format type to obtain a second document to be processed; Using the trained first model to process the second document to be processed in blocks to obtain a third document to be processed, wherein the first model is a correlation calculation model; Obtaining a first query statement input by a user, cleaning and performing standard processing on the first query statement to obtain a second query statement; Acquire historical query data of the user, and perform context expansion on the second query statement based on context expansion of the historical query data to obtain a third query statement; A first processing result is obtained based on the third document to be processed, the first query statement, the third query statement, a pre-trained second model, and a pre-trained third model, wherein the second model generates embedding vectors of documents and query statements based on a deep learning model recurrent neural network, and the third model generates embedding vectors of query statements based on a language model; Using a preset word meaning expansion library, a preset synonym database, semantic analysis technology and a first query statement, the first processing result is reviewed to obtain a second processing result, wherein the second processing result is whether the first processing result needs to be updated; A pre-trained fourth model is used to perform recall processing based on the second processing result to obtain a target processing result. The fourth model is a similarity prediction model. The second processing result is sorted and screened to obtain a target processing result.
2. The method according to claim 1, characterized in that: The preprocessing of the first document to be processed based on the document format type to obtain a second document to be processed includes: When the document format type of the first document to be processed is a PDF format type, using optical character recognition technology to extract text from the first document to be processed to obtain extracted text, using natural language processing technology to perform entity recognition and syntax analysis on the extracted text to obtain a document structure, and based on the document structure, identifying and removing headers, footers and watermarks in the extracted text to obtain a second document to be processed; When the document format type of the first document to be processed is a Word format type, extracting the first document to be processed using an application programming interface to obtain extracted text and a document structure, and identifying and removing a header, a footer, and a watermark in the extracted text based on the document structure to obtain a second document to be processed; When the document format type of the first document to be processed is an Excel format type, analyzing the first document to be processed to determine the table format, the header row, the data columns and the data types, verifying the data integrity of the first document to be processed based on the table format, the header row, the data columns and the data types to obtain a verification result, and correcting and marking the first document to be processed based on the verification result and a predefined rule to obtain a second document to be processed; When the document format type of the first document to be processed is the HTML format type, the web crawler technology is used to capture the target web page data based on the first document to be processed, and the target web page data is analyzed by an analyzer to obtain the web page structure of the first document to be processed, and the advertisements and navigation menus of the first document to be processed are identified and removed based on the web page structure to obtain the second document to be processed.
3. The method according to claim 1, characterized in that The first model is obtained by training a corpus; the second document to be processed is processed in blocks using the trained first model to obtain a third document to be processed, including: Using the first model to extract vectors from each paragraph in the second to-be-processed document, to obtain a paragraph embedding vector for each paragraph; Using the first model to perform similarity calculation based on the paragraph embedding vectors of each paragraph, to obtain a set of inter-segment similarities of adjacent paragraphs; The second document to be processed is processed in blocks based on the inter-segment similarity set using the first model to obtain a third document to be processed.
4. The method according to claim 1, characterized in that: Performing context expansion on the second query statement based on the historical query data to obtain a third query statement includes: Using natural language technology to perform part-of-speech tagging, dependency syntax analysis, and keyword extraction on the second query sentence to obtain query relationship data; The user's historical query data is obtained, and the second query statement is context-expanded based on the historical query data and the query relationship data to obtain a third query statement.
5. The method according to claim 4, characterized in that The first processing result is obtained based on the third document to be processed, the first query statement, the third query statement, the pre-trained second model, and the pre-trained third model, wherein the second model is a deep learning model recurrent neural network, and the third model is a language model, including: Inputting the first query statement and the third to-be-processed document into a pre-trained second model to generate a first embedding vector; Inputting the third query statement into a pre-trained second model to obtain a second embedding vector; Optimizing the second embedding vector using an optimization algorithm; A first processing result is obtained based on the first embedding vector and the second embedding vector.
6. The method according to claim 5, characterized in that The method of using a preset word meaning expansion library, a preset synonym database, a semantic analysis technology and a first query statement to review the first processing result to obtain a second processing result includes: Using a preset word meaning expansion library to perform keyword extraction and keyword deformation processing on the first query statement to obtain a first review result; Using a preset synonym database to perform keyword replacement processing on the first query statement to obtain a second review result; Perform semantic analysis on the first query statement using a preset semantic analysis technology to obtain a third review result; The first processing result is reviewed based on the first review result, the second review result and the third review result to obtain a second processing result.
7. The method according to claim 5, characterized in that The method further includes: monitoring in real time whether the query statement changes, and if it is detected that the query statement changes, recalculating the first embedding vector.
8. A query optimization system based on detection enhancement generation technology, characterized in that: include: A pre-processing unit, configured to obtain a first document to be processed; and determining a document format type of the first document to be processed, and preprocessing the first document to be processed based on a document preprocessing method corresponding to the document format type to obtain a second document to be processed; a block division unit configured to perform block processing on the second to-be-processed document using a trained first model to obtain a third to-be-processed document, wherein the first model is a relevance calculation model; a cleaning unit configured to obtain a first query statement input by a user, and perform cleaning and standard processing on the first query statement to obtain a second query statement; an expansion unit configured to obtain historical query data of the user, and perform context expansion on the second query statement based on context expansion of the historical query data to obtain a third query statement; A processing unit is configured to obtain a first processing result based on the third document to be processed, the first query statement, the third query statement, a pre-trained second model, and a pre-trained third model, wherein the second model generates embedding vectors of documents and query statements based on a deep learning model recurrent neural network, and the third model generates embedding vectors of query statements based on a language model; A review unit is configured to review the first processing result by using a preset word meaning expansion library, a preset synonym database, a semantic analysis technology and a first query statement to obtain a second processing result, wherein the second processing result is whether the first processing result needs to be updated; The recall unit is configured to use a pre-trained fourth model to perform recall processing based on the second processing result to obtain a target processing result. The fourth model is a similarity prediction model, which sorts and filters the second processing result to obtain the target processing result.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
