Knowledge retrieval method, system, device and product based on visual language large model

Abstract

The application discloses a knowledge retrieval method, system, device and product based on a visual language large model, belongs to the technical field of artificial intelligence, and comprises the following steps: constructing a knowledge base according to an uneditable document and professional field information, obtaining a query query statement, performing entity extraction and matching by using a large language model, obtaining matching information, performing three-way recall on the matching information based on the knowledge base by using a three-level progressive filtering mechanism, fusing and scoring three-way recall results by using a third-party reordering model, obtaining a comprehensive score according to the scoring results and corresponding weights, and obtaining a query result set according to the comprehensive score. The application uses a large language model to improve the quality of generated results, uses a database to improve the traceability of generated results, uses a three-level progressive filtering mechanism to perform three-way recall operation, so that the recall results are highly relevant, the recall coverage is improved, and meanwhile, the third-party reordering model is used to improve the matching degree of the results and user demand.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence, and particularly relates to a knowledge retrieval method, system, device and product based on a visual language large model. BACKGROUND

[0002] With the development of artificial intelligence technology, natural language processing and large models are widely used in various fields. RAG (Retrieval-augmented Generation) is a generation enhancement framework widely used in the field of natural language processing and large models. The basic idea is to use a retrieval module to retrieve relevant documents or text blocks in an external knowledge base before the generation model generates an answer, and input the retrieval results and user input content into the generation model to improve the relevance, knowledge coverage, accuracy and timeliness of the output content.

[0003] The retrieval methods commonly used in existing RAG systems mainly include: (1) vector retrieval based on dense vectors: using pre-training models such as BERT (Bidirectional Encoder Representations from Transformers), DPR (Dense Passage Retrieval) and ColBERT (Contextualized Late Interaction over BERT) to encode text into dense vectors, and using vector similarity to achieve efficient retrieval, which is suitable for capturing deep semantic similarity; (2) symbolic retrieval based on sparse representation: realizing traditional keyword-level matching by calculating the TF-IDF weight of keywords, which is suitable for specific words with high accuracy; (3) hybrid retrieval combining dense and sparse representations: combining the above two methods, improving recall effect through linear weighting and intersection filtering, and taking into account semantic relevance and keyword hit rate; (4) multi-round retrieval: for complex question and answer or reasoning scenarios, multiple information chains are retrieved step by step, so that the generation model has stronger reasoning ability.

[0004] However, the above methods still have corresponding shortcomings, such as: (1) strong limitations and low generation quality accuracy: it is difficult to cover structured or encoded information. Dense vectors are mainly achieved through semantic representation learning. They are not good at recognizing structured or encoded keywords and are prone to missed or false recalls. Moreover, existing retrieval methods often use the retrieval granularity of "paragraph" or "document", which leads to low recall accuracy, redundant information, and affects the focus and relevance of generated content; (2) lack of source tracing ability: existing RAG systems do not retain the original location information of the recalled text, making it difficult to trace the source of the answer, and the credibility and verifiability are low; (3) chaotic context order and lack of adaptive sorting mechanism: in existing technologies, the retrieved text blocks are often rearranged or spliced ​​before being sent to the generation model. There is a lack of effective sorting strategy, which easily leads to disordered context and poor logical coherence of language generation. Furthermore, using fixed rules or manually setting weights to integrate different recall results makes it impossible to automatically learn the optimal sorting strategy based on actual business data, which affects the quality of recall results and sorting effect.

[0005] Therefore, how to provide an effective technical solution to address the problems of strong limitations, low generation quality and accuracy, lack of traceability, chaotic context order, and lack of adaptive sorting mechanism in existing technologies has become an urgent problem to be solved in existing technologies. Summary of the Invention

[0006] The purpose of this invention is to provide a knowledge retrieval method, system, device, and product based on a large visual language model, in order to solve the above-mentioned problems existing in the prior art.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a knowledge retrieval method based on a large visual language model, comprising:

[0009] The process involves acquiring uneditable documents and professional domain information, performing layout analysis on the uneditable documents using layout analysis technology, obtaining layout analysis results, restoring the order of the layout analysis results based on a preset reading order, obtaining document restoration results, segmenting the document restoration results to obtain text blocks, using a visual language model to perform semantic optimization on the text blocks to obtain title information and content information, and constructing a knowledge base based on text blocks, title information, content information, and professional domain information.

[0010] The system obtains the user's query statement, extracts entities from the query statement using a large language model, normalizes the entity information to obtain standard entity information, and matches the standard entity information with a preset vehicle model database to obtain matching information.

[0011] Based on the knowledge base, a three-level progressive filtering mechanism is used to perform three-way recall operations on the query statement, standard entity information and matching information to obtain three-way recall results. The three-way recall results are then fused to obtain a fused result. Based on the relevance score in the fused result, the relevance score set is extracted.

[0012] Based on the relevance score set, a third-party re-ranking model is used to process the query statement to obtain a one-dimensional relevance score. The relevance score set and the one-dimensional relevance score are merged to obtain a total relevance score set. Based on the logistic regression model, the total relevance score set is linearly weighted using pre-learned feature weights to obtain a comprehensive score. The relevance score set is then sorted based on the comprehensive score to obtain the query result set.

[0013] In one possible design, the three-way recall results include a first-way recall result, a second-way recall result, and a third-way recall result; the three-way recall operation based on the knowledge base and using a three-level progressive filtering mechanism on the query statement, standard entity information, and matching information yields the following three-way recall results:

[0014] Input the query statement into the vector model to obtain a dense vector of the query statement;

[0015] Based on the knowledge base, the KNN algorithm and a three-level progressive filtering mechanism are used to calculate the similarity of dense vectors of query statements to obtain the first similarity set. The first similarity set is then filtered to obtain the first preliminary candidate set.

[0016] The BM25 algorithm is used to perform a relevance weighting calculation on the query statement and the first preliminary candidate set to obtain the first total relevance score;

[0017] The first preliminary candidate set is scored based on the first total relevance score to obtain the first scoring result. The first preliminary candidate set is then ranked based on the first scoring result to obtain the first ranking result.

[0018] Samples are selected from the first ranking results to obtain the first set of highly relevant results.

[0019] In one possible design, based on a knowledge base, a three-level progressive filtering mechanism is used to perform a three-way recall operation on the query statement, standard entity information, and matching information, resulting in three recall results, which also include:

[0020] Based on a pre-defined professional data dictionary, the query statement is encoded to obtain a sparse vector of the query statement;

[0021] Based on the knowledge base, the KNN algorithm and a three-level progressive filtering mechanism are used to calculate the similarity of the sparse vectors of the query statement to obtain a second similarity set. The second similarity set is then filtered to obtain a second preliminary candidate set.

[0022] The BM25 algorithm is used to perform relevance weighting calculation on the query statement and the second preliminary candidate set to obtain the second total relevance score;

[0023] The second preliminary candidate set is scored based on the second total relevance score to obtain the second scoring result. The second preliminary candidate set is then ranked based on the second scoring result to obtain the second ranking result.

[0024] Sample selection is performed on the second ranking results to obtain the second set of highly relevant results.

[0025] In one possible design, based on a knowledge base, a three-level progressive filtering mechanism is used to perform a three-way recall operation on the query statement, standard entity information, and matching information, resulting in three recall results, which also include:

[0026] The text block is processed to obtain the fault code. Based on the knowledge base and the fault code, a third preliminary candidate set is obtained.

[0027] The BM25 algorithm and a three-level progressive filtering mechanism are used to perform relevance weighting calculation on the query statement and the third preliminary candidate set to obtain the third total relevance score;

[0028] The third preliminary candidate set is scored based on the third total relevance score to obtain the third scoring result. The third preliminary candidate set is then ranked based on the third scoring result to obtain the third ranking result.

[0029] Sample selection is performed on the third ranking results to obtain the third set of highly relevant results.

[0030] In one possible design, the total relevance score set includes a first relevance score, a second relevance score, a third relevance score, and a one-dimensional relevance score; the calculation expression for the comprehensive score is:

[0031] ;

[0032] In the formula, For the overall score, The weight of the first relevance score, The first relevance score, The weight of the second relevance score, The second relevance score, The weight of the third relevance score, The third relevance score, The weights for the one-dimensional relevance score. This represents the one-dimensional correlation score.

[0033] In one possible design, the entity information includes brand field information, manufacturer field information, vehicle series field information, model year field information, and vehicle type field information, and the matching information includes power type field information.

[0034] In one possible design, the three-level progressive filtering mechanism includes a first-level filtering condition, a second-level filtering condition, and a third-level filtering condition; the three-level progressive filtering mechanism is specifically as follows:

[0035] The first level of filtering requires a complete match of the brand field, manufacturer field, vehicle series field, model year field, vehicle type field, and powertrain type field.

[0036] The second-level filtering criteria are: brand field information, manufacturer field information, vehicle series field information, model year field information, and power type field information are completely matched, while vehicle model field information is fuzzy matched.

[0037] The third-level filtering conditions are: brand field information, manufacturer field information, vehicle series field information, and power type field information are completely matched, and model field information is fuzzy matched to obtain matching results. The matching results are sorted by relevance based on the year field information to obtain sorted results. The sorted results are then selected based on preset selection conditions.

[0038] Secondly, the present invention provides a knowledge retrieval system based on a large visual language model, comprising:

[0039] The knowledge base construction module is used to acquire non-editable documents and professional domain information. It uses layout analysis technology to perform layout analysis on the non-editable documents to obtain layout analysis results. Based on a preset reading order, it restores the order of the layout analysis results to obtain document restoration results. The document restoration results are then segmented to obtain text blocks. A visual language model is used to perform semantic optimization processing on the text blocks to obtain title information and content information. Based on the text blocks, title information, content information, and professional domain information, a knowledge base is constructed.

[0040] The data matching module is used to obtain the user's query statement, extract entities from the query statement using a large language model to obtain entity information, normalize the entity information to obtain standard entity information, and match the standard entity information with a preset vehicle model database to obtain matching information.

[0041] The recall fusion module is used to perform three-way recall operations on query statements, standard entity information and matching information based on the knowledge base and using a three-level progressive filtering mechanism to obtain three-way recall results. The three-way recall results are then fused to obtain a fused result. Based on the relevance score in the fused result, the relevance score set is extracted.

[0042] The result generation module is used to process the query statement based on the relevance score set using a third-party re-ranking model to obtain a one-dimensional relevance score. The relevance score set and the one-dimensional relevance score are merged to obtain a total relevance score set. Based on the logistic regression model, the total relevance score set is linearly weighted using pre-learned feature weights to obtain a comprehensive score. The relevance score set is then sorted based on the comprehensive score to obtain the query result set.

[0043] Thirdly, the present invention provides a computer device comprising a memory, a processor, and a transceiver connected in sequence and communication, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the knowledge retrieval method as described above.

[0044] Fourthly, the present invention provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform the knowledge retrieval method as described above.

[0045] The beneficial effects of this invention are as follows:

[0046] (1) This invention discloses a knowledge retrieval method, system, device and product based on a large visual language model. By using a large language model to process query statements, it can handle a large number of rich data types, including structured data, serialized data and encoded data, effectively improving robustness and generalization ability, while improving the quality of generated results.

[0047] (2) By constructing a knowledge base and saving the data generated during the query result set generation process into the knowledge base, this invention can achieve accurate text tracing of the recall results, which facilitates subsequent original text location and content tracing, enhances interpretability and verifiability, and improves the credibility of the results.

[0048] (3) This invention performs layout analysis and order restoration on the document, restores the text order of the document according to the preset reading order, avoids text confusion, and adopts a sorting weight model, which can automatically learn the weight distribution of various relevance scores in the actual sorting, and then calculate the comprehensive score, effectively improving the matching degree between the sorting results and user needs, improving the intelligent effect, and improving the readability and accuracy of the generated results. Attached Figure Description

[0049] Figure 1 A flowchart of a knowledge retrieval method provided in the first aspect of an embodiment of the present invention;

[0050] Figure 2 This is a block diagram of a knowledge retrieval system provided in the second aspect of an embodiment of the present invention. Detailed Implementation

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in conjunction with the accompanying drawings and descriptions of the embodiments or the prior art. Obviously, the following description of the structure of the accompanying drawings is only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0052] It should be understood that although the terms first, second, etc., may be used herein to describe various units, these units should not be limited by these terms. These terms are only used to distinguish one unit from another. For example, a first unit may be referred to as a second unit, and similarly, a second unit may be referred to as a first unit, without departing from the scope of the exemplary embodiments of the invention.

[0053] It should be understood that the term "and / or" that may appear in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, B exists alone, and A and B exist simultaneously. The term " / and" that may appear in this document describes another relationship between related objects, indicating that two relationships can exist. For example, A / and B can mean: A exists alone, and A and B exist alone. In addition, the character " / " that may appear in this document generally indicates that the related objects before and after it are in an "or" relationship.

[0054] Example:

[0055] like Figure 1 As shown, the first aspect of this embodiment provides a knowledge retrieval method based on a large visual language model, which can be executed, but is not limited to, by a computer device or virtual machine with certain computing resources, such as a personal computer or smartphone, or by a virtual machine; the knowledge retrieval method includes, but is not limited to, the following steps:

[0056] S1. Obtain uneditable documents and professional domain information, use layout analysis technology to perform layout analysis on the uneditable documents, obtain layout analysis results, restore the order of the layout analysis results based on the preset reading order, obtain document restoration results, segment the document restoration results to obtain text blocks, use a visual language model to perform semantic optimization processing on the text blocks to obtain title information and content information, and construct a knowledge base based on text blocks, title information, content information and professional domain information;

[0057] It should be noted that uneditable documents here include, but are not limited to, PDF documents; layout analysis technology is based on the combination of computer vision and natural language processing (NLP). Its principle is to identify and understand the document through computer vision and NLP technologies. When performing layout analysis on uneditable documents, different elements are labeled and then identified and replaced. For example, titles, tables, text, formulas, and images are labeled separately, and OCR (Optical Character Recognition) technology is used to extract the text content of the labels and text; OCR technology is used to extract tables to obtain Markdown formatted table content; OCR technology is used to recognize formulas to obtain formula text; and images are saved to Object Storage Service (OSS) and image links are generated to replace the images.

[0058] In this embodiment, the preset reading order includes reading from top to bottom and from left to right, which can be set according to the actual situation and are not limited here; the purpose of cutting the text into blocks is to put a knowledge description in a complete text block to ensure that the content retrieved later is complete and facilitates subsequent data processing. The specific process includes: using the title as an identifier, if the current position is identified as a title, then starting from the current position until the next position identified as a title ends, this area is a text block.

[0059] Furthermore, the Vision-Language Model (VLM) is a multimodal artificial intelligence model that combines computer vision and natural language processing. Its principle is to enable semantic alignment and reasoning across different modalities by jointly learning representations of images and text. The knowledge base is stored in Elasticsearch (ES), a distributed search engine based on Lucene. The knowledge base includes, but is not limited to, fields such as `diagnosis_code`, `vlm_title`, `vlm_text`, `synonyms`, `mineru_title`, `filename`, and `pdf_path`. The `diagnosis_code` field represents the fault code, the `vlm_title` field represents the semantically optimized title information, and the `vlm_text` field represents the semantically optimized title information. The fields are semantically optimized content information, the synonyms field is synonyms of entity information, the mineru_title field is the title information in the layout analysis results, the filename field is the name of the document corresponding to the text block, and the pdf_path field is the link of the document corresponding to the text block on OSS. The text block is extracted based on a preset fault code dictionary, regular expressions, and a large language model to obtain the fault codes. The optimized content information is then subjected to sparse vectorization processing. Specifically, a professional domain dictionary built based on business accumulation is used to perform one-hot encoding on the optimized content information. This method has good results in scenarios that focus on keyword recall, and is especially suitable for the accurate recall of key entity information such as fault codes and corresponding fault parts. It can improve the recognition ability and coverage of specific semantic units in the retrieval process.

[0060] Thus, by introducing a visual language model to semantically optimize text blocks, missing subjects or other key semantic information in the title can be supplemented, improving the completeness and accuracy of the title information. Simultaneously, the content information can be linguistically polished, descriptive logic optimized, and word order confusion avoided due to errors in text sequence restoration, thereby enhancing overall semantic coherence and readability.

[0061] S2. Obtain the user's query statement, use a large language model to extract entities from the query statement to obtain entity information, normalize the entity information to obtain standard entity information, and match the standard entity information with the preset vehicle model database to obtain matching information;

[0062] The entity information includes brand field information, manufacturer field information, vehicle series field information, model year field information, and vehicle type field information; the matching information includes power type field information; the large language model here is DeepSeek-R1, which is not limited in this embodiment of the invention.

[0063] S3. Based on the knowledge base, a three-level progressive filtering mechanism is used to perform three-way recall operations on the query statement, standard entity information and matching information to obtain three-way recall results. The three-way recall results are then fused to obtain a fused result. Based on the relevance score in the fused result, the fused result is extracted to obtain a relevance score set.

[0064] The three recall results include the first recall result, the second recall result, and the third recall result.

[0065] It should be noted that the three-level progressive filtering mechanism is as follows: The first-level filtering condition is a complete match of the brand, manufacturer, model series, year, vehicle type, and powertrain information fields; the second-level filtering condition is a complete match of the brand, manufacturer, model series, year, and powertrain information fields, with fuzzy matching for the vehicle type information field; the third-level filtering condition is a complete match of the brand, manufacturer, model series, and powertrain information fields, with fuzzy matching for the vehicle type information field. The matching results are then ranked based on relevance to the year information field, resulting in a ranking result. Finally, a selection is made based on preset selection criteria. The ranking strategy for relevance ranking prioritizes matching the target year; if no target year result is found, the absolute value of the difference between the candidate year and the target year is calculated, with a smaller absolute value indicating higher relevance; the preset selection criterion is that when the absolute values ​​of the differences are the same, the result with the younger year is selected first.

[0066] Specifically, exact match refers to Elasticsearch's structured exact query method, which performs a term query operation on a specified field. This query method requires that the input value exactly matches the index value of the corresponding field in the document, and does not perform tokenization or fuzzy processing. For example, when performing exact match on fields such as brand, manufacturer, model, year, and power type, the document will only be matched if the value of the query condition is strictly identical to the value of the corresponding field in the document. Fuzzy match refers to Elasticsearch's full-text search mechanism, which performs a match query operation on a specified field. This query method first performs tokenization on the input value and then matches the tokenized results with the inverted index of the corresponding field in the document. A relevance score is calculated based on the degree of matching. The matching results can include documents that completely contain the query term or documents that partially contain the query term. Fuzzy match is suitable for situations such as vehicle model fields that have variations, abbreviations, or partial descriptions, and it has strong fault tolerance and search flexibility.

[0067] Specifically, in step S3, based on the knowledge base, a three-level progressive filtering mechanism is used to perform a three-way recall operation on the query statement, standard entity information, and matching information, resulting in three recall results, including:

[0068] S31. Input the query statement into the vector model to obtain the dense vector of the query statement;

[0069] S32. Based on the knowledge base, the KNN algorithm (K-Nearest Neighbor) and a three-level progressive filtering mechanism are used to calculate the similarity of the dense vectors of the query statement to obtain the first similarity set. The first similarity set is then filtered to obtain the first preliminary candidate set.

[0070] S33. The BM25 algorithm (Best Matching 25) is used to perform a relevance weighted calculation on the query statement and the first preliminary candidate set to obtain the first total relevance score;

[0071] S34. The first preliminary candidate set is scored according to the first total relevance score to obtain the first scoring result. The first preliminary candidate set is sorted according to the first scoring result to obtain the first sorting result.

[0072] S35. Select samples from the first ranking results to obtain the first set of highly relevant results.

[0073] It should be noted that the first preliminary candidate set includes the fields vlm_title, mineru_title, vlm_text, filename, and pdf_path. In step S33, preferably, the vlm_title field is assigned a weight of 3 times, and the filename field is assigned a weight of 2 times. When selecting samples from the first ranking results, a preset first selection rule is adopted. In this embodiment, the preset first selection rule is the top N most relevant samples, where N is a natural number and can be set according to specific circumstances. Using the first-path recall scheme can effectively handle text blocks with similar semantics but different expressions, improving the matching ability and recall rate at the semantic level. For example, if the query mentions "circuit diagram of XX" but the knowledge base contains "schematic diagram of XX", accurate recall can still be achieved.

[0074] Furthermore, in step S3, based on the knowledge base, a three-level progressive filtering mechanism is used to perform a three-way recall operation on the query statement, standard entity information, and matching information to obtain three-way recall results, which also include:

[0075] S36. Based on a pre-defined professional data dictionary, the query statement is encoded to obtain a sparse vector of the query statement;

[0076] S37. Based on the knowledge base, the KNN algorithm and a three-level progressive filtering mechanism are used to calculate the similarity of the sparse vectors of the query statement to obtain a second similarity set. The second similarity set is then filtered to obtain a second preliminary candidate set.

[0077] S38. The BM25 algorithm is used to perform a relevance weighted calculation on the query statement and the second preliminary candidate set to obtain the second total relevance score;

[0078] S39. The second preliminary candidate set is scored according to the second total relevance score to obtain the second scoring result. The second preliminary candidate set is then ranked based on the second scoring result to obtain the second ranking result.

[0079] S310. Select samples from the second ranking results to obtain the second set of highly relevant results.

[0080] It should be noted that the query statement is encoded using a one-hot encoding method. One-hot encoding is a technique that converts categorical variables into binary vectors. Its principle is to represent N categories using an N-bit binary vector, with each category corresponding to a single 1 bit in the vector, and the rest being 0 bits. The second preliminary candidate set includes the vlm_title, mineru_title, vlm_text, filename, and pdf_path fields. In step S38, preferably, the vlm_title field is assigned a weight of 3 times, and the filename field is assigned a weight of 2 times. When selecting samples from the second ranking results, a preset second selection rule is used. In this embodiment, the preset second selection rule is the top N most relevant samples, where N is a natural number and can be set according to specific circumstances. Using the second-path recall scheme has a significant effect when processing text blocks containing professional domain terms, achieving higher accuracy and recall rates, such as for the retrieval of professional terms like "time-matching".

[0081] Furthermore, in step S3, based on the knowledge base, a three-level progressive filtering mechanism is used to perform a three-way recall operation on the query statement, standard entity information, and matching information to obtain three-way recall results, which also include:

[0082] S311. Process the text block to obtain fault codes, and obtain a third preliminary candidate set based on the knowledge base and fault codes;

[0083] S312. The BM25 algorithm and a three-level progressive filtering mechanism are used to perform relevance weighting calculation on the query statement and the third preliminary candidate set to obtain the third total relevance score;

[0084] S313. Based on the third total relevance score, score the third preliminary candidate set to obtain the third scoring result, and rank the third preliminary candidate set based on the third scoring result to obtain the third ranking result;

[0085] S314. Select samples from the third ranking results to obtain the third set of highly relevant results.

[0086] It should be noted that the third preliminary candidate set includes the fields `diagnosis_code`, `vlm_title`, `mineru_title`, `vlm_text`, `filename`, `components`, `synonyms`, and `pdf_path`. The `components` field contains entity information. In step S312, the `diagnosis_code` and `vlm_title` fields are assigned a weight of 3 times, and the `filename`, `components`, and `synonyms` fields are assigned a weight of 2 times to enhance their influence on the final relevance score. When selecting samples for the third ranking results, a preset third selection rule is adopted. In this embodiment, the preset third selection rule is the top N most relevant samples, where N is a natural number and can be set according to specific circumstances. Using the third-way recall scheme is significantly effective in processing coded descriptions, achieving high-precision recall and improving the coverage and accuracy of professional retrieval, such as information from structured identifiers like fault codes.

[0087] S4. Based on the relevance score set, the query statement is processed using a third-party re-ranking model to obtain a one-dimensional relevance score. The relevance score set and the one-dimensional relevance score are merged to obtain a total relevance score set. Based on the logistic regression model, the total relevance score set is linearly weighted using pre-learned feature weights to obtain a comprehensive score. The relevance score set is then sorted based on the comprehensive score to obtain the query result set.

[0088] It should be noted that the third-party re-ranking model is a deep learning-based model. In this embodiment, it is the gte-rerank-v2 model from Alibaba Tongyi Qianwen. There are other similar text re-ranking models in the industry, such as BGE-Reranker from Zhiyuan, which are not limited here. The pre-learning process of feature weights includes: constructing a batch of training datasets, which include explicit target items; processing the training datasets through steps S2-S4 to obtain a training candidate set; labeling the training candidate set according to the target item, with the same item as the target item marked as 1 and different items marked as 0; extracting the training relevance score of each candidate item in the training candidate set; and inputting the training relevance score into the ranking weight model for training to obtain the feature weights.

[0089] The formula for calculating the overall score is as follows:

[0090] ;

[0091] In the formula, For the overall score, The weight of the first relevance score, The first relevance score, The weight of the second relevance score, The second relevance score, The weight of the third relevance score, The third relevance score, The weights for the one-dimensional relevance score. This represents the one-dimensional correlation score.

[0092] It should be noted that the ranking weight model can automatically learn the weight distribution of various relevance scores in the actual ranking, calculate and integrate multiple recall paths and comprehensive relevance scores, effectively improve the matching degree between ranking results and user needs, and enhance the ability of intelligence and customization.

[0093] Based on the aforementioned disclosures, this embodiment performs retrieval and recall at the text block level. Each text block is associated with link information and coordinate information from the original document. By retaining these attribute information, textual traceability of the recall results can be achieved, enhancing interpretability and verifiability, and improving the recall rate. A three-level progressive filtering mechanism is introduced, employing a strategy that gradually loosens filtering conditions. The initial level uses the strictest matching criteria for result selection. If no valid results can be recalled, subsequent levels of recall logic are triggered sequentially, gradually relaxing the filtering conditions to ensure highly relevant recall results with a certain recommendation capability. This enhances adaptability and recall coverage under diverse retrieval needs. A three-pronged parallel recall strategy is introduced: the first channel focuses on semantic relevance recall, obtaining similar text through deep semantic matching; the second channel is used to recall special fields that are difficult for semantic models to cover, such as structured key identifiers like fault codes; and the third channel supplements the recall of professional domain dictionaries not covered by the second channel, further improving recall completeness. Different relevance weights are set for different fields during the matching process, effectively reducing missed and false positives while ensuring recall accuracy, improving retrieval accuracy, enhancing the accurate understanding of user needs, and achieving precise data location.

[0094] like Figure 2 As shown, the second aspect of this embodiment provides a knowledge retrieval system based on a large visual language model, including:

[0095] The knowledge base construction module is used to acquire non-editable documents and professional domain information. It uses layout analysis technology to perform layout analysis on the non-editable documents to obtain layout analysis results. Based on a preset reading order, it restores the order of the layout analysis results to obtain document restoration results. The document restoration results are then segmented to obtain text blocks. A visual language model is used to perform semantic optimization processing on the text blocks to obtain title information and content information. Based on the text blocks, title information, content information, and professional domain information, a knowledge base is constructed.

[0096] The data matching module is used to obtain the user's query statement, extract entities from the query statement using a large language model to obtain entity information, normalize the entity information to obtain standard entity information, and match the standard entity information with a preset vehicle model database to obtain matching information.

[0097] The recall fusion module is used to perform three-way recall operations on query statements, standard entity information and matching information using a three-level progressive filtering mechanism to obtain three-way recall results. The three-way recall results are then fused to obtain a fused result. Based on the relevance score in the fused result, a set of relevance scores is extracted to obtain the fused result.

[0098] The result generation module is used to process the query statement based on the relevance score set using a third-party re-ranking model to obtain a one-dimensional relevance score. Based on the relevance score set and the one-dimensional relevance score, a total relevance score set is obtained. Based on the logistic regression model, the total relevance score set is linearly weighted using pre-learned feature weights to obtain a comprehensive score. Based on the comprehensive score, the relevance score set is sorted to obtain the query result set.

[0099] The working process, working details, and technical effects of the knowledge retrieval system provided in the second aspect of this embodiment can be found in the knowledge retrieval method described in the first aspect, and will not be repeated here.

[0100] This embodiment provides a computer device including a memory, a processor, and a transceiver sequentially and communicatively connected. The memory stores a computer program, the transceiver sends and receives messages, and the processor reads the computer program and executes the knowledge retrieval method described in the first aspect. Specifically, the memory may include, but is not limited to, random-access memory (RAM), read-only memory (ROM), flash memory, first-in-first-out (FIFO) memory, and / or first-in-last-out (FILO) memory, etc.; the processor may include, but is not limited to, a microprocessor of the STM32F105 series. Furthermore, the computer device may also include, but is not limited to, a power module, a display screen, and other necessary components.

[0101] The working process, working details and technical effects of the aforementioned computer device provided in the third aspect of this embodiment can be found in the knowledge retrieval method described in the first aspect, and will not be repeated here.

[0102] The fourth aspect of this embodiment provides a computer program product, including a computer program or instructions, which, when executed by a computer, are used to implement the knowledge retrieval method as described in the first aspect.

[0103] The working process, working details, and technical effects of the aforementioned computer program product provided in this embodiment can be found in the knowledge retrieval method described in the first aspect, and will not be repeated here.

[0104] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A knowledge retrieval method based on a large visual language model, characterized in that, include: The process involves acquiring uneditable documents and professional domain information, performing layout analysis on the uneditable documents using layout analysis technology, obtaining layout analysis results, restoring the order of the layout analysis results based on a preset reading order, obtaining document restoration results, segmenting the document restoration results to obtain text blocks, using a visual language model to perform semantic optimization on the text blocks to obtain title information and content information, and constructing a knowledge base based on text blocks, title information, content information, and professional domain information. The system obtains the user's query statement, extracts entities from the query statement using a large language model, obtains entity information, normalizes the entity information to obtain standard entity information, and matches the standard entity information with a preset vehicle model database to obtain matching information. The entity information includes brand field information, manufacturer field information, vehicle series field information, model year field information, and vehicle model field information, and the matching information includes power type field information. Based on a knowledge base, a three-level progressive filtering mechanism is used to perform three-way recall operations on query statements, standard entity information, and matching information to obtain three-way recall results. The three-way recall results are then fused to obtain a fused result. Based on the relevance score in the fused result, a relevance score set is extracted. The three-level progressive filtering mechanism includes a first-level filtering condition, a second-level filtering condition, and a third-level filtering condition. The three-level progressive filtering mechanism is as follows: The first level of filtering requires a complete match of the brand field, manufacturer field, vehicle series field, model year field, vehicle type field, and powertrain type field. The second-level filtering criteria are: brand field information, manufacturer field information, vehicle series field information, model year field information, and power type field information are completely matched, while vehicle model field information is fuzzy matched. The third-level filtering conditions are: brand field information, manufacturer field information, vehicle series field information and power type field information are completely matched, and model field information is fuzzy matched to obtain matching results. The matching results are sorted by relevance based on the year field information to obtain sorted results. The sorted results are selected based on preset selection conditions. Based on the relevance score set, a third-party re-ranking model is used to process the query statement to obtain a one-dimensional relevance score. The relevance score set and the one-dimensional relevance score are merged to obtain a total relevance score set. Based on the logistic regression model, the total relevance score set is linearly weighted using pre-learned feature weights to obtain a comprehensive score. The relevance score set is then sorted based on the comprehensive score to obtain the query result set.

2. The knowledge retrieval method based on a large visual language model according to claim 1, characterized in that, The three recall results include the first recall result, the second recall result, and the third recall result; The method, based on a knowledge base, employs a three-level progressive filtering mechanism to perform a three-way recall operation on the query statement, standard entity information, and matching information, resulting in three recall results, including: Input the query statement into the vector model to obtain a dense vector of the query statement; Based on the knowledge base, the KNN algorithm and a three-level progressive filtering mechanism are used to calculate the similarity of dense vectors of query statements to obtain the first similarity set. The first similarity set is then filtered to obtain the first preliminary candidate set. The BM25 algorithm is used to perform a relevance weighting calculation on the query statement and the first preliminary candidate set to obtain the first total relevance score; The first preliminary candidate set is scored based on the first total relevance score to obtain the first scoring result. The first preliminary candidate set is then ranked based on the first scoring result to obtain the first ranking result. Samples are selected from the first ranking results to obtain the first set of highly relevant results.

3. The knowledge retrieval method based on a large visual language model according to claim 2, characterized in that, Based on a knowledge base, a three-level progressive filtering mechanism is used to perform a three-way recall operation on the query statement, standard entity information, and matching information, resulting in three-way recall results, which also include: Based on a pre-defined professional data dictionary, the query statement is encoded to obtain a sparse vector of the query statement; Based on the knowledge base, the KNN algorithm and a three-level progressive filtering mechanism are used to calculate the similarity of the sparse vectors of the query statement to obtain a second similarity set. The second similarity set is then filtered to obtain a second preliminary candidate set. The BM25 algorithm is used to perform relevance weighting calculation on the query statement and the second preliminary candidate set to obtain the second total relevance score; The second preliminary candidate set is scored based on the second total relevance score to obtain the second scoring result. The second preliminary candidate set is then ranked based on the second scoring result to obtain the second ranking result. Sample selection is performed on the second ranking results to obtain the second set of highly relevant results.

4. The knowledge retrieval method based on a large visual language model according to claim 2, characterized in that, Based on a knowledge base, a three-level progressive filtering mechanism is used to perform a three-way recall operation on the query statement, standard entity information, and matching information, resulting in three-way recall results, which also include: The text block is processed to obtain the fault code. Based on the knowledge base and the fault code, a third preliminary candidate set is obtained. The BM25 algorithm and a three-level progressive filtering mechanism are used to perform relevance weighting calculation on the query statement and the third preliminary candidate set to obtain the third total relevance score; The third preliminary candidate set is scored based on the third total relevance score to obtain the third scoring result. The third preliminary candidate set is then ranked based on the third scoring result to obtain the third ranking result. Sample selection is performed on the third ranking results to obtain the third set of highly relevant results.

5. The knowledge retrieval method based on a large visual language model according to claim 1, characterized in that, The total relevance score set includes a first relevance score, a second relevance score, a third relevance score, and a one-dimensional relevance score; the calculation expression for the comprehensive score is: OVE score =A w *A score +B w *B score +C w *C score +rerank w *rerank score ; In the formula, OVE score For the overall score, A w As the weight of the first relevance score, A score For the first relevance score, B w B is the weight of the second relevance score. score For the second relevance score, C w C is the weight of the third relevance score. score The third relevance score, rerank w The rerank is the weight of the one-dimensional relevance score. score This represents the one-dimensional correlation score.

6. A knowledge retrieval system based on a large visual language model, used to implement the method according to any one of claims 1 to 5, characterized in that, include: The knowledge base construction module is used to acquire non-editable documents and professional domain information. It uses layout analysis technology to perform layout analysis on the non-editable documents to obtain layout analysis results. Based on a preset reading order, it restores the order of the layout analysis results to obtain document restoration results. The document restoration results are then segmented to obtain text blocks. A visual language model is used to perform semantic optimization processing on the text blocks to obtain title information and content information. Based on the text blocks, title information, content information, and professional domain information, a knowledge base is constructed. The data matching module is used to obtain the user's query statement, extract entities from the query statement using a large language model to obtain entity information, normalize the entity information to obtain standard entity information, and match the standard entity information with a preset vehicle model database to obtain matching information. The entity information includes brand field information, manufacturer field information, vehicle series field information, model year field information, and vehicle model field information, and the matching information includes power type field information. The recall fusion module is used to perform three-way recall operations on query statements, standard entity information and matching information based on a knowledge base and using a three-level progressive filtering mechanism to obtain three-way recall results. The three-way recall results are then fused to obtain a fusion result. Based on the relevance score in the fusion result, a relevance score set is extracted. The three-level progressive filtering mechanism includes a first-level filtering condition, a second-level filtering condition and a third-level filtering condition. The three-level progressive filtering mechanism is as follows: The first level of filtering requires a complete match of the brand field, manufacturer field, vehicle series field, model year field, vehicle type field, and powertrain type field. The second-level filtering criteria are: brand field information, manufacturer field information, vehicle series field information, model year field information, and power type field information are completely matched, while vehicle model field information is fuzzy matched. The third-level filtering conditions are: brand field information, manufacturer field information, vehicle series field information and power type field information are completely matched, and model field information is fuzzy matched to obtain matching results. The matching results are sorted by relevance based on the year field information to obtain sorted results. The sorted results are selected based on preset selection conditions. The result generation module is used to process the query statement based on the relevance score set using a third-party re-ranking model to obtain a one-dimensional relevance score. The relevance score set and the one-dimensional relevance score are merged to obtain a total relevance score set. Based on the logistic regression model, the total relevance score set is linearly weighted using pre-learned feature weights to obtain a comprehensive score. The relevance score set is then sorted based on the comprehensive score to obtain the query result set.

7. A computer device, characterized in that, The device includes a memory, a processor, and a transceiver that are sequentially and communicatively connected. The memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the knowledge retrieval method as described in any one of claims 1 to 5.

8. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or the instructions are executed by the computer, they implement the knowledge retrieval method as described in any one of claims 1 to 5.

Images