Multi-modal retrieval enhancement generation optimization method in aviation field based on tree index

Abstract

The invention provides an aviation field multi-modal retrieval enhancement generation optimization method based on a tree index, and relates to the technical field of aviation field large model retrieval enhancement, and the method comprises the following steps: S1, constructing a source data set of an aviation field multi-modal knowledge base; s2, constructing and retrieving a tree-shaped index based on an embedded model; s3, performing multi-dimensional score optimization on the retrieval candidate set; and S4, performing fusion sorting on the multi-dimensional scores to obtain a related context set for guiding generation of question and answer results. By improving index establishment and score calculation parts generated by traditional retrieval enhancement, the problems of low recall rate, poor multilayer semantic information understanding and the like when a retrieval enhancement generation system faces a multi-modal long document in the aviation field are solved, and increasingly complicated requirements of the aviation industry are met.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of aviation field large model retrieval enhancement, and particularly relates to an aviation field multi-modal retrieval enhancement generation optimization method based on a tree index. BACKGROUND

[0002] In recent years, large language models (LLMs) based on the Transformer architecture have made breakthrough progress in the field of natural language processing (NLP). Such models, through self-supervised pre-training on large-scale text data, have shown excellent language understanding and generation capabilities, and are widely used in dialog systems, automatic abstracts, machine translation and code generation tasks. However, although large language models have good language modeling capabilities in general contexts, they still face a series of challenges in specific domains, high-precision tasks or knowledge-intensive scenarios, especially in the limitations of knowledge boundaries, factual consistency and controllability.

[0003] One major problem is that the knowledge source of LLMs is limited to the information contained in its training corpus, so when faced with queries or long-tail domain problems outside the training corpus, the model is prone to the so-called "hallucination" phenomenon, that is, generating fluent language but factually incorrect or fictional content. This behavior in medical, legal, financial, scientific research and other scenarios with high requirements for factual accuracy can cause serious consequences, limiting the practical availability of large language models in high-trust applications.

[0004] To solve the above problems, the academic and industrial communities have proposed the retrieval-augmented generation (RAG) technical framework. The RAG method decouples the external knowledge retrieval mechanism from the language model, and introduces an external updateable, structured or unstructured knowledge base as a fact supplement source. Before generating an answer, the system will first encode the user's input query semantically, and retrieve a number of relevant document fragments or entries from the knowledge base through similarity calculation, and then send these contents together with the original input to the language model for answer generation. By explicitly introducing relevant knowledge, RAG can effectively improve the model's performance in factual consistency, explainability and cross-domain generalization, significantly reducing the error generation rate.

[0005] The advantages of RAG not only lie in improving generation accuracy, but also in the flexibility and scalability brought by its modular architecture. Compared to directly "solidifying" knowledge in a large language model, through an external retrieval mechanism, dynamic updating and multi-source integration of knowledge can be achieved. For example, in search engines, combining RAG architecture can achieve more real-time and more comprehensive question and answer services; in enterprise knowledge management systems, combining RAG can build an intelligent assistant that supports dynamic business changes. In addition, RAG is also widely used in intelligent customer service, scientific research writing assistance, educational question answering, and other fields.

[0006] Although the current RAG method has achieved remarkable results in practical applications, it still has several technical challenges, such as the selection of retrieval document granularity, the semantic matching of knowledge and queries, the fusion ability of multi-modal information, and the utilization efficiency and controllability of retrieval information in the generation process. Especially in the multi-modal scene, user queries may involve images, tables and other non-text information, and traditional text retrieval and generation frameworks are difficult to handle directly. Therefore, building an efficient and controllable retrieval-enhanced generation system for multi-modal scenarios has become an important direction in current research and application.

[0007] In summary, to solve the illusion problem of large language models in professional knowledge scenarios and further improve their multi-modal interaction capabilities, it is necessary to propose an enhanced generation method that integrates multi-modal retrieval mechanisms to achieve high-quality natural language responses in complex information environments. The development of this technology not only helps to promote the evolution of artificial intelligence systems in the direction of specialization and credibility, but also provides a solid technical foundation for intelligent question answering systems, search engines, and multi-modal human-computer interaction applications. SUMMARY

[0008] In order to solve the above technical problems of the prior art, the purpose of the present application is to provide a tree index-based multi-modal retrieval-enhanced generation optimization method in the aviation field, which aims to improve the index establishment and score calculation parts of the traditional retrieval-enhanced generation to alleviate the low recall rate and poor multi-level semantic information understanding of the RAG system when facing multi-modal long documents in the aviation field, and to meet the increasingly complex needs of the aviation industry. By establishing a tree index on the original knowledge base, the adjacent information in the knowledge base can be fused to alleviate the loss of coherent semantic information caused by the segmentation granularity in the traditional RAG system. By introducing the score prior of long documents, the similarity score is adjusted using the likelihood method to exclude the influence of noise and irrelevant content.

[0009] Specifically, the present application provides a tree index-based multi-modal retrieval-enhanced generation optimization method in the aviation field, which comprises the following steps: S1, constructing a source data set of a multi-modal knowledge base in the aviation field; S2, tree index construction and retrieval based on embedding model, specifically comprising: S21, constructing a tree index based on an embedding model: taking the aviation multi-modal page obtained in S1 as a bottom node for representation, and completing node aggregation and tree index construction from bottom to top; S22, converting the input natural language query into a query embedding, performing hierarchical semantic retrieval on the tree index structure, and obtaining a retrieval candidate set as , wherein, is a candidate node set obtained by using a greedy strategy, is a candidate node set obtained by using a fixed-width strategy, and the retrieval candidate set is the node set input to the subsequent score optimization module; S3, multi-dimensional score optimization on the retrieval candidate set, specifically comprising: S31, score optimization based on likelihood value, the formula being as follows: ; , wherein, is the original retrieval score, is a likelihood score based on the spatial distribution rule of the similarity between pages in the aviation document, is a fusion coefficient, is the score after likelihood value optimization; S32, score optimization based on path context, the formula being as follows: ; , wherein, is the score after context optimization, is the embedding vector of the input aviation business query; is the embedding representation of the document page at the bottom node; is the parent node embedding of the page at the layer in the tree; is a path attenuation coefficient, used to adjust the weight of the context score at each layer and control the degree of influence of high-layer information on the score; is the layer of the tree index; S4, multi-dimensional score fusion and sorting are performed to obtain a relevant context set for guiding the generation of the question and answer result, and the aviation field multi-modal retrieval enhancement generation optimization is completed.

[0010] Preferably, the tree index constructed in step S21 is as follows: ; ; , wherein, is the layer A collection of image block nodes, ; Indicates taking The smaller one, This is the aggregation step size parameter; The number of nodes; express The set of child nodes; for In the The corresponding layer Number of child nodes After the termination condition is met, a tree index diagram is obtained. , It is the union of nodes at each level in the tree index.

[0011] Preferably, step S1 specifically includes: collecting relevant standards, airworthiness provisions, maintenance manuals, design manuals, and test reports in the aviation field, and parsing each document into a page-based image organization format to obtain a multimodal knowledge base in the aviation field. The source dataset, ; in, For the knowledge base Page 1 This represents the total number of pages in all documents of the knowledge base.

[0012] Preferably, step S21 specifically includes the following sub-steps: S211, Bottom-level node representation: Knowledge base Each page image in Input multimodal vector embedding model Extract the vector embedding representation of each page image. From page images Its vector embedding Each node that forms the bottom layer of the tree index ,Right now: ; S212, For the current layer top-level node set The number of nodes is According to the set aggregation step size parameters , will the current layer Adjacent The images in each child node are stitched together to generate... The first The layer consists of aggregated image blocks, each constructed as follows: ; Initially, ; For image stitching operations; For the first Layer by splicing the first The first layer image obtained A collection of image patches, Indicates the first The layer is used to construct the first The first layer A collection of images of aggregated image patches; S213, will Each aggregated image patch of the layer is input into a multimodal vector embedding model. Extract its semantic vector representation: ; The obtained node representation is as follows: ; No. The node set of the layer is: , ; S214, Record each generated first... Layer nodes Corresponding child node list This establishes parent-child connections, forming the set of directed edges in the tree index. ; S215. Repeat steps S212-S214 until any termination condition is met, at which point the tree index diagram is obtained. .

[0013] Preferably, in step S212, when When fixed, from index Starting with the image at index , select sequentially up to index . Images, enabling up to [number] clicks Aggregate in groups; when Cannot be When divisible, the upper bound index of the last group is That is, only the remaining sub-images are stitched together; The termination condition in step S215 is: the current number of top-level nodes is less than the threshold. Or the number of tree levels reaches a preset upper limit threshold. .

[0014] Preferably, step S22 specifically includes the following sub-steps: S221. Natural Language Query Embedding: Embedding natural language queries entered by aviation business personnel. After preprocessing, the data is input into the multimodal vector embedding model. In this process, the corresponding embedding vector is obtained. In a tree index, let the root node be... The child nodes of the root node are , Indicates the first The first layer Each node has an embedded representation as follows: ; S222. A hybrid hierarchical retrieval scheme combining greedy search and fixed-width search is adopted to obtain the candidate set, specifically: S2221. Using a greedy strategy, at each level, select the node most similar to the semantics of the query embedding vector from the current position, and recursively form a query path with the highest relevance layer by layer, starting from the root node. Begin by calculating the current layer layer by layer. All nodes and query vectors cosine similarity score : , ; in, For the first Total number of nodes in the layer; Nodes are added to a priority queue based on their similarity scores, and then the node with the highest score is selected as the best matching node for that layer. : ; Then the child nodes of this node As the candidate set for the next layer, repeat step S2221 to eventually form a greedy path: ; in, The number of levels in the tree; Indicates from the first Layer to the first The best matching node of the layer If the highest similarity score in the current layer is lower than a preset threshold, the search will stop prematurely, stopping the search at that layer. The best matching node is denoted as the end node. ; S2222. Introduce a fixed-width strategy to perform horizontal expansion at each layer. For the first layer... Select all nodes in the layer that are related to... The most similar The nodes form a fixed-width candidate set. : ; in, This means taking the top two with the highest similarity. There are 10 candidate nodes, and then all the child nodes of these candidate nodes are gathered into the node set of the next layer. Repeat step S2222 until the query reaches the bottom leaf node, finally obtaining the union of the terminal nodes. : ; in, This indicates the range from the 1st floor to the 2nd floor. All fixed-width candidate sets of the layer Perform a union operation to gather the total set of candidate nodes; S223, Set the greedy strategy node set With fixed-width strategy node set union As a candidate set for retrieval.

[0015] Preferably, step S4 specifically includes the following sub-steps: S41. Constructing the fusion score: Based on the multi-dimensional scores of each candidate node obtained in step S3, the final score is calculated using the fusion formula: ; in, For weight adjustment parameters, The final score after fusion; S42. Diverse Reordering Strategy: For the obtained candidate node set And the score obtained after merging with it through step S41 The pages are reordered to obtain a set of relevant pages used to guide the generation of subsequent questions and answers. , The total number of candidate nodes; this step specifically includes: S421. Initialize the final candidate set. In each round, candidate nodes are selected from the set using the following heuristic objective function. Select the node with the highest score and the lowest similarity to the already selected nodes. and join ; ; in, Candidate nodes The fusion score; The weight of the redundancy penalty term; Candidate nodes and Vector cosine similarity; express With the present The maximum similarity between the most similar nodes in a set; S422. Repeat step S421 until the set is complete. Once the number of nodes reaches the preset upper limit for the number of candidates, a set of relevant pages is obtained to guide the generation of subsequent question-and-answer questions. ; S43. Generate question and answer results: Combined with user natural language queries The input is fed into a multimodal large language model for generative reasoning, resulting in the final aviation question-answering results.

[0016] Preferably, step S31 specifically includes the following sub-steps: S311. Construct the log-likelihood expression as follows: ; in, Indicates the number of candidate nodes; let the distance from the target answer page be... Candidate pages of a page, with scores following a variance of . normal distribution ,function The similarity score indicates that the score varies with the score. The expected value of the change; S312. By reconstructing the likelihood score and fusing it with the weights, we obtain a score optimization formula based on the likelihood value: The optimization objective is: ; Introducing hyperparameters Adjust the relative weights of the linear and squared terms to construct the likelihood score: ; Finally, the fusion coefficient is introduced. The original score and the likelihood score are linearly combined to obtain the final adjusted score.

[0017] Preferably, step S32 specifically includes the following sub-steps: S321, Path Context Embedding Representation: For a specific page of a document Let the embedding sequence from the leaf node representing this node to the top-level node be: ; in, This is an embedded representation of the page itself; This indicates that the node is the [number]th node in the tree. The parent node of the layer is embedded. ; S322, Path Context Score: Introducing a path weighted fusion mechanism, attenuation weights are added to embeddings at different levels to obtain the context score formula.

[0018] Preferably, in step S2221, the terminal nodes are leaf-mapped: .

[0019] The above formula yields This is the set of candidate pages for the greedy strategy.

[0020] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) This invention constructs a hierarchical index by using pages as the basic unit of the RAG retrieval system and integrates adjacent content. Page images serve as a unified carrier to simultaneously carry multimodal elements such as text, tables, curves, schematic diagrams / schematic diagrams, formulas and layout structures. It preserves the structure and semantics of cross-pages, avoids semantic fragmentation and text-image alignment errors caused by pure text segmentation, and achieves efficient organization and retrieval of aviation data, thereby providing users with a richer information interaction experience.

[0021] (2) This invention implements a hierarchical retrieval strategy on a tree index that takes into account both relevance and coverage, thereby achieving efficient location and semantic expansion of long documents. Under the premise of controllable computing resources, it can quickly lock the suspected answer area, ensure full coverage of various types of evidence such as standard clauses, maintenance procedures, and diagram descriptions, solve the problems of insufficient recall and high off-topic rate of traditional RAG in complex multimodal data, and improve the stability and applicability of the system in aviation question answering and knowledge retrieval.

[0022] (3) This invention optimizes the scores of candidate pages by combining the location information patterns of documents with the path context to suppress redundant and noisy pages, thereby achieving more accurate and reliable result ranking. It can effectively reduce the risk of illusion generated by RAG, improve the consistency and controllability of answers, and thus meet the high precision and high reliability requirements of the aviation field for knowledge retrieval and content generation. Attached Figure Description

[0023] Fig. 1 This is a schematic diagram of the overall process of the present invention; Fig. 2 This is a flowchart of the method of the present invention; Fig. 3 This is a schematic diagram of the tree index graph construction process of the present invention. Detailed Implementation

[0024] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[0025] Specifically, this invention provides a tree-index-based method for enhancing and optimizing multimodal retrieval in the aviation field, such as... Figs. 1 to 3 As shown, it includes the following steps: S1. Construct the source dataset for a multimodal knowledge base in the aviation field. Specific steps include: collecting relevant aviation standards, airworthiness regulations, maintenance manuals, design manuals, test reports, and other documents; parsing each document into a page-based image-organized format to obtain the multimodal knowledge base for the aviation field. The source dataset, ; in, For the knowledge base Page 1 This represents the total number of pages in all documents of the knowledge base.

[0026] S2. Building and retrieving a tree-based index based on the embedding model, specifically including the following sub-steps: S21. Construct a tree index based on the embedding model: Use the aviation multimodal page obtained in step S1 as the bottom-level node, and complete node aggregation and tree index construction from bottom to top. The constructed tree index is as follows: ; ; in, For the first Layer A collection of image block nodes, ; Indicates taking and The smaller one, This is the aggregation step size parameter; The number of nodes; express The set of child nodes; for In the The corresponding layer Number of child nodes After the termination condition is met, a tree index diagram is obtained. , It is the union of nodes at each level in the tree index.

[0027] Step S21 specifically includes the following sub-steps: S211, Bottom-level node representation: Knowledge base Each page image in Input multimodal vector embedding model Extract its vector embedding representation From page images Its vector embedding Each node that forms the bottom layer of the tree index ,Right now: .

[0028] S212, For the current layer (Initially, The top-level node set of ) The number of nodes is According to the set aggregation step size parameters , will the current layer Adjacent The images in each child node are stitched together to generate... The first The layer consists of aggregated image blocks, each constructed as follows: ; in, This indicates an image stitching operation; For the first Layer by splicing the first The first layer image obtained A collection of image patches, Indicates the first The layer is used to construct the first The first layer A collection of images of aggregated image patches: when When fixed, from index Starting with the image at index , select sequentially up to index . Images, enabling up to [number] clicks Aggregate in groups; when Cannot be When divisible, the upper bound index of the last group is This means that only the remaining sub-images are stitched together.

[0029] S213, will Each aggregated image patch of the layer is input into the embedding model. Extract its semantic vector representation: .

[0030] Node representation obtained: .

[0031] No. The node set of the layer is: , .

[0032] S214, Each generated Layer nodes Record its corresponding child node list This establishes parent-child connections, forming the set of directed edges in the tree index. .

[0033] S215. Repeat steps S212-S214 until any stopping condition is met. The stopping condition is: the current number of top-level nodes is less than a threshold. Or the number of tree levels reaches a preset upper limit threshold. .

[0034] S22. Convert the natural language query input by aviation business personnel (such as "minimum bending radius requirement for laying this type of aluminum cable") into a query embedding, and then perform hierarchical semantic retrieval on a tree index structure to obtain the retrieval candidate set. ,in, This is the set of candidate nodes obtained using a greedy strategy. Given the candidate node set obtained using a fixed-width strategy, retrieve the candidate set. This refers to the set of nodes input to the subsequent score optimization module. Step S22 specifically includes the following sub-steps: S221. Natural Language Query Embedding: Natural language queries entered by aviation business personnel. (For example, "Minimum bending radius requirement for laying this type of aluminum cable") After preprocessing, it is input into the same embedded model as the tree index construction stage. In this process, its embedding vector is obtained. In a tree index, let the root node be... Its subordinate nodes are , Indicates the first The first layer Each node has an embedded representation as follows: .

[0035] S222. A hybrid hierarchical retrieval scheme combining greedy search and fixed-width search is adopted to obtain the candidate set: S2221. The greedy strategy selects the node most similar to the semantics of the query embedding vector from the current position at each level, recursively forming the most relevant query path layer by layer downwards, starting from the root node. Begin by calculating the current layer layer by layer. All nodes and query vectors cosine similarity score : , ; in, For the first Total number of nodes in the layer; Nodes are added to a priority queue based on their similarity scores, and then the node with the highest score is selected as the best matching node for that layer. : .

[0036] Then the child nodes of this node As the candidate set for the next layer, repeat step S2221 to eventually form a greedy path: ; in, The number of levels in the tree; Indicates from the first Layer to the first The best matching node of the layer If the highest similarity score in the current layer is lower than a preset threshold, the search will stop prematurely, stopping the search at that layer. The best matching node is denoted as the end node. .

[0037] To ensure that all candidates are page-level leaf nodes, the terminal nodes are leafized: .

[0038] S2222. Introduce a fixed-width strategy to perform horizontal expansion at each layer. For the first layer... Select all nodes in the layer that are related to... The most similar The nodes form a fixed-width candidate set. : ; in, This means taking the top two with the highest similarity. Then, all child nodes of these candidate nodes are aggregated into the node set of the next layer. Repeat step S222 until the query reaches the bottom leaf node, finally obtaining the union of the terminal nodes. : .

[0039] in, This indicates the range from the 1st floor to the 2nd floor. All fixed-width candidate sets of the layer Perform a union operation to gather the total set of candidate nodes.

[0040] S223, Set the greedy strategy node set With fixed-width strategy node set union As a candidate set for retrieval.

[0041] S3. Perform multi-dimensional score optimization on the retrieval candidate set, which includes the following sub-steps: S31. Score optimization based on likelihood value, the formula is as follows: ; in, This is the original search score. This is a likelihood score based on the spatial distribution pattern of similarity between pages in aviation documents. The fusion coefficient is... The score is optimized based on the likelihood value. Step S31 specifically includes the following sub-steps: S311. Construct the likelihood function: The set of candidate node pages obtained in step S2 using the hierarchical semantic retrieval method is as follows: .

[0042] in, Given the total number of candidate nodes, the corresponding cosine similarity score is: .

[0043] Statistical analysis of actual aviation document retrieval samples revealed that the greater the page distance between a candidate page and the actual answer page, the more exponentially the similarity score decreases. Therefore, let the distance to the target answer page be... The candidate pages of the page, whose scores follow a variance of . normal distribution , The similarity score varies with page distance. The expected value of the change is used to characterize the decay trend of "the greater the page spacing, the lower the similarity". Let the probability density function of the normal distribution be denoted as . For page similarity score Calculate the first The joint likelihood values ​​for each location are the target answer page: .

[0044] Right now Substituting the probability density function of the normal distribution into the equation, we get: .

[0045] Taking the log-likelihood, we obtain the log-likelihood expression: .

[0046] S312, Likelihood score reconstruction and weight fusion, where the optimization objective is: .

[0047] Introducing hyperparameters Adjust the relative weights of the linear and squared terms to construct the likelihood score: ; Further introduce fusion coefficient The original score and the likelihood score are linearly combined to obtain the final adjusted score.

[0048] S32. Path context-based score optimization, defining the context score as: ; in, The score is optimized for the context. The embedding vector of the input aviation business query; This represents the embedding of a document page within the underlying nodes. Embed the page in the parent node at level L in the tree; This is the path decay coefficient, used to adjust the weight of the context scores at each layer, controlling the degree of influence of higher-level information on the score; For the first tree index layer.

[0049] Preferably, step S32 specifically includes the following sub-steps: S321. Path context embedding representation: for a specific page of a document Let the embedding sequence from the leaf node representing this node to the top-level node be: ; in, This is an embedded representation of the page itself; This indicates that the node is the [number]th node in the tree. The parent node of the layer is embedded. .

[0050] S322, Path Context Score: Introducing a path-weighted fusion mechanism, attenuation weights are added to embeddings at different levels, and the context score is defined as: ; in, Represents the user query vector; This is the path decay coefficient, used to adjust the weight of the context scores at each layer, controlling the degree of influence of higher-level information on the score.

[0051] S4. The scores from multiple dimensions are fused and ranked to obtain a set of relevant contexts to guide the generation of question-and-answer results. Step S4 specifically includes the following sub-steps: S41. Constructing the fusion score: Based on the multi-dimensional scores of each candidate node obtained in step S3, the final score is calculated using the fusion formula: ; in, It is a weighting adjustment parameter. This serves as the basis for the node's participation in the final sorting and rearrangement; the multi-dimensional score includes the original cosine similarity score. Optimization score based on likelihood distribution Path context enhancement score .

[0052] S42. Diverse rearrangement strategies: If only nodes with high scores are selected from highest to lowest during the sorting process, it can easily lead to duplicate selection of highly similar pages, resulting in information redundancy and affecting the diversity and effective length of the generated nodes. Therefore, a diversity re-sorting strategy is adopted. For the aforementioned candidate node set... And the score obtained after merging with it through step S41 .

[0053] S421. First, initialize the final candidate set. Each round uses the following heuristic objective function to... Select the node with the highest score and the lowest similarity to the already selected nodes. and join .

[0054] ; in, Candidate nodes The fusion score; The weight of the redundancy penalty term; Indicates candidate nodes and Vector cosine similarity; express With the present The maximum similarity between the most similar nodes in a set.

[0055] S422. Repeat step S421 until the set is complete. Once the number of elements in the dataset reaches the preset maximum number of candidates, a final set of relevant pages is obtained to guide subsequent question-and-answer generation. .

[0056] S43. Generate question and answer results: Combined with user natural language queries The input is fed into a multimodal large language model for generative reasoning, resulting in the final aviation question-answering result. The generated result can simultaneously display relevant page information, including the standard name, page number, etc., to facilitate traceability for business personnel.

[0057] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A tree index-based aviation domain multi-modal retrieval enhancement generation optimization method, characterized in that: It comprises the following steps: S1, constructing a source data set of an aviation field multi-modal knowledge base; S2, tree index construction and retrieval based on an embedding model, specifically comprising: S21, constructing a tree index based on an embedding model: the aviation multi-modal page obtained in S1 is represented as a bottom node, and node aggregation and tree index construction are completed from bottom to top; S22, convert the input natural language query into a query embedding, perform hierarchical semantic retrieval on the tree index structure, and obtain a retrieval candidate set as wherein, is a candidate node set obtained by using a greedy strategy, is a candidate node set obtained by using a fixed-width strategy, and the retrieval candidate set is the node set input to the subsequent score optimization module. S3, multi-dimensional score optimization for the retrieval candidate set, specifically comprising: S31, score optimization based on likelihood value, the formula being as follows: ； wherein, is the original search score, is the likelihood score based on the spatial distribution rule of the similarity between pages in the aviation document, is the fusion coefficient, is the score after optimization of the likelihood value; S32, score optimization based on path context, the formula being as follows: ； wherein, is the score optimized for the context, is the embedding vector of the input airline business query; is the embedding representation of the document page at the underlying node; is the parent node embedding of the page at the layer in the tree; is the path decay coefficient, used to adjust the weight of the context score at each layer, controlling the degree of influence of high-level information on the score; is the layer of the tree index; S4, fusion sorting for multi-dimensional scores, obtaining a relevant context set for guiding the generation of a question and answer result, and completing aviation field multi-modal retrieval enhancement generation optimization.

2. The tree index based aviation domain multi-modal search enhancement generation optimization method according to claim 1, characterized in that: The tree index constructed in step S21 is as follows: ； ； wherein, is the first layer of the tree index, is the first polymer image block node, represents the smaller of and , is a parameter of the aggregation step size; is the number of nodes; represents a set of child nodes of ; is the corresponding to the first layer of the tree index, is the first child node, ; after the termination condition is reached, a tree index graph is obtained, is the union set of nodes at each layer in the tree index, is the directed edge set in the tree index.

3. The tree index based aviation domain multi-modal search enhancement generation optimization method of claim 1, wherein: The step S1 specifically comprises: collecting relevant standards, airworthiness clauses, maintenance manuals, design manuals and test reports in the field of aviation, and parsing each obtained document into a form organized by page images to obtain a multi-modal knowledge base in the field of aviation of source data sets, ; wherein, is the number of pages of the knowledge base, is the number of pages of the knowledge base, is the total number of pages of all documents of the knowledge base.

4. The tree index based aviation domain multi-modal search enhancement generation optimization method of claim 1, wherein: Step S21 specifically comprises the following sub-steps: S211, the bottom layer node indicates: inputting each page image in the knowledge base a multi-modal vector embedding model , extracting a vector embedding representation of each page image , each node of the bottom layer of the tree index is composed of a page image and its vector embedding , namely:​​ ； S212、For the current layer , the top node set of the current layer , the number of nodes is , according to the set aggregation step size parameter , the images in the adjacent sub-nodes in the current layer are spliced to generate aggregated image blocks of the layer, and each aggregated image block is constructed by the following way: ； wherein, at the initial time, ; for image stitching operation; for the first layer through stitching the first layer image to obtain the first aggregated image block, represents the image set in the first layer for constructing the first aggregated image block of the first layer. S213、the each aggregated image block of the layer is input to a multi-modal vector embedding model extracting a semantic vector thereof representing: ； get node is represented as: ； No. The node set of the layer is: , ； S214, record each generated first Layer node Corresponding child node list , constitute parent-child connection relationship, form a directed edge set in tree index ; S215, repeating steps S212-S214 until termination when any termination condition is met, and obtaining the tree index map after the termination condition is met .

5. The tree index based aviation domain multi-modal search enhancement generation optimization method of claim 4, wherein: In step S212, when fixed, starting from the image with index , the image with index is selected in turn, so that at most groups are aggregated; when cannot be divided by , the upper index of the last group is , that is, only the remaining sub-images are spliced; The termination condition in step S215 is that the current number of top-level nodes is less than a threshold value or the number of layers of the tree reaches a preset upper threshold value .

6. The tree index based aviation domain multi-modal search enhancement generation optimization method of claim 1, wherein: Step S22 specifically comprises the following sub-steps: S221. Natural Language Query Embedding: Embedding natural language queries entered by aviation business personnel. After preprocessing, the data is input into the multimodal vector embedding model. In this process, the corresponding embedding vector is obtained. In a tree index, let the root node be... The child nodes of the root node are , Indicates the first The first layer Each node has an embedded representation as follows: ; S222, a hybrid hierarchical retrieval scheme combining greedy search and fixed-width search is adopted to obtain a candidate set, specifically as follows: S2221、Utilize the greedy strategy to select the node with the most similar semantics to the query embedding vector from the current position at each layer, and recursively form a query path with the highest relevance degree layer by layer from the root node Start, calculate the current layer layer by layer The cosine similarity score of all nodes and query vectors :​ , ； wherein, is the number of layers, total number of layer nodes; adding the nodes to a priority queue according to their similarity scores, and then selecting the node with the highest score as the best matching node for that layer : ； The child nodes of this node are then As a candidate set of next layer, repeat step S2221, and finally form a greedy path: ； wherein, is the number of layers of the tree; denotes the best matching node from the layer to the layer, ; if the highest similarity score of the current layer is lower than a pre-set threshold, the search is stopped early, and the best matching node of the stopping layer is denoted as the end node ; S2222. Introduce a fixed-width strategy to perform horizontal expansion at each layer. For the first layer... Select all nodes in the layer that are related to... The most similar The nodes form a fixed-width candidate set. : ； wherein, representing the top nodes with the highest similarity, and then collecting all the child nodes of these candidate nodes as the node set of the next layer , repeating step S2222 until the query reaches the bottom leaf node, and finally obtaining the union of the end nodes : ； wherein, represents a union operation on all the fixed-width candidate sets from the 1st layer to the layer, and the total candidate node set is collected; layer, and the total candidate node set is collected; S223, the union of the greedy strategy node set and the fixed-width strategy node set as the search candidate set.​ 7. The tree index based aviation domain multi-modal search enhancement generation optimization method of claim 1, wherein: Step S4 specifically comprises the following sub-steps: S41, constructing a fusion score: based on the multi-dimensional scores of each candidate node obtained in step S3, a fusion formula is used to calculate the final score: ； wherein, is a weight adjustment parameter, is the final score after fusion. S42, diversity rearrangement strategy: for the obtained candidate node set and the score obtained after fusion by S41 step reordering is performed to obtain a relevant page set for guiding subsequent question and answer generation , is the total number of candidate nodes; this step specifically includes: S421, initialize the final candidate set , in each round the node with the highest score and the smallest similarity to the already selected nodes is chosen from the candidate node set by the following heuristic objective function and added to the final candidate set ; ； wherein, is a fusion score for the candidate node ; is a weight for a redundancy penalty term; is a vector cosine similarity for the candidate node and ; denotes a maximum similarity to the most similar node in the current set of nodes; S422, repeat step S421 until the number of nodes in the set reaches a preset upper limit of candidate number, obtaining a relevant page set for guiding subsequent question and answer generation ; S43, generating a question and answer result: taking In combination with the user natural language query Input into a multimodal large language model for generation inference to form the final aviation question and answer result.

8. The tree index based aviation domain multi-modal search enhancement generation optimization method of claim 1, wherein: Step S31 specifically comprises the following sub-steps: S311, constructing a logarithmic likelihood expression as follows: ； wherein, denotes the number of candidate nodes; let the candidate pages of the target answer page, the score obeys a normal distribution with variance , the function denotes the expected value of the similarity score as a function of ; S312, reconstructing and fusing the likelihood score to obtain a score optimization formula based on likelihood value: Wherein the optimization target is: ； Introducing hyperparameters Adjusting the relative weights of the linear and squared terms, constructing the likelihood score term: ； Finally, a fusion coefficient is introduced The final adjusted score is obtained by linearly combining the raw score and the likelihood score.

9. The tree index based aviation domain multi-modal search enhancement generation optimization method of claim 1, wherein: Step S32 specifically comprises the following sub-steps: S321, the path context embedding representation: to a certain page of the document , the embedding representation sequence from the leaf node representing the node to the top node is: ； wherein, is an embedded representation of the page itself; represents the node's position in the tree as the parent node's embedding at layer l, ; S322, path context score: a path weighting fusion mechanism is introduced, and decay weights are added to different levels of embedding to obtain a context score formula.

10. The tree index based aviation domain multi-modal search enhancement generation optimization method of claim 6, wherein: In step S2221, the end nodes are leafed and mapped: ； The resulting formula is is the candidate set of pages for the greedy strategy.

Images