A large model rag retrieval method and system for mixed text and image output

Abstract

This invention belongs to the field of large-scale artificial intelligence model technology and discloses a method and system for large-scale RAG retrieval with mixed text and image output. The method preprocesses the original document corpus to generate a corpus tree, and processes the corpus of images, tables, etc., generating key-value pairs which are stored in a database. It generates word vectors for question-and-answer statements and searches them against the corpus tree to find the most relevant corpus. The most relevant corpus is concatenated into a prompt and then sent to the large-scale text-to-speech model. The large-scale question-and-answer model outputs relevant results. Based on the results output by the large-scale text-to-speech model, the key information of images and tables in the results is retrieved and replaced to generate the final question-and-answer results. This invention solves the problem of existing technologies struggling to handle data containing text, images, and charts, and generates highly accurate final question-and-answer results.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence large model technology, and in particular relates to a large model RAG retrieval method and system with mixed text and image output. Background Technology

[0002] In 2006, Geoffrey Hinton proposed a layer-by-layer unsupervised pre-training method to alleviate the difficulty of training deep networks due to gradient vanishing. Since then, deep neural networks have achieved groundbreaking developments in artificial intelligence fields such as computer vision and natural language processing. In 2017, Google proposed the Transformer architecture, a neural network structure based on self-attention mechanisms, laying the foundation for large-scale model pre-training algorithms. In 2018, OpenAI and Google released the GPT-1 and BERT large-scale models respectively, making pre-trained large-scale models mainstream in the field of natural language processing. At the end of 2022, the language large-scale model ChatGPT released by OpenAI attracted widespread attention. Powered by "large models + big data + high computing power," ChatGPT can complete various tasks through natural language interaction, possessing multi-scenario, multi-purpose, and interdisciplinary task processing capabilities. Large-scale models have gradually become a key infrastructure in the field of artificial intelligence. Through large-scale pre-training on massive amounts of unlabeled data, large-scale models can learn a large amount of linguistic and world knowledge, and through key technologies such as instruction fine-tuning and human alignment, they possess general solution capabilities for multiple tasks, exhibiting strong scalability. However, the inherent technical nature of large-scale models means that the accuracy of their outputs in certain specialized fields is questionable, which seriously affects their practical application. To address the issues of insufficient specialized knowledge and training with privacy-sensitive data, RAG retrieval technology emerged. However, since current data includes not only text but also supplementary information such as images and charts, text-based large-scale speech models and RAG technology struggle to process such data containing text, images, and charts. Summary of the Invention

[0003] To overcome the problems existing in related technologies, the present invention discloses a method and system for large-scale RAG retrieval with mixed text and image output.

[0004] The technical solution is as follows: A large-scale RAG retrieval method with mixed text and image output, comprising the following steps:

[0005] S1: Preprocess the original document corpus to generate a corpus tree, and process the graph and table corpus to generate key-value pairs and store them in the database;

[0006] S2 generates word vectors for the question-and-answer statements and searches them against the corpus tree to find the corpus with the highest relevance coefficient;

[0007] S3 concatenates the most relevant corpus into a prompt word and sends it to the text-to-speech model.

[0008] S4, the large text model outputs relevant results;

[0009] S5, based on the output of the large text language model, retrieves and replaces graph and table key information in the results to generate the final question-and-answer results.

[0010] Step S1 specifically includes:

[0011] S101, construct the corpus tree structure based on the directory; extract the title of the document as the root node of the corpus tree; recursively construct the child nodes of the corpus tree, the first level child node is the next level title and the adjacent non-title pre-defined paragraph, the second level title is the next level title and the adjacent non-title pre-defined paragraph, and so on, recursively traversing to construct the entire corpus tree.

[0012] S102, extract the figures and tables from the document, and perform dictionary hash mapping between table names and table content, and figure names and image content, and store them in the database. At the same time, the table names and figure names are unique.

[0013] S103: For the plain text data in the document, perform word vector processing to generate word vector representations, and attach all word vectors of each node to the corresponding tree node; use the Chinese word segmentation package Jieba to perform word segmentation on the entire text document; generate word vectors for the segmented word data, and use the FastText algorithm to encode the segmented words into vectors.

[0014] In step S102, the dictionary hash mapping process is as follows:

[0015] S1021, For tabular data, extract all text and data information from the table and connect them in the order of rows; use a hash function to calculate the hash value of the text, connect the hash value with the table name as the key, and use the chart content as the value to store it in the database;

[0016] S1022, For image data, first, the image is reduced to 8*8 pixels, then the color image is converted to grayscale, the processed image is subjected to Discrete Cosine Transform (DCT), the average pixel value of the transformed image is calculated, the value of each pixel is compared with the average value, if it is greater than the average value, it is 1, and if it is less than the average value, it is 0, resulting in a 64-bit sequence. The 64-bit sequence is input into a hash function to calculate the hash value, and the hash value is concatenated with the image name as the key and the image content as the value, and stored in the database.

[0017] Step S103 specifically includes:

[0018] S1031. Use the Chinese word segmentation package Jieba to segment the entire text document;

[0019] S1032. Generate word vectors from the segmented word data. The word vectors are generated using the FastText algorithm to encode the segmented words.

[0020] In step S2, word vectors are generated for the question-and-answer statements, including: segmenting the input question-and-answer statements into search terms and vectorizing them; using the Chinese word segmentation package jieba and the Fasttext word vectorization algorithm to convert the search question-and-answer statements into word vector representations; generating multiple word vectors; preprocessing the word vectors of the search statements by deleting prepositions and interrogative words to obtain the word vectors of the search terms.

[0021] In step S2, the corpus tree is searched, which includes performing RAG retrieval on the question-and-answer statements for which word vectors have been generated. The specific steps are as follows:

[0022] S201, RAG includes Retriever and Generator. Retriever is used to locate the corresponding document, and Generator is used to generate the corresponding answer.

[0023] S202, perform word vector retrieval in the corpus tree; if the root node does not have a word vector equal to the word vector of the search term, continue traversing the child nodes. If none of the child nodes contain the word vector of the search term, this corpus is unrelated to the search term, and the remaining corpus is processed in turn; if all corpus is unrelated to the search term, skip directly to step S3; if the content in the corpus tree contains the word vector of the search term, traverse all nodes in turn to find the corpus with the highest relevance. The corpus with the highest relevance is the one with the highest matching degree, calculated using the following formula; first compare the root node. If the word vector of the search term matches the word vector of the root node with a high degree of matching, the matching degree is calculated by the formula, and then sorted to determine the order; extract the root node and all its child nodes for use in step S3; if the matching degree is low, continue traversing its child nodes; the formula for calculating the word vector matching degree is as follows:

[0024] ;

[0025] In the formula, These are two vectors to be compared. For cosine similarity, Let be the magnitude of the vector;

[0026] S203, Sort the word vector segments retrieved in step S202 according to their relevance, and then take them in sequence. The most relevant word vector segment The value is set to represent the number of most relevant corpora retrieved; then the word vectors are restored to the corresponding text corpora to generate the final search results.

[0027] In step S202, the method for calculating the matching degree between the word vector of the search term and the word vector of the root node includes:

[0028] S2021, There are four word vectors defined;

[0029] S2022. If the number of word vectors in the root node is less than four, then perform recursive operation directly; if the number of word vectors in the root node is greater than or equal to four, then use the improved Gaussian mixture clustering algorithm to perform clustering and obtain the final four clustered word vectors.

[0030] S2023. If the word vector of the search term is less than the number of root node vectors, then calculate which Gaussian distribution the word vector of the search term belongs to, as obtained in step S2022; the Gaussian distribution is:

[0031] ;

[0032] In the formula, The maximum value, The coefficients of the Gaussian distribution function are... It is a Gaussian mixture distribution function; The number of Gaussian distributions, To add up values ​​from 1 to K, To obtain the maximum value;

[0033] S2024, the maximum value is obtained by calculating sequentially. indivual The relevance value between the retrieved word vector and the corpus word vector is obtained by adding them together, and the expression is:

[0034] ;

[0035] In the formula, This is the cumulative value of the correlation. This is the iterative symbol value from 1 to K for accumulating symbols.

[0036] In step S2022, the improved Gaussian mixture clustering algorithm includes:

[0037] S20221, to be clustered as Given a Gaussian distribution, set the Gaussian distribution function as follows:

[0038] ;

[0039] In the formula, It is a Gaussian mixture distribution. The function is a Gaussian distribution. For the parameters of the Gaussian distribution, , The mean, For variance; The category is a Gaussian distribution, with values ​​ranging from 0 to... ;

[0040] Gaussian distribution function The expression is:

[0041] ;

[0042] In the formula, Standard deviation It is an exponential function;

[0043] S20222, The likelihood function of the improved Gaussian mixture clustering is:

[0044] ;

[0045] In the formula, Let be the likelihood function. For the number of samples, All are hyperparameter coefficients. For weight regularization, The first in the Gaussian mixture distribution The probability distribution of the Gaussian components. For divergence calculation; It is a logarithmic function;

[0046] Divergence calculation The expression is:

[0047] ;

[0048] In the formula, The formula for calculating KL divergence is used to calculate the distribution. and The divergence;

[0049] Set initial values ,in, The coefficient is a real number. for The mean of a Gaussian distribution function, They are respectively The variance of a Gaussian distribution function;

[0050] S20223, Calculation:

[0051] ;

[0052] In the formula, This is the proportionality coefficient, with a value ranging from 0 to 1; For the first The coefficients of a Gaussian distribution function, It is a Gaussian distribution function;

[0053] S20224, Update parameters:

[0054] ;

[0055] In the formula, For the updated number The iterative value of the mean of a Gaussian distribution function. For the updated number The iterative values ​​of the variance of a Gaussian distribution function; For the updated number The iterative values ​​of the coefficients of the Gaussian distribution function. This represents the probability distribution of the nth Gaussian component. The category is Gaussian distribution. This is the formula for calculating differentials. This is the divergence calculation formula from the previous step;

[0056] S20225. Iterate through steps S20221-S20224 until convergence.

[0057] In step S3, the most relevant corpus is concatenated into a prompt word and sent to the text-to-speech model, including:

[0058] Based on the RAG retrieval results in step S2, the data is concatenated into a prompt and sent to the text-to-speech model. The corpus tree from step S1 is then processed by the text-to-speech model to obtain corpus nodes, where graphs and tables are assembled into the prompt in the form of keywords.

[0059] In step S5, based on the output of the large text language model, the graph and table key information in the replacement results is retrieved to generate the final question and answer results. This includes processing the question and answer results, firstly searching whether graph information appears in the question and answer results, and if it does, it is the corresponding key of the graph stored in the data in step S1. The corresponding graph is retrieved according to the key name, and then the final output answer is generated.

[0060] Another object of the present invention is to provide a large-scale RAG retrieval system with mixed text and image output, the system implementing the aforementioned large-scale RAG retrieval method with mixed text and image output, the system comprising:

[0061] The corpus tree generation module is used to preprocess the original document corpus, generate a corpus tree, and process the corpus of graphs and tables to generate key-value pairs and store them in the database.

[0062] The corpus tree retrieval module is used to generate word vectors for question-and-answer statements and search them against the corpus tree to find the most relevant corpus.

[0063] The Prompt concatenation module is used to concatenate the most relevant corpus into a prompt, which is then sent to the text-to-speech model.

[0064] The relevant results output module is used by the question-answering model to output relevant results;

[0065] The final question-and-answer result generation module is used to retrieve and replace the key information of graphs and tables in the results based on the output of the text-to-speech model, and generate the final question-and-answer results.

[0066] Combining all the above technical solutions, the beneficial effects of this invention are as follows:

[0067] First, this invention solves the problem of existing technologies struggling to handle data containing text, images, and charts, generating highly accurate final question-and-answer results. Furthermore, it utilizes an improved Gaussian mixture distribution algorithm to achieve rapid RAG retrieval, thus improving document retrieval efficiency. This invention effectively addresses the issue of large text language models being unable to output image information and achieves rapid RAG hybrid retrieval, greatly expanding the application scenarios of large text language models, improving the retrieval and response effects of various intelligent question-and-answer applications, and enhancing efficiency.

[0068] Secondly, this invention solves the problem that large text-based language models cannot output image information, enriching the application scenarios and effects of large text-based speech models. This invention addresses the issue that intelligent retrieval systems based on large text-based language models cannot output image information, breaking the bias that such systems can only process text information and adding image retrieval capabilities. Attached Figure Description

[0069] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention;

[0070] Figure 1 This is a flowchart of the large-model RAG retrieval method with mixed text and image output provided in the embodiments of the present invention;

[0071] Figure 2 This is a corpus tree diagram generated from the example corpus provided in this embodiment of the invention;

[0072] Figure 3 This is a corpus tree node diagram obtained through retrieval provided in an embodiment of the present invention;

[0073] Figure 4 This is a planning diagram of the first mining face provided in an embodiment of the present invention;

[0074] Figure 5 This is a planning diagram of the second mining face provided in an embodiment of the present invention. Detailed Implementation

[0075] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0076] The innovation of this invention lies in the following: This invention realizes the mixed output retrieval of text and images using a large text language model. Through an improved Gaussian mixture distribution, it quickly retrieves and matches the corresponding text content. Through hash mapping, it realizes the correspondence between images and text, thus solving the problem that the large text language model cannot directly retrieve and output images.

[0077] Example 1, such as Figure 1 As shown, the large-model RAG retrieval method with mixed text and image output provided in this embodiment of the invention includes:

[0078] S1: Preprocess the original document corpus to generate a corpus tree, and process the graph and table corpus to generate key-value pairs and store them in the database;

[0079] S2 generates word vectors for the question-and-answer statements and searches them against the corpus tree to find the corpus with the highest relevance coefficient;

[0080] S3 concatenates the most relevant corpus into a prompt word and sends it to the text-to-speech model.

[0081] S4, the large text model outputs relevant results;

[0082] S5, based on the output of the large text language model, retrieves and replaces graph and table key information in the results to generate the final question-and-answer results.

[0083] For example, step S1 specifically includes:

[0084] S101, construct the corpus tree structure based on the directory; extract the title of the document as the root node of the corpus tree; recursively construct the child nodes of the corpus tree, the first level child node is the next level title and the adjacent non-title pre-defined paragraph, the second level title is the next level title and the adjacent non-title pre-defined paragraph, and so on, recursively traversing to construct the entire corpus tree.

[0085] The root node represents the upper-level heading, and the child nodes represent the lower-level headings and their content. The leaf nodes represent the final text paragraphs. The generated tree structure is as follows: Figure 2 The corpus tree diagram generated from the example corpus is shown. The specific construction process is as follows: First, the title of the document is extracted as the root node of the corpus tree (taking the material on the current status of disaster and prevention as an example, the root node is "2.3 TMMK Disaster and Prevention Status"). Then, the child nodes of the corpus tree are constructed recursively. The first-level child nodes are the next-level headings and the adjacent non-heading paragraphs (taking the material on the current status of disaster and prevention as an example, the first-level child node is "2.3.1 Mine Overview -- TMMK Coal Mine is mined underground, with a mining elevation of +1258m to +1033m and a production capacity of 9 million tons per year. The minefield is defined by 18 inflection points, covering an area of ​​approximately 176.1346 km²). 2 It is divided into the first mining area (83.2861 km²). 2 ) and reserved area (92.8485km) 2 The project is divided into two parts. The first mining area is approved to mine coal seams 3, 4, and 5 (3...). -1 3 -2 3 -3 4 -3 4 -4 5 -2上 5 -2 Coal seams (including coal seams 4 and 5) are mined at elevations of +1258m to +1033m; the reserved area is for mining coal seams 4 and 5. -3 4 -4 5 -2上 5 -2 Coal seams), mining elevation: +1196m~+1033m”, the second level title is the next level title and the adjacent non-title expected section (taking the disaster and prevention status material later as an example, the first level sub-node is "(1) Mine field development method--the mine adopts adit development, single-level development of the whole mine field, and arranges four shafts: main adit, secondary adit, Daluyan return air inclined shaft, and Puniugeta return air inclined shaft; the main roadway adopts a centralized layout form, in 5 -2 A set of main roadways is arranged along the main and secondary adits of the coal seam, running through the entire mine field. The three main roadways are 50m apart. The middle one is the belt conveyor roadway, and the other two are the auxiliary transportation roadway and the return air roadway, respectively. The entire corpus tree is constructed by recursively traversing the roadway in sequence.

[0086] S102, extract the figures and tables from the document, and perform dictionary hash mapping between table names and table content, and figure names and image content, and store them in the database. At the same time, the table names and figure names are unique.

[0087] S103: For the plain text data in the document, perform word vector processing to generate word vector representations, and attach all word vectors of each node to the corresponding tree node; use the Chinese word segmentation package Jieba to perform word segmentation on the entire text document; generate word vectors for the segmented word data, and use the FastText algorithm to encode the segmented words into vectors.

[0088] The specific steps are as follows:

[0089] S1031. First, use the Chinese word segmentation package Jieba to segment the entire text document.

[0090] S1032. Generate word vectors from the segmented word data. The word vectors will be generated as follows: the FastText algorithm will be used to encode the segmented words into vectors.

[0091] For example, in step S2, generating word vectors for the question-and-answer statement includes: segmenting the input question-and-answer statement into search terms and vectorizing it, taking the following search question-and-answer statement "What is the general situation of the TMMK coal mine?" as an example:

[0092] Using the Chinese word segmentation package jieba and the Fasttext word vectorization algorithm, the search query "What is the general situation of the TMMK coal mine?" is converted into a word vector representation, generating 10 word vectors. After preprocessing the word vectors of the search query by removing prepositions and interrogative words, the following 7 word vector representations of the search term "TMMK coal mine general situation" are obtained. One representation of the word vectors is shown below:

[0093] Coal mine: [-2.6500077, -0.02617998, -1.2856188, 0.9191595, ...].

[0094] For example, in step S2, generating word vectors for the question-and-answer statements and performing a search with the corpus tree includes performing a RAG search on the question-and-answer statements with generated word vectors, specifically including the following:

[0095] S201, a typical RAG mainly consists of two parts: Retriever and Generator. Retriever is responsible for locating the corresponding document, and Generator is responsible for generating the corresponding answer.

[0096] S202, perform word vector retrieval in the corpus tree from step S1. If the root node does not have a word vector equal to the word vector of the search term, continue traversing the child nodes. If none of the child nodes contain the word vector of the search term, it means that this corpus has no relation to the search term, and other corpora are processed in turn. If all corpora have no relation to the search term, skip directly to step S3. If the content in the corpus tree contains the word vector of the search term, traverse all nodes in turn to find the corpus with the highest relevance. First, compare the root node. If the word vector of the search term matches the word vector of the root node with a high degree of matching, extract the root node and all its child nodes for use in step S3. If the matching degree is low, continue traversing its child nodes. The formula for calculating the word vector matching degree is as follows:

[0097] ;

[0098] In the formula, These are two vectors to be compared. For cosine similarity, Let be the magnitude of the vector;

[0099] For example, the method for calculating the matching degree between the word vector of the search term and the word vector of the root node includes:

[0100] S2021. Taking the search term "TMMK Coal Mine Overview" as an example, four word vectors are defined (the number of search vectors should correspond to the number of search terms).

[0101] S2022. If the number of word vectors in the root node is less than four, proceed directly to the next recursive operation; if the number of word vectors in the root node is greater than or equal to four, use the improved Gaussian mixture clustering algorithm to cluster them, obtaining the final four clustered word vectors. The improved Gaussian mixture clustering algorithm is shown below:

[0102] S20221, Suppose we want to cluster as K is the number of Gaussian distributions (K is the number of Gaussian distributions, which is 4 in this example), and the Gaussian distribution function is set as follows:

[0103] ;

[0104] In the formula, It is a Gaussian mixture distribution. The function is a Gaussian distribution. For the parameters of the Gaussian distribution, , The mean, For variance; The category is a Gaussian distribution, with values ​​ranging from 0 to... ;

[0105] Gaussian distribution function The expression is:

[0106] ;

[0107] In the formula, Standard deviation It is an exponential function;

[0108] S20222, The likelihood function of the improved Gaussian mixture clustering is:

[0109] ;

[0110] In the formula, Let be the likelihood function. For the number of samples, All are hyperparameter coefficients. For weight regularization, The first in the Gaussian mixture distribution The probability distribution of the Gaussian components. For divergence calculation; It is a logarithmic function;

[0111] Divergence calculation The expression is:

[0112] ;

[0113] In the formula, The formula for calculating KL divergence is used to calculate the distribution. and The divergence;

[0114] Set initial values ,in, The coefficient is a real number. for The mean of a Gaussian distribution function, They are respectively The variance of a Gaussian distribution function;

[0115] S20223, Calculation:

[0116] ;

[0117] In the formula, This is the proportionality coefficient, with a value ranging from 0 to 1; For the first The coefficients of a Gaussian distribution function, It is a Gaussian distribution function;

[0118] S20224, Update parameters:

[0119] ;

[0120] In the formula, For the updated number The iterative value of the mean of a Gaussian distribution function. For the updated number The iterative values ​​of the variance of a Gaussian distribution function; For the updated number The iterative values ​​of the coefficients of the Gaussian distribution function. This represents the probability distribution of the nth Gaussian component. The category is Gaussian distribution. This is the formula for calculating differentials. This is the divergence calculation formula from the previous step;

[0121] S20225. Iterate through steps S20221-S20224 until convergence.

[0122] S2023. If the word vector of the search term is less than the number of root node vectors, then calculate which Gaussian distribution the word vector of the search term belongs to, as obtained in step S2022; the Gaussian distribution is:

[0123] ;

[0124] In the formula, The maximum value, The coefficients of the Gaussian distribution function are... It is a Gaussian mixture distribution function; The number of Gaussian distributions, To add up values ​​from 1 to K, To obtain the maximum value;

[0125] S2024, the maximum value is obtained by calculating sequentially. indivual The relevance value between the retrieved word vector and the corpus word vector is obtained by adding them together, and the expression is:

[0126] ;

[0127] In the formula, This is the cumulative value of the correlation; This is the iterative symbol value from 1 to K for accumulating symbols.

[0128] S203, sort the word vector segments retrieved in step S202 according to their relevance, and take the m most relevant word vector segments in sequence (m is a set value, representing the number of most relevant corpora retrieved). Then, restore the word vectors to the corresponding text corpora to generate the final search results.

[0129] For example, step S3, concatenating the most relevant corpus into a prompt and then sending it to the large text-to-speech model, includes: based on the RAG retrieval results from step S2 above, concatenating them into a prompt and sending it to the large model. Continuing with the corpus tree from step S1 as an example, after the large model's computational output, the corpus nodes are obtained as follows: Figure 3 As shown. The figures and tables are still assembled into the prompt in the form of keywords.

[0130] For example, step S4 specifically includes: asking the large model a question, such as retrieving the question "What is the general situation of the TMMK coal mine?" The large model will output relevant results, as shown below: "The coal seams in the TMMK coal mine are all near-horizontal coal seams with no major geological structures. The main coal seam is the Jurassic Yan'an Formation 5." -2 The coal seam exhibits bifurcation and merging within the minefield. The mine's designed production capacity is 9.00 Mt / a. It employs longwall fully mechanized mining, mining the entire height in one pass, and using a retreating caving method to manage the roof of the working face. The mining face planning diagram is shown below. Figure 4 As shown.

[0131] For example, step S5 specifically includes: processing the question-and-answer results. First, it searches whether information such as charts appears in the question-and-answer results. For example, in the results above, the phrase "as shown in the schematic diagram of mine excavation continuity" appears. This is the chart corresponding key stored in the data in step S1. The corresponding chart is retrieved according to the key name, and then the final output answer is generated, as shown below:

[0132] "The coal seams in the TMMK coal mine are all near-horizontal, with no major geological structures. The main mining area is the Jurassic Yan'an Formation 5." -2 The coal seam exhibits bifurcation and merging within the minefield. The mine's designed production capacity is 9.00 Mt / a, employing longwall fully mechanized mining to extract the entire height in one pass, and retreating caving method for roof management. The mining face planning diagram is shown below. Figure 5 As shown.

[0133] Example 2: This invention provides a large-scale RAG retrieval system with mixed text and image output, comprising:

[0134] The corpus tree generation module is used to preprocess the original document corpus, generate a corpus tree, and process corpora such as graphs and tables to generate key-value pairs and store them in the database.

[0135] The corpus tree retrieval module is used to generate word vectors for question-and-answer statements and search them against the corpus tree to find the most relevant corpus.

[0136] The Prompt concatenation module is used to concatenate the most relevant corpus into a prompt, which is then sent to the text-to-speech model.

[0137] The relevant results output module is used by the question-answering model to output relevant results;

[0138] The final question-and-answer result generation module is used to retrieve and replace the key information of graphs and tables in the results based on the output of the text-to-speech model, and generate the final question-and-answer results.

[0139] To facilitate understanding of the object of application of this invention, some contents recorded in the "Current Status of Disasters and Prevention" materials in the prior art are selected as the object of application of this invention, as follows.

[0140] 2.3 Current Status of TMMK Disasters and Prevention

[0141] 2.3.1 Mine Overview: The TMMK coal mine is mined underground, with an elevation ranging from +1258m to +1033m. Its production capacity is 9 million tons per year. The minefield is defined by 18 inflection points, covering an area of ​​approximately 176.1346 km². 2 It is divided into the first mining area (83.2861 km²). 2 ) and reserved area (92.8485km) 2 The project is divided into two parts. The first mining area is approved to mine coal seams 3, 4, and 5 (3...). -1 3 -2 3 -3 4 -3 4 -4 5 -2 上 5 -2 Coal seams (including coal seams 4 and 5) are mined at elevations of +1258m to +1033m; the reserved area is for mining coal seams 4 and 5. -3 4 -4 5 -2上 5 -2 (Coal seams, etc.), with mining elevations of +1196m to +1033m.

[0142] (1) Well-field development method.

[0143] The mine adopts adit development, with single-level development of the entire minefield. It has four shafts: main adit, auxiliary adit, Daluyan return air inclined shaft, and Puniugeta return air inclined shaft. The main roadways are arranged in a centralized manner, with a group of main roadways running through the entire minefield along the main and auxiliary adits in the coal seam. The three main roadways are spaced 50m apart. The middle one is the belt conveyor roadway, and the other two are the auxiliary transportation roadway and the return air roadway, respectively.

[0144] (2) Panel division and mining sequence.

[0145] The coal seams in the mine are all near-horizontal, with no major geological structures. Single-level panel mining is employed. According to 5 -2The coal seam bifurcation line and the mineable boundaries of each coal seam above it are divided into 8 panels along the east-west direction, including 6 panels for coal seam No. 5 and 2 panels for coal seam No. 4 (panels No. 2 and No. 7). Coal seam No. 5 (5 -2 5 -2上 The coal seam mining roadways are arranged in layers, 5 -2上 The coal seam is divided into three panels: panel one, panel three, and panel six; the remaining five... -2 The coal seam is divided into panels five, four, and eight from east to west. All working faces employ a retreat mining method, advancing from the panel boundary towards the main roadway. According to the initial mine design, panels three and eight are selected as the first panels.

[0146] (3) Coal mining methods.

[0147] The longwall fully mechanized mining method is used to mine the entire height in one go, and the retreat mining method is used to manage the roof of the working face.

[0148] (4) Current status of mining.

[0149] TMMK Coal Mine mainly mines the Jurassic Yan'an Formation 5. -2 The coal seam exhibits bifurcation and merging within the mining area. The mine's designed production capacity is 9.00 Mt / a, but its current production capacity has not yet reached the designed capacity.

[0150] According to the initial mine design, the three panels (5) -2上 Coal), Bapan District (5 -2 The first panel is the coal mining area. Several working faces in the third panel have been mined. The mine is currently producing the fourth and eighth panels. The working faces being mined are the 45206 working face in the fourth panel and the 85216 fully mechanized working face in the eighth panel.

[0151] (5) Mining and Excavation Planning. See Table 1 for the Parameters of the Longwall Mining Face in 2024-2025;

[0152] Table 1. Parameters of Longwall Mining Faces in 2024-2025

[0153]

[0154] 2.3.2 Current status of mine water hazards.

[0155] The hydrogeological type of TMMK coal mine is medium, with an average water inflow of 82 m³ over the past three years. 3 / h, maximum inflow 100m³ 3 The main water sources in the mine are atmospheric precipitation, surface water, water from old workings, and groundwater aquifers. The primary water hazards currently posed by the mine are water accumulation in old workings and surface water.

[0156] (1) Atmospheric precipitation.

[0157] Located in the Loess Plateau of northern Shaanxi in northwestern inland China, the TMMK coal mine experiences relatively low rainfall, with an average annual rainfall of approximately 474.6 mm, mainly concentrated between July and September, with a maximum daily rainfall of 136.3 mm. Mine water inflow varies seasonally, generally replenishing the mine indirectly through infiltration of groundwater. The peak water inflow is delayed after rainfall. While the rainfall in the Shenfu mining area, where the TMMK mine is located, is not significant, the uneven and uncertain temporal distribution of atmospheric precipitation means that concentrated or prolonged periods of heavy rainfall during the rainy season can pose a threat to mine safety and production.

[0158] (2) Surface water.

[0159] Distribution of surface rivers.

[0160] There are three main rivers in the mining area, all of which form dendritic tributaries. The valleys in the first mining area are mainly tributaries of the TMMK in the eastern and southern parts of the mining field.

[0161] During the flood season, surface runoff in the valleys within the mining area is rapid, making water accumulation difficult. However, due to the shallow burial of the coal seams, the overlying aquitard is mainly the Neogene Psiponic Jingle Formation laterite aquitard, intermittently exposed in the gully and watershed areas; in addition, there are thin mudstone aquitards between the coal seams. Therefore, surface water has a certain impact on mine water inflow, especially in areas with well-developed fractures such as the bottom of surface valleys and coal seam outcrops. Surface floods may flow into the mine along these fractures, threatening the safety of the working face and the mine.

[0162] Distribution of surface rivers.

[0163] According to the "Hidden Disaster Investigation Report of TMMK Coal Mine of a Certain Energy Company" submitted by the Coalfield Geological Planning and Exploration Institute of a certain province in December 2023, based on the mining plan for the next five years, the mine will open 3... -3 5 -2 The calculation of the height of the water-conducting fracture zone in the coal seam is detailed in Table 2.

[0164] Table 2 Calculation Table for Height of Two Zones on the Working Face

[0165]

[0166] Mining subsidence situation.

[0167] In 2023, TMMK Coal Mine adopted a comprehensive method of "DOM image interpretation + field investigation and verification" to conduct a goaf subsidence survey in the entire area. According to this ground survey, it was found that the manifestation of goaf subsidence in the area was mainly ground cracks, with no obvious subsidence pits. The development and distribution of ground cracks were mainly concentrated above the coal mining faces of the fourth and eighth panels of the first mining area.

[0168] (3) Water accumulation in old vacant areas.

[0169] ①The old kiln is empty and waterlogged.

[0170] The accumulated water volume in the old kiln caverns is 1,966,769 m³. 3 The extent of the water accumulation is now clear.

[0171] (a) 3 -3 Information on the goaf and water accumulation in old coal mines.

[0172] Based on previous geological surveys, downhole directional transient electromagnetic data, and 3D seismic survey data, Laoyao 3... -3 There are two main coal seam goaf areas, one in the southern part of the third panel area. -3 Coal mine goaf areas and eastern Liupanshan 3 -3 Old coal mine goaf. Southern Sanpan 3 -3 The area of ​​the old coal mine goaf is 3,931,253 m². 2 Excluding standing water; 3 in the eastern part of Liupan District -3 The area of ​​the old coal mine goaf is 1,016,723 m². 2 The water volume is 27,959 m³. 3 .

[0173] (b) 5 -2上 Information on the goaf and water accumulation in old coal mines.

[0174] Based on previous geological surveys, borehole data, transient electromagnetic data, and 3D seismic data, the previous 5 -2上 The old coal seam goaf areas are mainly distributed in the southern part of the Sanpan District and the eastern part of the Liupan District.

[0175] (c)5 -2 Coal mining goaf and water accumulation conditions.

[0176] Based on the old mine survey and advanced drilling data provided by the mine, combined with geological exploration and transient electromagnetic data, the previous 5 -2 The old coal mine goaf areas are mainly distributed in the southern part of the Fourth and Eighth Panels, 5 -2 The area of ​​the old coal mine goaf is 7,992,052 square meters, and the volume of water accumulated in the goaf is 1,585,230 cubic meters. 3 The formation of the goaf was caused not only by the mining of old mine shafts within the original mine, but also by the illegal mining activities of two coal mines, Mine 1 and Mine 2, located to the south of the mine.

[0177] ②Water accumulation in the goaf of this mine.

[0178] Sanpan District Mining 5 -2上 The coal seam was mined from nine working faces, resulting in nine goaf areas with a total area of ​​7.7210 km². 2 The mined-out area has accumulated 251,984 m³ of water.

[0179] Mining in Sipan District 5 -2The coal seam was mined from five working faces, resulting in five goaf areas with a total area of ​​4.3568 km². 2 The mined-out area has accumulated 143,109 m³ of water.

[0180] 5 mining operations in Bapan District -2 The coal seam was mined from 14 working faces, resulting in 14 goaf areas with a total area of ​​9.9946 km². 2 The goaf accumulated 258,050 m³ of water. In the north wing, seven working faces were mined, resulting in seven goaf areas with a total area of ​​6.0267 km². 2 The goaf accumulated 157,000 m³ of water; the south wing completed the mining of 7 working faces, forming 7 goaf areas with a total area of ​​3.9679 km². 2 The mined-out area has accumulated 101,050 m³ of water. 3 .

[0181] Currently, 28 goaf areas have formed in the northern and southern wings of the eighth mining area and in the third and fourth mining areas within the first mining area, with a total area of ​​22.0724 km². 2 The water accumulation in the goaf area is 653,143 m³. A summary table of water accumulation in the goaf area of ​​the mining area is shown in Table 3, and a table of water accumulation in the mining area and surrounding goaf areas is shown in Table 4.

[0182] Table 3. Overview of Water Accumulation in Mining Goaf Areas

[0183]

[0184] Table 4. Water accumulation in the mining area and surrounding goaf areas

[0185]

[0186] First mining area 5 -2上 After coal seam mining, the accumulated water in the goaf of each working face will threaten the lower 5. -2 Production at each coal face, when the 5 below -2 When a coal face collapses and the roof's water-conducting cracks connect to water accumulated in the overlying goaf, the goaf water can overflow, causing water damage. Besides the overlying goaf water, water accumulated in the same layer of goaf also serves as a water source. -2上 Coal and 5 -2 Water accumulation in the goaf between different working faces can affect the production of adjacent working faces at lower positions. When the water volume is large and the water pressure is high, it may breach the protective coal pillars between the working faces, posing a certain threat to mine safety.

[0187] (4) Aquifer water.

[0188] (4.1) Quaternary diving.

[0189] The TMMK mining area is located in the Loess Plateau region of northern Shaanxi, characterized by uneven terrain and strong topographical dissection. In the valleys, the bedrock has been eroded and covered by alluvial and diluvial deposits, forming unconfined aquifers. When the mining face is close to a river valley, the water-conducting fracture zone in the roof of the working face may connect to the unconfined aquifer, or the water-conducting fracture zone in the roof of the working face may connect to a "skylight" that directly contacts the unconfined aquifer, allowing the unconfined water to be introduced into the working face.

[0190] (4.2) Burning water.

[0191] The phenomenon of igneous rock formation is relatively common in mines. Generally, the outcrops of coal seams with igneous rock formation are located at a relatively high position, above the erosion base level, because 5 -2上 Coal and 5 -2 When there is no coal outcrop in the production area, if there are other coal seams with igneous rock above the mining face, and if the water-conducting fracture zone in the roof is connected to it, the igneous rock water may be introduced into the working face to form a water inrush, but the amount of water inrush is generally small.

[0192] (4.3) Water in the roof of the coal seam.

[0193] The main coal seam currently mined at TMMK Coal Mine is Yan'an Formation 5. -2上 Coal and 5 -2 The coal seam and the confined aquifers of the Yan'an Formation above it are affected by water-conducting fractures that affect the confined aquifers above the roof after the coal seam is mined. Water from the roof will enter the working face along the fractures, which will have a certain impact on mining and production.

[0194] Based on the known results of borehole pumping tests and spring flow surveys, the unit yield of the Yan'an Formation aquifer is q < 0.01 L / s·m and Q < 1 L / s. Therefore, the water-bearing capacity of the Yan'an Formation sandstone aquifer is weak, and the roof yield under normal conditions during mining is generally small.

[0195] Due to the complexity of geological conditions, although 5 -2 The water volume in the sandstone aquifers of the coal roof is generally small, but the possibility of relatively large roof water inflows in certain special areas cannot be ruled out, such as areas with well-developed geological structures, near river valleys, and coal seam outcrops. Due to structural fractures, unconfined groundwater, and altered rocks, the roof water volume may be large after the working face is mined. Therefore, changes in roof water volume should be monitored during subsequent working face mining to prevent water inrush.

[0196] 2.3.3 Current status of water hazard prevention and control.

[0197] (1) Supplementary geological and hydrogeological exploration of the mine.

[0198] Since 2009, the mine has carried out a series of geological and hydrogeological supplementary explorations.

[0199] (2) Geophysical exploration work in the mine

[0200] Since 2002, the mine has conducted six ground geophysical exploration operations.

[0201] (3) Downhole exploration work

[0202] ① Downhole geophysical exploration work.

[0203] (a) Transient electromagnetic detection at a fully mechanized mining face.

[0204] After the longwall mining face is delineated, the YCS2000A mine transient electromagnetic instrument is used to promptly detect water hazards within a 120m range above the roof of the working face in the return roadway, haulage roadway, and cut-off sections. This provides technical support for evaluating the safety of the longwall mining face. During detection, the equipment is placed on the roadway floor, with measuring points spaced 10m apart, and each measuring point is positioned at a 30° angle upwards for detection. The transient electromagnetic instrument allows for the delineation of water-rich areas at different horizontal levels, providing a direct and accurate determination of the location and morphology of water-rich areas within a 120m range above the roof of the working face, and a preliminary evaluation of their water-richness. This provides technical support for subsequent water exploration and drainage work and for evaluating the safety of the longwall mining face.

[0205] (b) Advanced transient electromagnetic detection at the tunneling face.

[0206] Every 80m of tunnel excavation, a forward transient electromagnetic detection is conducted, capable of detecting along the bedding plane and in the direction above the working face, with a detection range of up to 100m for water-rich anomaly zones. This enables the prediction and forecasting of water-rich areas and provides technical support for subsequent water exploration and drainage work and safety assessments.

[0207] ② Downhole drilling work.

[0208] During the production process, the principles of "prediction and forecasting, investigation of any doubts, exploration before excavation, and treatment before mining" were followed in carrying out water exploration and release work. The details of the water exploration and release work are as follows:

[0209] From July to August 2018, drilling and drainage were carried out on the 45203 working face, including the two roadways and cut-outs, to drain the water accumulated in the goaf of the 35104 working face overlying the 45203 working face. A total of 27 water exploration and drainage boreholes were drilled underground, with a total drilling footage of 1152m and a total water drainage volume of 54540m³. 3 .

[0210] From August 2018 to July 2020, the Dongfeng Joint-venture Coal Mine in TMMK Township, above the 35109 working face, was inspected. -3 Water was drained from the old coalfield. This operation involved drilling 46 boreholes for water exploration and drainage, with a total drilling footage of 5311m and a total water drainage volume of 12000m³. 3 .

[0211] From April to July 2021, the mine conducted drilling and drainage operations on the two roadways and cut-outs of the 45204 working face to drain the water accumulated in the goaf of the 35105 working face above the 45204 working face. A total of 41 water exploration and drainage boreholes were drilled underground, with a total drilling footage of 837.5m and a total water drainage volume of 17,600m³. 3 .

[0212] In response to the current key issues of water prevention and control in the mine, namely water accumulation in old workings and goaf areas, the coal mine collected data from neighboring coal mines and carried out systematic investigation and exploration work.

[0213] TMMK Coal Mine compiled (from January 2017 to June 2022) water exploration and drainage ledgers for the third, fourth, and eighth mining and tunneling faces, monthly reports on water exploration and drainage boreholes, advance drilling ledgers, goaf water accumulation drilling ledgers, and mine water inflow observation and record ledgers, summarizing its experience in water exploration and drainage.

[0214] (4) Various geological reports.

[0215] Since 2010, the mine has compiled various geological reports.

[0216] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention and within the spirit and principles of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method for retrieving large-scale RAG models with mixed text and image output, characterized in that, The method includes the following steps: S1: Preprocess the original document corpus to generate a corpus tree, and process the graph and table corpus to generate key-value pairs and store them in the database; S2 generates word vectors for the question-and-answer statements and searches them against the corpus tree to find the corpus with the highest relevance coefficient; S3 concatenates the most relevant corpus into a prompt word and sends it to the text-based large language model. S4, the large text model outputs relevant results; S5, based on the output of the large text language model, retrieves and replaces graph and table key information in the results to generate the final question-and-answer results; In step S2, the retrieval with the corpus tree includes performing RAG retrieval on the question-and-answer statements for which word vectors have been generated. The specific steps are as follows: S201, RAG includes Retriever and Generator. Retriever is used to locate the corresponding document, and Generator is used to generate the corresponding answer. S202, perform word vector retrieval in the corpus tree; if the root node does not have a word vector equal to the word vector of the search term, continue traversing the child nodes. If none of the child nodes contain the word vector of the search term, this corpus is unrelated to the search term, and the remaining corpus is processed in turn; if all corpus is unrelated to the search term, skip directly to step S3; if the content in the corpus tree contains the word vector of the search term, traverse all nodes in turn to find the corpus with the highest relevance; first compare the root node. If the word vector of the search term matches the word vector of the root node with a high degree of matching, extract the root node and all its child nodes for use in step S3; if the matching degree is low, continue traversing its child nodes; the formula for calculating the word vector matching degree is as follows: ； In the formula, These are two vectors to be compared. For cosine similarity, Let be the magnitude of the vector; S203, Sort the word vector segments retrieved in step S202 according to their relevance, and then take them in sequence. The most relevant word vector segment The value is set to represent the number of most relevant corpora retrieved; then the word vectors are restored to the corresponding text corpora to generate the final search results; In step S202, the method for calculating the matching degree between the word vector of the search term and the word vector of the root node includes: S2021, There are four word vectors defined; S2022. If the number of word vectors in the root node is less than four, then perform recursive operation directly; if the number of word vectors in the root node is greater than or equal to four, then use the improved Gaussian mixture clustering algorithm to perform clustering and obtain the final four clustered word vectors. S2023. If the word vector of the search term is less than the number of root node vectors, then calculate which Gaussian distribution the word vector of the search term belongs to, as obtained in step S2022; the Gaussian distribution is: ； In the formula, The maximum value, The coefficients of the Gaussian distribution function are... It is a Gaussian mixture distribution function; The number of Gaussian distributions From 1 to The number of Gaussian distributions ( ) represents the maximum value; S2024, the maximum value is obtained by calculating sequentially. indivual The relevance value between the retrieved word vector and the corpus word vector is obtained by adding them together, and the expression is: ； In the formula, This is the cumulative value of the correlation; In step S2022, the improved Gaussian mixture clustering algorithm includes: S20221, to be clustered as Given a Gaussian distribution, set the Gaussian distribution function as follows: ； In the formula, It is a Gaussian mixture distribution. The function is a Gaussian distribution. For the parameters of the Gaussian distribution, , The mean, For variance; The category is a Gaussian distribution, with values ​​ranging from 0 to... ; Gaussian distribution function The expression is: ； S20222, The likelihood function of the improved Gaussian mixture clustering is: ； In the formula, Let be the likelihood function. For the number of samples, All are hyperparameter coefficients. For weight regularization, The first in the Gaussian mixture distribution The probability distribution of the Gaussian components. For divergence calculation; Divergence calculation The expression is: ； In the formula, The formula for calculating KL divergence is used to calculate the distribution. and The divergence; Set initial values ,in, The coefficient is a real number. for The mean of a Gaussian distribution function, They are respectively The variance of a Gaussian distribution function; S20223, Calculation: ； In the formula, This is the proportionality coefficient, with a value ranging from 0 to 1; For the first The coefficients of a Gaussian distribution function, It is a Gaussian distribution function; S20224, Update parameters: ； ； ； In the formula, For the updated number The iterative value of the mean of a Gaussian distribution function. For the updated number The iterative values ​​of the variance of a Gaussian distribution function; For the updated number The iterative values ​​of the coefficients of the Gaussian distribution function. This represents the probability distribution of the nth Gaussian component. The category is Gaussian distribution. This is the formula for calculating differentials. This is the divergence calculation formula from the previous step; S20225. Iterate through steps S20221-S20224 until convergence.

2. The large-scale RAG retrieval method with mixed text and image output according to claim 1, characterized in that, Step S1 specifically includes: S101, construct the corpus tree structure based on the directory; extract the title of the document as the root node of the corpus tree; recursively construct the child nodes of the corpus tree, the first level child node is the next level title and the adjacent non-title corpus segment, the second level title is the next level title and the adjacent non-title corpus segment, and so on, recursively traversing to construct the entire corpus tree. S102, extract the figures and tables from the document, and perform dictionary hash mapping between table names and table content, and figure names and image content, and store them in the database. At the same time, the table names and figure names are unique. S103: For the plain text data in the document, perform word vector processing to generate word vector representations, and attach all word vectors of each node to the corresponding tree node; use the Chinese word segmentation package Jieba to perform word segmentation on the entire text document; generate word vectors for the segmented word data, and use the FastText algorithm to encode the segmented words into vectors.

3. The large-model RAG retrieval method with mixed text and image output according to claim 2, characterized in that, In step S102, the dictionary hash mapping process is as follows: S1021, For tabular data, extract all text and data information from the table and connect them in the order of rows; use a hash function to calculate the hash value of the text, connect the hash value with the table name as the key, and use the chart content as the value to store it in the database; S1022, For image data, first, the image is reduced to 8*8 pixels, then the color image is converted to grayscale, the processed image is subjected to Discrete Cosine Transform (DCT), the average pixel value of the transformed image is calculated, the value of each pixel is compared with the average value, if it is greater than the average value, it is 1, and if it is less than the average value, it is 0, resulting in a 64-bit sequence. The 64-bit sequence is input into a hash function to calculate the hash value, and the hash value is concatenated with the image name as the key and the image content as the value, and stored in the database.

4. The large-model RAG retrieval method with mixed text and image output according to claim 2, characterized in that, Step S103 specifically includes: S1031. Use the Chinese word segmentation package Jieba to segment the entire text document; S1032. Generate word vectors from the segmented word data. The word vectors are generated using the FastText algorithm to encode the segmented words.

5. The large-scale RAG retrieval method with mixed text and image output according to claim 1, characterized in that, In step S2, word vectors are generated for the question-and-answer statements, including: segmenting the input question-and-answer statements into search terms and vectorizing them; using the Chinese word segmentation package jieba and the Fasttext word vectorization algorithm to convert the search question-and-answer statements into word vector representations; generating multiple word vectors; preprocessing the word vectors of the search statements by deleting prepositions and interrogative words to obtain the word vectors of the search terms.

6. The large-model RAG retrieval method with mixed text and image output according to claim 1, characterized in that, In step S3, the most relevant corpus is concatenated into a prompt word and sent to the text-to-speech model, including: Based on the RAG retrieval results in step S2, the data is concatenated into a prompt and sent to the text-to-speech model. The corpus tree from step S1 is then processed by the text-to-speech model to obtain corpus nodes, where graphs and tables are assembled into the prompt in the form of keywords. In step S5, based on the output of the large text language model, the graph and table key information in the replacement results is retrieved to generate the final question and answer results. This includes processing the question and answer results, firstly searching whether graph information appears in the question and answer results, and if it does, it is the corresponding key of the graph stored in the data in step S1. The corresponding graph is retrieved according to the key name, and then the final output answer is generated.

7. A large-scale RAG retrieval system with mixed text and image output, characterized in that, The system implements the large-model RAG retrieval method with mixed text and image output as described in any one of claims 1-6, and the system includes: The corpus tree generation module is used to preprocess the original document corpus, generate a corpus tree, and process the corpus of graphs and tables to generate key-value pairs and store them in the database. The corpus tree retrieval module is used to generate word vectors for question-and-answer statements and search them against the corpus tree to find the most relevant corpus. The Prompt concatenation module is used to concatenate the most relevant corpus into a prompt, which is then sent to the text-to-speech model. The relevant results output module is used by the question-answering model to output relevant results; The final question-and-answer result generation module is used to retrieve and replace the key information of graphs and tables in the results based on the output of the text-to-speech model, and generate the final question-and-answer results.

Images