Construction scheme multi-modal retrieval enhancement generation method and system based on large model

Abstract

According to the construction scheme multi-modal retrieval enhancement generation method and system based on the large model, intelligent scheme writing support is provided for constructors through a natural language processing technology by means of mass building industry data and knowledge, and the construction efficiency is improved. Comprising but not limited to the steps of assisting constructors in compiling a construction scheme, providing real-time answers to problems possibly encountered in the construction process, providing optimization suggestions according to project characteristics and the like; through the application of the large model, the working efficiency of constructors can be greatly improved, the waste of human resources is reduced, and the accuracy and feasibility of a construction scheme are ensured. Besides, the model can be continuously learned and updated, and the ability of assisting in compiling question answering is continuously optimized along with the accumulation of project experience, so that the model is more widely applied to each link in the building industry in the future.

Description

Technical Field

[0001] The present invention belongs to the field of computer artificial intelligence, and in particular relates to a multi-modal retrieval enhancement generation method for construction schemes based on a large model. Background Art

[0002] With the rapid development of my country's economy, the scale and complexity of the construction industry are constantly increasing, and the preparation and technical application of construction plans have become more complicated. The preparation of traditional construction plans mainly relies on the experience and professional knowledge of engineers and construction personnel, but this method is not only time-consuming and labor-intensive, but also prone to errors, especially in large-scale and complex engineering projects. In addition, construction plans often contain a large amount of text and image information, and traditional tools are difficult to effectively process these multimodal data, resulting in the efficiency and accuracy of plan preparation being difficult to guarantee. In recent years, the emergence of multimodal large models has improved the technical level of various industries. However, for the construction industry, large models often encounter situations where the answers of the models are inconsistent with industry facts. Although the retrieval enhancement generation (RAG) using external knowledge can improve the accuracy of the model, the limited retrieval context of the model may not cover all the necessary information, and excessive retrieval may introduce irrelevant and inaccurate references, thereby interfering with the generation of model answers. In order to solve the above problems, the present invention designs a new multimodal retrieval enhancement generation method for construction plans based on large models. Summary of the invention

[0003] In view of the deficiencies in the prior art, the present invention provides a large-model-based multimodal retrieval enhanced generation method and system for construction plans, which solves the problems that the existing multimodal RAG is inefficient in processing documents containing image and text relationships, and that during RAG retrieval, the retrieved context may not be sufficient to cover all necessary information, and too much retrieval may introduce irrelevant and inaccurate references, interfering with model generation.

[0004] The present invention achieves the above technical objectives through the following technical means.

[0005] A multi-modal retrieval and enhanced generation method for construction schemes based on a large model includes the following steps:

[0006] Step 1: Preprocess the construction plan, standardize text format, word segmentation and image processing, and ensure data consistency and quality;

[0007] Step 2: For the construction plan text page containing only text and tables, first extract the text information and convert the text in the table into editable text data. Then divide the extracted text into blocks, and then vectorize each text block into a numerical vector that can be understood by the machine learning model and finally store it in the text vector database.

[0008] Step 3: For the construction plan text page containing pictures, use rasterization technology to convert it into an image, perform Base64 encoding on the image, generate a summary for the encoded image to provide a content description, attach the summary to the corresponding image, and use the method in step 2 to generate text blocks and vectorize them. Finally, a complete vector representation of the image text page is formed, and the processed data is stored in the vector database;

[0009] Step 4: Improve the accuracy of large model answer generation through risk control solutions.

[0010] Furthermore, the specific process of step 1 is as follows:

[0011] Step 1.1: First, collect various construction plan documents, and use the python-docx library or PyPDF2 library to convert all construction plan documents into a standard text format. During the conversion process, clean up irrelevant elements including headers, footers, watermarks, and page numbers; according to the MuRAG model process in the document, process the text part and image part of the document as a whole;

[0012] Step 1.2: Use the regular expression library to clean the converted text, remove noise characters, extra spaces and format anomalies, and process the text and images in blocks so that the data can be used in the subsequent semantic retrieval and generation stages;

[0013] Step 1.3: For the image part in the construction plan, only Pillow or OpenCV is used to logically associate the image with the corresponding text part. The image file remains in its original state and establishes a basic association with the text part of the document. The MuRAG model framework converts the document page into an image to avoid processing the image separately.

[0014] Step 1.4: Use natural language processing tools to preliminarily annotate the logical structure of the document, including chapters, paragraphs, and titles, including identifying and marking different logical parts in the document, providing a structured basis for subsequent content segmentation and processing; generate embedding vectors to vectorize and store text blocks and image blocks;

[0015] Step 1.5: Perform consistency check on the processed text and images, store the cleaned and annotated documents and images in MongoDB or AWS S3 file system; generate embeddings for each image and text block to build a semantic knowledge base.

[0016] Furthermore, the specific process of step 2 is as follows:

[0017] Step 2.1: For the construction plan text preprocessed in step 1, formulate a block division strategy based on the content theme and logical structure. i, using sentence and paragraph segmentation in natural language processing technology; automatically identifying sentence boundaries and paragraph structures of texts through SpaCy, and analyzing the logical relationships of texts, dividing long texts into several independent blocks, setting the i-th text page P i Contains multiple text blocks, which are represented as follows:

[0018]

[0019] Among them, m i Indicates the number of text blocks in page i, [Text-Chunk] i,j Represents the jth text block in the i-th page;

[0020] Step 2.2: After the text is segmented, use the pre-trained Transformer model to vectorize each text block and generate its corresponding semantic vector V i,j :

[0021] V i,j =Transformer([Text] i,j )

[0022] The vector representation of each text page is the set of vectors of all text blocks in that page:

[0023]

[0024] Step 2.3: Use FAISS to store the processed text vectors in the vector database. The data in the vector database will be used for subsequent search and generation tasks.

[0025] Furthermore, the specific process of step 3 is as follows:

[0026] Step 3.1: Generate text blocks and vectorize them for the text part, and use rasterization technology to convert the construction plan text page containing the picture into a vector. i Converted to an image that preserves the spatial relationship of all elements on the page, in the following form:

[0027] I(θ=P i )=[Pgimg] i

[0028] Among them, [Pgimg] i is the image representation of page i, and I(θ) image is the conversion method;

[0029] Step 3.2: Use Base64 encoding technology to convert the image into Base64 encoding format. The generated encoded image is represented as follows:

[0030] E n(θ=[Pgimg] i ) → base64-encoded

[0031] Among them, E n A function representing the encoding process;

[0032] Step 3.3: Use automatic tools to analyze the encoded image content and generate a text summary that describes the key information in the image, as shown below:

[0033] S n (θ=E n ([Pgimg] i ))=ImageSummary

[0034] Among them, S n represents the function that generates the summary, E n ([Pgimg] i ) means encoding the image and finally generating ImageSummary, i.e., image summary;

[0035] Step 3.4: Utilize the JSON data structure to attach the image summary to the image and generate a combined representation of the entire text page:

[0036] A t (θ=S n (E n ( [ Pgimg ]i )))=ImagewithSummary

[0037] Among them, A t A function representing an additional summary, S n (E n ([Pgimg] i )) represents the image after encoding and generating summary, and finally obtains the image representation with summary;

[0038] Step 3.5: Use the pre-trained image embedding model to generate a vector representation of the image, including the encoded image and the attached summary. i :

[0039] I i =ImageModel ( A t (θ=S n (E n ([Pgimg] i ))))

[0040] Among them, A t (θ=S n (En ([Pgimg] i ))) represents the image after encoding, extracting summary and association;

[0041] Finally, the vector representation of the text page containing the image is the image vector I i And all text block vectors V i,j Combination of:

[0042]

[0043] Store it in the vector database.

[0044] Furthermore, the specific process of step 4 is as follows:

[0045] Step 4.1: The Llama3-8b large model performs vector similarity search on the text and image vector database through user query, using various algorithms including dot product, cosine similarity, and Manhattan distance; by calculating the similarity between the query vector and the vectors stored in the database, multiple relevant documents are retrieved from the database;

[0046] Step 4.2: Calculate the ratio of the number of correct answers generated by the model to the total number of answers, and then calculate a value called p k1 and p k2 The two probability values ​​are calculated and the minimum value of the two probabilities is taken. The Bonferroni correction method is used to compare the results of multiple hypothesis tests under different values ​​of the number of schemes k to determine the k value that brings the minimum factual risk:

[0047] p k1 =exp(-nh1(FR(k)∧α,α))

[0048]

[0049] In the formula, p k1 is a probability value that evaluates the factual risk FR(k) of the model when k retrieval contexts are used through the Kullback-Leibler divergence h1, n represents the number of samples in the training or validation dataset used to evaluate the performance of the model, h1 represents the Kullback-Leibler divergence, nh1 represents a weighted measure of the difference between the factual risk FR(k) of the model given k retrieval contexts and the preset risk upper limit α, α represents the risk upper limit, p k2 represents the probability that the actual number of errors does not exceed the expected number of errors at a given risk level, e represents the base of the natural logarithm, P represents the probability that the value observed in a given binomial distribution is less than or equal to nFR(k), and Bin(n,α) represents the binomial distribution with parameters n and α;

[0050] Step 4.3: After determining the k value, the top-k documents and user queries are further provided to the multimodal large model for processing and generating the answers required by the user.

[0051] A system for implementing the above-mentioned large-model-based multi-modal retrieval and enhanced generation method for construction plans, comprising:

[0052] The preprocessing module is used to process the text format of the construction plan, remove noise characters, redundant spaces and format anomalies, process text and images in blocks, logically associate images with corresponding text parts, and preliminarily mark the logical structures of the document such as chapters, paragraphs, and titles. By generating embedded vectors, the text blocks and image blocks are vectorized and stored, and the consistency check of text and images is performed;

[0053] The text and table document processing module extracts text information from construction plans that only contain text and tables, converts the text in the table into editable text data, divides the extracted text into blocks, vectorizes each text block, converts it into a numerical vector that can be understood by the machine learning model, and stores the numerical vector in the text vector database;

[0054] Image document processing module, for construction plans containing pictures, uses rasterization technology to convert the text page of the construction plan containing pictures into an image, and encodes the image in Base64 for LLM processing, generates a summary for the encoded image to provide a content description, attaches the summary to the corresponding image, generates text blocks and vectorizes it in terms of text, and finally forms a complete vector representation of the image text page, and stores the processed data in the vector database;

[0055] The risk control module improves the accuracy of answer generation of large models through risk control solutions.

[0056] The present invention has the following beneficial effects:

[0057] The present invention can utilize a large amount of construction industry data and knowledge, and provide intelligent solution writing support for construction personnel through natural language processing technology, including but not limited to: assisting construction personnel in writing construction plans, providing real-time answers to problems that may be encountered during the construction process, and providing optimization suggestions based on project characteristics; through the application of such a large model, construction personnel can greatly improve work efficiency, reduce waste of human resources, and ensure the accuracy and feasibility of construction plans. In addition, the model can also continuously learn and update, and with the accumulation of project experience, it continuously optimizes its ability to assist in writing problem answers, so that it can be more widely used in various links in the construction industry in the future. Through the present invention, construction personnel can retrieve and generate construction plans more efficiently, improve decision-making efficiency, and ensure the accuracy and feasibility of the plan. The present invention solves the problems that the existing multimodal RAG is inefficient in processing documents containing image and text relationships, and that when searching for RAG, the retrieved context may not be sufficient to cover all necessary information, and too much search may introduce irrelevant and inaccurate references, interfering with the generation of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a flow chart of the multi-modal retrieval and enhanced generation method of construction plans based on a large model described in the present invention. DETAILED DESCRIPTION

[0059] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments, but the protection scope of the present invention is not limited thereto.

[0060] Reference Figure 1 The multi-modal retrieval and enhanced generation method of the construction scheme based on the large model of the present invention comprises the following steps:

[0061] Step 1: Preprocess the construction plan, standardize text format, word segmentation and image processing, and ensure data consistency and quality;

[0062] Step 1.1: First, collect various types of construction plan documents, import the construction plan documents into the document processing pipeline, and use Python's python-docx library or PyPDF2 library to convert all construction plan documents into a standard text format (such as TXT or DOCX); during the conversion process, clean up irrelevant elements such as headers, footers, watermarks, and page numbers to ensure that the text content is neat and unified, laying the foundation for subsequent processing; according to the MuRAG model process in the document, the text part and the image part of the document are processed as a whole, and are not split into separate processing units to ensure that the relationship between the image and the text is preserved.

[0063] Step 1.2: Use re (regular expression library) to clean the converted text, use regular expressions to remove noise characters, extra spaces and format anomalies, and ensure that all texts meet unified standards in format; the document points out that text and images need to be processed in blocks so that the subsequent semantic retrieval and generation stages can effectively use these data.

[0064] Step 1.3: For the image part in the construction plan, no processing is performed on the image. Only Pillow or OpenCV is used to logically associate the image with the corresponding text part. The image file remains in its original state and establishes a basic association with the text part of the document, such as recording the position and reference relationship of the image in the document to ensure the correct positioning of the image in the document; the MuRAG model framework converts the document page into an image to avoid processing the image separately, thereby effectively maintaining the association between the image and the text.

[0065] Step 1.4: Use natural language processing tools (such as NLTK or SpaCy) to preliminarily annotate the logical structures of the document, such as chapters, paragraphs, and titles, including identifying and marking different logical parts in the document, such as overview, step-by-step instructions, bill of materials, etc., to provide a structured basis for subsequent content segmentation and processing; by generating embedding vectors, the text blocks and image blocks are vectorized and stored for efficient subsequent retrieval and generation.

[0066] Step 1.5: Perform consistency checks on the processed text and images to ensure that the structure and content of all documents are as expected. Store the cleaned and annotated documents and images in MongoDB or AWS S3 file systems to ensure that all data is normalized and ready for storage to ensure data integrity and consistency. Generate embeddings for each image and text block to build a semantic knowledge base to improve retrieval accuracy.

[0067] Step 2: For the construction plan text page containing only text and tables, first extract the text information and convert the text in the table into editable text data; next, divide the extracted text into blocks for subsequent management and retrieval; then, vectorize each text block and convert it into a numerical vector that can be understood by the machine learning model, and finally store these vectors in the text vector database; the specific process is as follows:

[0068] Step 2.1: For the construction plan text preprocessed in step 1, formulate a reasonable block division strategy based on the content theme and logical structure. i , using sentence and paragraph segmentation in natural language processing technology; automatically identifying sentence boundaries and paragraph structures of texts through SpaCy, and analyzing the logical relationships of texts, dividing long texts into several independent blocks, setting the i-th text page P iContains several text blocks, which are represented as follows:

[0069]

[0070] Among them, m i Indicates the number of text blocks in page i, [Text-Chunk] i,j Represents the jth text block in the i-th page;

[0071] Step 2.2: After the text is segmented, use the pre-trained Transformer model (such as BERT, RoBERTa, etc.) to vectorize each text block and generate its corresponding semantic vector V i,j :

[0072] V i,j =Transformer([Text] i,j )

[0073] The vector representation of each text page is represented as the set of vectors of all text blocks in that page:

[0074]

[0075] Step 2.3: Use FAISS to store the processed text vectors in an efficient vector database. The data in the vector database will be used for subsequent search and generation tasks.

[0076] Step 3: For the construction plan text page containing pictures, use rasterization technology to convert the construction plan text page containing pictures into images, encode these images in Base64 for LLM processing, generate summaries for the encoded images to provide content descriptions, attach the summaries to the corresponding images, and use the method in step 2 to generate text blocks and vectorize them. Finally, a complete vector representation of the image text page is formed, and these processed data are stored in the vector database. The specific process is as follows:

[0077] Step 3.1: For the text part, similar to step 2, generate text blocks and vectorize them. For the image part, use rasterization technology (such as PyMuPDF, PDFPlumber, etc.) to convert the construction plan text page containing the image into P i Converted to an image, the goal is to preserve the spatial relationship of all elements on the page, which helps to understand and interpret the document content, and its form is as follows:

[0078] I(θ=P i )=[Pgimg] i

[0079] Among them, [Pgimg] i is the image representation of page i, and I(θ) image is the conversion method;

[0080] Step 3.2: Convert the image to Base64 encoding format using Base64 encoding technology. This is a process of converting the binary data of an image into an ASCII string so that the encoded image can be stored and transmitted as a text string, making it easier to share between different systems and platforms. It also allows the image data to be processed by LLM in text form. The generated encoded image is represented as follows:

[0081] E n (θ=[Pgimg] i ) → base64-encoded

[0082] Among them, E n A function representing the encoding process;

[0083] Step 3.3: Use automatic tools such as artificial intelligence to analyze the encoded image content and generate a short text summary that describes the key information in the image. The text summary is concise and can capture the main features and content of the image, as shown below:

[0084] S n (θ=E n ([Pgimg] i ))=ImageSummary

[0085] Among them, S n represents the function that generates the summary, E n ([Pgimg] i ) means encoding the image and finally generating ImageSummary (image summary);

[0086] Step 3.4: Utilize the JSON data structure to attach the image summary to the image and generate a combined representation of the entire text page:

[0087] A t (θ=S n (E n ( [ Pgimg ]i )))=ImagewithSummary

[0088] Among them, A t A function representing an additional summary, S n (E n ([Pgimg] i )) represents the image after encoding and generating summary, and finally obtains the image representation with summary;

[0089] Step 3.5: Use the processed image data, including the encoded image and the attached summary, to generate a vector representation of the image using a pre-trained image embedding model (such as ResNet, CLIP, etc.) i :

[0090] I i =ImageModel ( A t (θ=S n (E n ([Pgimg] i ))))

[0091] Among them, A t (θ=S n (E n ([Pgimg] i ))) represents the image after encoding, extracting summary and association;

[0092] Finally, the vector representation of the text page containing the image is the image vector I i And all text block vectors V i,j Combination of:

[0093]

[0094] Stored in the vector database.

[0095] Step 4: Limited retrieval context may not cover all necessary information, and excessive retrieval may introduce irrelevant and inaccurate references, thereby interfering with model generation. The accuracy of large model answer generation is improved through risk control schemes. The specific process is as follows:

[0096] Step 4.1: The Llama3-8b large model performs vector similarity search on the text and image vector database through user query, using various algorithms including dot product, cosine similarity, and Manhattan distance; by calculating the similarity between the query vector and the vectors stored in the database, multiple relevant documents are retrieved from the database;

[0097] Step 4.2: By calculating the ratio of the number of correct answers generated by the Llama model to the total number of answers, we then calculate a value called p k1 and p k2 The two probability values ​​are calculated and the minimum value of the two probabilities is taken; finally, the Bonferroni correction method is used to compare the results of multiple hypothesis tests under different values ​​of the number of schemes k to determine the k value that brings the minimum factual risk;

[0098] p k1 =exp(-nh1(FR(k)∧α,α))

[0099]

[0100] In the formula, p k1 is a probability value, which evaluates the factual risk FR(k) of the model when using k retrieval contexts through the Kullback-Leibler divergence h1, n represents the number of samples in the training or validation dataset used to evaluate the performance of the model, h1 represents the Kullback-Leibler divergence, nh1 represents a weighted measure of the difference between the factual risk FR(k) of the model given k retrieval contexts and the preset risk upper limit α, α represents the risk upper limit, p k2 represents the probability that the actual number of errors does not exceed the expected number of errors at a given risk level, e represents the base of the natural logarithm, which is approximately 2.71828, P represents the probability, that is, the probability that the observed value is less than or equal to nFR(k) in a given binomial distribution, and Bin(n,α) represents the binomial distribution with parameters n and α.

[0101] Step 4.3: After determining the k value, the top-k documents and user queries are further provided to the multimodal large model for processing and generating the answers required by the user.

[0102] The present invention also provides a large-model-based construction plan multi-modal retrieval enhanced generation system, comprising:

[0103] The preprocessing module is used to process the text format of the construction plan, remove noise characters, redundant spaces and format anomalies, process text and images in blocks, logically associate images with corresponding text parts, and preliminarily mark the logical structures of the document such as chapters, paragraphs, and titles. By generating embedded vectors, the text blocks and image blocks are vectorized and stored, and the consistency check of text and images is performed;

[0104] The text and table document processing module extracts text information from construction plans that only contain text and tables, converts the text in the table into editable text data, divides the extracted text into blocks, vectorizes each text block, converts it into a numerical vector that can be understood by the machine learning model, and stores the numerical vector in the text vector database;

[0105] Image document processing module, for construction plans containing pictures, uses rasterization technology to convert the text page of the construction plan containing pictures into an image, and encodes the image in Base64 for LLM processing, generates a summary for the encoded image to provide a content description, attaches the summary to the corresponding image, generates text blocks and vectorizes it in terms of text, and finally forms a complete vector representation of the image text page, and stores the processed data in the vector database;

[0106] The risk control module improves the accuracy of answer generation of large models through risk control solutions.

[0107] The embodiments are preferred implementations of the present invention, but the present invention is not limited to the above-mentioned implementations. Any obvious improvements, substitutions or modifications that can be made by those skilled in the art without departing from the essential content of the present invention belong to the protection scope of the present invention.

Claims

1. A multi-modal retrieval and enhanced generation method for construction schemes based on a large model, characterized in that: The process includes the following: Step 1: Preprocess the construction plan, standardize text format, word segmentation and image processing, and ensure data consistency and quality; Step 2: For the construction plan text page containing only text and tables, first extract the text information and convert the text in the table into editable text data. Then divide the extracted text into blocks, and then vectorize each text block into a numerical vector that can be understood by the machine learning model and finally store it in the text vector database. Step 3: For the construction plan text page containing pictures, use rasterization technology to convert it into an image, perform Base64 encoding on the image, generate a summary for the encoded image to provide a content description, attach the summary to the corresponding image, and use the method in step 2 to generate text blocks and vectorize them. Finally, a complete vector representation of the image text page is formed, and the processed data is stored in the vector database; Step 4: Improve the accuracy of large model answer generation through risk control solutions.

2. The method for multimodal retrieval and enhanced generation of construction plans based on a large model according to claim 1 is characterized in that: The specific process of step 1 is as follows: Step 1.1: First, collect various construction plan documents, and use the python-docx library or PyPDF2 library to convert all construction plan documents into a standard text format. During the conversion process, clean up irrelevant elements including headers, footers, watermarks, and page numbers; according to the MuRAG model process in the document, process the text part and image part of the document as a whole; Step 1.2: Use the regular expression library to clean the converted text, remove noise characters, extra spaces and format anomalies, and process the text and images in blocks so that the data can be used in the subsequent semantic retrieval and generation stages; Step 1.3: For the image part in the construction plan, only Pillow or OpenCV is used to logically associate the image with the corresponding text part. The image file remains in its original state and establishes a basic association with the text part of the document. The MuRAG model framework converts the document page into an image to avoid processing the image separately. Step 1.4: Use natural language processing tools to preliminarily annotate the logical structure of the document, including chapters, paragraphs, and titles, including identifying and marking different logical parts in the document, providing a structured basis for subsequent content segmentation and processing; generate embedding vectors to vectorize and store text blocks and image blocks; Step 1.5: Perform consistency check on the processed text and images, store the cleaned and annotated documents and images in MongoDB or AWS S3 file system; generate embeddings for each image and text block to build a semantic knowledge base.

3. The method for multimodal retrieval and enhanced generation of construction plans based on a large model according to claim 1 is characterized in that: The specific process of step 2 is as follows: Step 2.1: For the construction plan text preprocessed in step 1, formulate a block division strategy based on the content theme and logical structure. i , using sentence and paragraph segmentation in natural language processing technology; automatically identifying sentence boundaries and paragraph structures of texts through SpaCy, and analyzing the logical relationships of texts, dividing long texts into several independent blocks, setting the i-th text page P i Contains multiple text blocks, which are represented as follows: Among them, m i Indicates the number of text blocks in page i, [Text-Chunk] i,j Represents the jth text block in the i-th page; Step 2.2: After the text is segmented, use the pre-trained Transformer model to vectorize each text block and generate its corresponding semantic vector V i,j : V i,j =Transformer([Text] i,j ) The vector representation of each text page is the set of vectors of all text blocks in that page: Step 2.3: Use FAISS to store the processed text vectors in the vector database. The data in the vector database will be used for subsequent search and generation tasks.

4. The method for multimodal retrieval and enhanced generation of construction plans based on a large model according to claim 1 is characterized in that: The specific process of step 3 is as follows: Step 3.1: Generate text blocks and vectorize them for the text part, and use rasterization technology to convert the construction plan text page containing the picture into a vector. i Converted to an image that preserves the spatial relationship of all elements on the page, in the following form: I(θ=P i )=[Pgimg] i Among them, [Pgimg] i is the image representation of page i, and I(θ) image is the conversion method; Step 3.2: Use Base64 encoding technology to convert the image into Base64 encoding format. The generated encoded image is represented as follows: IN n (θ=[Pgimg] i )→base64-encoded Among them, E n A function representing the encoding process; Step 3.3: Use automatic tools to analyze the encoded image content and generate a text summary that describes the key information in the image, as shown below: S n (θ6E n ([Pgimg] i ))mImageSummary Among them, S n represents the function that generates the summary, E n ([Pgimg] i ) means encoding the image and finally generating ImageSummary, i.e., image summary; Step 3.4: Utilize the JSON data structure to attach the image summary to the image and generate a combined representation of the entire text page: A t (θ=S n (E n ([Pgimg] i )))=ImagewithSummary Among them, A t A function representing an additional summary, S n (E n ([Pgimg] i )) represents the image after encoding and generating summary, and finally obtains the image representation with summary; Step 3.5: Use the pre-trained image embedding model to generate a vector representation of the image, including the encoded image and the attached summary. i : I i =ImageModel(A t (θ=S n (AND n ([Pgimg] i )))) Among them, A t (θ=S n (E n ([Pgimg] i ))) represents the image after encoding, extracting summary and association; Finally, the vector representation of the text page containing the image is the image vector I i And all text block vectors V i,j Combination of: Store it in the vector database.

5. The method for multimodal retrieval and enhanced generation of construction plans based on a large model according to claim 1 is characterized in that: The specific process of step 4 is as follows: Step 4.1: The Llama3-8b large model performs vector similarity search on the text and image vector database through user query, using various algorithms including dot product, cosine similarity, and Manhattan distance; by calculating the similarity between the query vector and the vectors stored in the database, multiple relevant documents are retrieved from the database; Step 4.2: Calculate the ratio of the number of correct answers generated by the model to the total number of answers, and then calculate a value called p k1 and p k2 The two probability values ​​are calculated and the minimum value of the two probabilities is taken. The Bonferroni correction method is used to compare the results of multiple hypothesis tests under different values ​​of the number of schemes k to determine the k value that brings the minimum factual risk: p k1 =exp(-nh1(FR(k)∧α,α)) In the formula, p k1 is a probability value that evaluates the factual risk FR(k) of the model when k retrieval contexts are used through the Kullback-Leibler divergence h1, n represents the number of samples in the training or validation dataset used to evaluate the performance of the model, h1 represents the Kullback-Leibler divergence, nh1 represents a weighted measure of the difference between the factual risk FR(k) of the model given k retrieval contexts and the preset risk upper limit α, α represents the risk upper limit, p k2 represents the probability that the actual number of errors does not exceed the expected number of errors at a given risk level, e represents the base of the natural logarithm, P represents the probability that the value observed in a given binomial distribution is less than or equal to nFR(k), and Bin(n,α) represents the binomial distribution with parameters n and α; Step 4.3: After determining the k value, the top-k documents and user queries are further provided to the multimodal large model for processing and generating the answers required by the user.

6. A system for implementing the large model-based multi-modal retrieval and enhanced generation method of construction schemes according to claim 1, characterized in that: include: The preprocessing module is used to process the text format of the construction plan, remove noise characters, redundant spaces and format anomalies, process text and images in blocks, logically associate images with corresponding text parts, and preliminarily mark the logical structures of the document such as chapters, paragraphs, and titles. By generating embedded vectors, the text blocks and image blocks are vectorized and stored, and the consistency check of text and images is performed; The text and table document processing module extracts text information from construction plans that only contain text and tables, converts the text in the table into editable text data, divides the extracted text into blocks, vectorizes each text block, converts it into a numerical vector that can be understood by the machine learning model, and stores the numerical vector in the text vector database; Image document processing module, for construction plans containing pictures, uses rasterization technology to convert the text page of the construction plan containing pictures into an image, and encodes the image in Base64 for LLM processing, generates a summary for the encoded image to provide a content description, attaches the summary to the corresponding image, generates text blocks and vectorizes it in terms of text, and finally forms a complete vector representation of the image text page, and stores the processed data in the vector database; The risk control module improves the accuracy of answer generation of large models through risk control solutions.

Images