A multimodal assisted precision manufacturing database interaction method based on large models

Abstract

The present invention belongs to the technical field of assisted precision manufacturing, and discloses a multimodal assisted precision manufacturing database interaction method based on a large model, comprising: obtaining monitoring videos and user queries of precision manufacturing production scenes; rewriting the user queries into new queries; performing schema retrieval in a precision manufacturing database according to the new queries to obtain a schema set; performing feature extraction on the monitoring videos and the new queries, and performing multimodal fusion on production scene video feature vectors, production scene audio feature vectors, and user query text feature vectors to obtain a fused feature vector; using the fused feature vector to search an external vector database to obtain a retrieval result document set; constructing an SQL statement to generate prompt words according to the new query and the retrieval result document set; inputting the SQL statement generation prompt words into the large model to generate an SQL statement; putting the SQL statement into the precision manufacturing database for execution, and outputting a final retrieval result.

Description

Technical Field

[0001] The present invention belongs to the technical field of assisted precision manufacturing, and in particular relates to a multimodal assisted precision manufacturing database interaction method based on a large model. Background Art

[0002] In the field of precision manufacturing, from component production to technological innovation, every process involves retrieving high-quality data from large amounts of data. Rapidly acquiring accurate, high-quality data has become a major challenge for technicians and production managers. Precision manufacturing refers to an advanced manufacturing method that produces and processes products with extremely high precision requirements. It is widely used in high-end industries such as aerospace, electronics, automotive, and industrial robotics. The core of precision manufacturing lies in strict process control, extremely low tolerances, and a high degree of assurance of product performance and quality. In modern precision manufacturing, various precision manufacturing technologies are constantly evolving, with the extensive use of highly automated and intelligent production equipment such as CNC machine tools, industrial robots, and complex sensor networks. The complexity and data intensity of products and production processes are rapidly increasing. The massive amount of data generated at every production stage, such as the operating status of production equipment, fault information, spatial relationships between equipment, and acoustic signals and abnormal noise during equipment operation, are key factors affecting product quality and production efficiency. Therefore, how to quickly retrieve accurate, high-quality data from databases, effectively manage, process, and analyze this data, and improve production efficiency and product quality has become a major challenge in improving precision manufacturing and maintaining competitiveness.

[0003] Against this backdrop, the need for large language models (LLMs) to interact with databases and quickly acquire high-quality data has become increasingly prominent. First, the data generated in precision manufacturing is diverse and complex, encompassing temperature, humidity, pressure, current, and noise in the production environment, equipment status and fault information during the production process, and equipment scheduling plans. Traditionally, this data requires a complex screening, organization, and interpretation process to effectively support production management and decision-making. Large language models, with their exceptional natural language processing capabilities, can convert users' natural language queries into SQL statements (i.e., NL2SQL), rapidly extracting target data directly from the database's massive data volume. This significantly reduces the time and difficulty of data acquisition and produces high-quality data, helping technicians and production managers efficiently complete tasks such as production line layout, process design, troubleshooting, and technological innovation.

[0004] In recent years, pre-trained large language models (such as BERT, T5, and GPT) have performed exceptionally well in NL2SQL tasks. These models offer excellent understanding of natural language and can be fine-tuned to adapt to SQL generation tasks. For example, IRNet, a BERT-based model, combines grammatical constraints and semantic parsing to parse natural language into an abstract syntax tree (AST) and then generate SQL queries. Another example is UnifiedSKG, a framework based on the T5 pre-trained language model. By unifying the input and output formats of tasks and sharing model weights and knowledge, it improves cross-task generalization and performance. It is designed to handle a variety of cross-task structured knowledge generation tasks. Its goal is to unify different forms of structured knowledge generation tasks (such as table question answering, NL2SQL, and knowledge graph question answering) within a single framework.

[0005] In addition, Retrieval-Augmented Generation (RAG) is beginning to be applied to natural language processing. By retrieving relevant information from external knowledge bases and assisting in the generation of more accurate text, the quality of the resulting data can be further improved. Under the RAG mechanism, when a user enters a complex query, the system first retrieves additional information related to the query, such as the database schema, table relationships, field explanations, and so on. This external information is integrated into the reasoning and generation process of the large model, helping the system better understand user intent and thus generate more accurate SQL statements. Therefore, adopting this mechanism can greatly enhance the large model's understanding of user intent, further improve data quality, and benefit subsequent production scheduling, product innovation, and other aspects, thereby enhancing the company's overall competitiveness.

[0006] Existing large-model-based NL2SQL systems typically use natural language descriptions in a single text modality to process and parse user input and generate corresponding SQL statements. These methods reduce the complexity and workload of database interactions to a certain extent, enabling technicians and production managers to leverage large language models to efficiently interact with the database via natural language and quickly acquire large amounts of data. However, relying solely on language-level interaction makes it difficult to convey the vast array of spatial structures and acoustic features present in precision manufacturing scenarios to large models. Due to the lack of a parsed production environment, large models cannot fully analyze users' data requirements for actual scenarios, making it difficult for existing methods to guarantee the quality of the resulting data.

[0007] At present, the main problems in database interaction with the help of large language models (LLM) in the field of precision manufacturing include the following aspects.

[0008] (1) The spatial structure and acoustic characteristics of the production environment and technical products are ignored. In the field of precision manufacturing, technicians and managers often face complex and ever-changing production environments and technical products. Simple language descriptions are difficult to convey to the large model the spatial structure and acoustic characteristics of the environment and products, as well as the user's data requirements for a specific part. Traditional NL2SQL systems are usually limited to processing a single text modality and rely solely on data in the text modality. Large models often have difficulty understanding the spatiotemporal characteristics of the objects described by users such as technicians and production managers, and are unable to fully analyze the correspondence between the user's data requirements and the target database. As a result, they are unable to fully meet the user's actual needs and may even misinterpret the user's true intentions and provide incorrect data feedback to the user.

[0009] (2) Fine-tuning large models in specialized fields is costly. Fine-tuning NL2SQL requires a large amount of well-labeled training data, which usually contains natural language queries and their corresponding SQL statements. In addition, the database may involve hundreds of tables, thousands of fields, complex multi-table associations, nested queries, and specialized technical terms. However, in real-world scenarios, the table structure, field names, table relationships, etc. of each database in various links of precision manufacturing may be different and highly complex. Existing data is very scarce, the data labeling process is very complex and expensive, and the technology in the specialized field related to the data is updated rapidly. To generate high-quality training data, technicians not only need to have an in-depth study of the database structure, SQL syntax, and industry terminology, but also need to have a comprehensive understanding of the latest technologies in the field. In addition, as the technology in the field is updated, the existing large model must be fine-tuned continuously, which greatly increases the cost and difficulty of building large models in specialized fields.

[0010] (3) Limitations of RAG retrieval. The application of RAG technology is still mainly focused on processing text modal data, and fails to fully integrate multimodal information such as spatial structure and sound signals. These multimodal data are key elements for equipment monitoring and fault diagnosis in precision manufacturing. RAG systems that cannot effectively integrate multimodal information are limited in performance in complex production environments and cannot fully meet the data retrieval and analysis needs of production managers and technicians for complex production processes and technical products. Summary of the Invention

[0011] The purpose of the present invention is to overcome the defects of the prior art and provide a large-model-based multimodal assisted precision manufacturing database interaction method. By fusing production scene video features, audio features and user query text features, and utilizing a multimodal fusion strategy based on an attention mechanism, the method directly uses a large model without fine-tuning, thereby saving the cost of large-model training while improving the accuracy and efficiency of SQL generation, achieving efficient database interaction, and quickly acquiring high-quality data.

[0012] The technical solution provided by the present invention is:

[0013] A multimodal assisted precision manufacturing database interaction method based on a large model, comprising:

[0014] Step 1: Obtain surveillance video of precision manufacturing production scenes and obtain user queries;

[0015] Step 2: rewrite the user query into a new query that complies with the large model language specification; perform a schema search in the precision manufacturing database based on the new query, obtain relevant data tables and fields, and obtain a schema set;

[0016] Step 3: Perform feature extraction on the surveillance video to obtain a production scene video feature vector and a production scene audio feature vector; perform feature extraction on the new query to obtain a user query text feature vector;

[0017] Step 4: Perform multimodal fusion on the production scene video feature vector, the production scene audio feature vector, and the user query text feature vector based on the attention mechanism to obtain a user query fusion feature vector;

[0018] Step 5: Search an external vector database using the user query fusion feature vector to obtain a search result document set;

[0019] Step 6: Construct an SQL statement to generate prompt words based on the new query, the search result document set, and the schema set;

[0020] Step 7: Input the SQL statement generation prompt word into the large model for reasoning to generate the SQL statement;

[0021] Step 8: Put the SQL statement into the precision manufacturing database for execution to generate search data as the search result.

[0022] Preferably, the multimodal assisted precision manufacturing database interaction method based on a large model further includes:

[0023] The user determines whether the search data meets the requirements. If so, the search data is output as the final search result; if not, the process returns to step 2 for a new round of data search.

[0024] Preferably, in step 2, rewriting the user query into a new query that complies with the large model language specification includes the following steps:

[0025] Step 1: Use the embedding model to convert the user query into a vector. Call an external database and retrieve query samples stored in the external vector database based on similarity retrieval rules. Use the retrieved query samples as reference examples. Input the reference examples and the user query into the big model, and input the query classification rules into the big model. Use the big model to classify the user query and obtain the category of the user query.

[0026] Step 2: Using the rewriting model, after decomposing the target task into one or more subtasks according to the user query category, the user query is converted into one or more rewritten subqueries to obtain the new query.

[0027] Preferably, there are four types of user queries, namely, explicit fact queries, implicit fact queries, explainable reasoning queries and hidden reasoning queries.

[0028] Preferably, in step 2, the method for obtaining the schema set includes the following steps:

[0029] Step a: breaking down the new query into words or phrases, so that the entire new query is divided into different semantic keywords;

[0030] Step b: using embedding technology and matching algorithm, matching the semantic keywords with tables and fields in the precision manufacturing database, and using the matching results as a schema set.

[0031] Preferably, in step b, the method for matching the semantic keywords with the tables and fields in the precision manufacturing database is:

[0032] Using embedding technology to convert the semantic keywords and the table names and field names in the precision manufacturing database into vectors;

[0033] Calculate the cosine similarity between the semantic keyword vector and the table and field vectors to obtain a similarity result set;

[0034] Set a similarity threshold and perform schema pruning based on the similarity threshold, retaining tables and fields above the threshold as matching results.

[0035] Preferably, in step 4, the method for obtaining the user query fusion feature vector is:

[0036] Inputting the production scene video feature vector, the production scene audio feature vector and the user query text feature vector into a multimodal fusion strategy model;

[0037] The multimodal fusion strategy model splices the production scene video feature vector and the production scene audio feature vector to obtain a spliced feature vector;

[0038] A multi-head cross attention mechanism is used to perform a weighted combination of the concatenated feature vector and the user query text feature vector to obtain a user query fusion feature vector.

[0039] Preferably, in step 5, the method for obtaining the search result document set is:

[0040] Using the user query fusion feature vector to search in an external vector database, calculating the correlation between the user query fusion feature vector and the document based on cosine similarity, screening out documents with a correlation greater than 0, and obtaining a recalled document set; using the RAG rearrangement model, calculating the rearrangement score of each recalled document in the recalled document set, and obtaining a rearrangement score set;

[0041] Set a re-ranking score threshold and retain the recalled documents above the threshold as the retrieval result document set.

[0042] Preferably, the external vector database adopts the Pinecone vector database.

[0043] The beneficial effects of the present invention are:

[0044] (1) The present invention introduces data in three modalities: video, audio, and text, and utilizes a multimodal fusion strategy based on an attention mechanism to comprehensively analyze the user's query text, the video features of various precision manufacturing scenarios, and the audio features. This multimodal fusion approach enables the system to more comprehensively understand the user's actual needs and production environment, significantly improving the quality of the resulting data.

[0045] (2) The present invention enables production technicians who are not familiar with the field of large models to efficiently manage production databases with the help of large models without fine-tuning, solving the problem that ordinary producers cannot use large models through professional operations; by combining multimodal data with an automated prompt construction process of an external knowledge base, producers do not need to master complex large model usage methods and prompt construction techniques, but can interact with the database through the large model to obtain high-quality data, thereby improving production efficiency and data management speed.

[0046] (3) The present invention does not require fine-tuning of large models for professional technical fields. By accurately understanding the user's intention and combining it with an external knowledge base, technicians and production managers can directly call the large model without fine-tuning to obtain high-quality data, saving a lot of cost for fine-tuning the large model.

[0047] (4) After the SQL is generated, the present invention provides a verification and feedback mechanism: if the data obtained by interacting with the database does not meet the requirements, it means that the generated SQL statement does not meet the standards, and the system will return it and re-enter the generation process; this cyclic process ensures the correctness and validity of the final output data based on the historical SQL generation results, and avoids the possible impact of erroneous SQL operations on database interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a flowchart of the large-model-based multimodal assisted precision manufacturing database interaction method described in the present invention.

[0049] Figure 2 This is a flowchart of the multimodal fusion strategy based on the attention mechanism described in the present invention. DETAILED DESCRIPTION

[0050] The present invention will be described in further detail below in conjunction with the accompanying drawings so that those skilled in the art can implement the invention with reference to the description.

[0051] like Figure 1 As shown, the present invention provides a multimodal assisted precision manufacturing database interaction method based on a large model, which uses video and detection equipment to obtain on-site monitoring videos in precision manufacturing production scenarios, and generates user queries based on the needs input by technicians; rewrites the user query to obtain a new query statement that meets the specifications of the multimodal fusion link; performs schema retrieval based on the new query statement to obtain a streamlined schema result set; performs text feature extraction on the new query to obtain a user query text feature vector; performs feature extraction with the on-site monitoring video to obtain a production scene video feature vector and an audio feature vector; uses a multimodal fusion strategy based on an attention mechanism to perform demand analysis based on the text feature vector, video feature vector, and audio feature vector to obtain a complete user demand description; performs a search based on the user demand description to obtain the result document set in an external knowledge base; combines the user demand description with the result document set to obtain a special prompt word for SQL generation; uses the prompt word to perform reasoning in a general large model, applies the output SQL statement to the schema result set for iterative verification, and obtains the final data result. The specific implementation process of the present invention is as follows.

[0052] S1. Obtain surveillance video of the user's target precision manufacturing production scene.

[0053] S2. Obtain a natural language query input by the user as a user query.

[0054] S3. Rewrite the user query into a new query that complies with the big model language specification; wherein the query rewriting process is divided into two stages: query classification and query reconstruction.

[0055] Preferably, the step of rewriting the user query (statement) input by the user includes:

[0056] Step S31: Use the embedding model to convert the user query into a vector. Call an external database and, based on similarity retrieval rules, retrieve query samples stored in the external vector database. These retrieved query samples serve as reference examples. The reference examples and the user query are input into the big model, along with the query classification rules. The big model is then used to classify the user query to obtain a category. User queries are categorized into four categories: explicit fact queries, implicit fact queries, explainable reasoning queries, and hidden reasoning queries.

[0057] Preferably, the external vector database adopts the Pinecone vector database.

[0058] Step S32: For queries of known categories, use the query extension rewriting model to decompose the target task into one or more subtasks based on its category and target task, and then convert the original user query into one or more rewritten subqueries to obtain a new user query;

[0059] S4. Based on the new query, a schema search is performed in the precision manufacturing database to obtain relevant data tables and fields, and a simplified schema set is obtained.

[0060] Preferably, the step of performing schema retrieval in the precision manufacturing database includes:

[0061] Step S41: Decompose the query into words or phrases through word segmentation and keyword extraction technology, so that the entire query is divided into different semantic keywords; wherein these semantic keywords involve key information such as entities, relationships or time ranges involved in the query.

[0062] Step S42: Using embedding technology and matching algorithm, the semantic keywords are matched with tables and fields in the database, and the matching results are used as a schema set.

[0063] The process of matching the semantic keywords with the tables and fields in the database is as follows:

[0064] Step S421: Using embedding technology to convert the semantic keywords and table names and field names in the precision manufacturing database into vector representations, and mapping the text into a shared vector space using a pre-trained language model;

[0065] Step S422: After vectorization is completed, the cosine similarity between the semantic keyword vector and the table and field vectors is calculated to obtain a similarity result set;

[0066] Step S423: Set a similarity threshold, perform schema pruning based on the similarity result set, and obtain a streamlined schema result set; wherein, tables and fields with similarity lower than the threshold will be excluded, and tables and fields with similarity higher than the threshold will be retained.

[0067] S5. Perform text feature extraction on the new query to obtain a user query text feature vector; perform feature extraction on the on-site monitoring video to obtain a production scene video feature vector and an audio feature vector.

[0068] Preferably, the specific steps of extracting features from the on-site surveillance video and the new query include:

[0069] Step S51: Use the pre-trained C3D video encoder to extract the spatial and temporal features of the video. The obtained video features are represented as a vector ;in, is the number of video frames, ( =1,2,…, ) is the Encoding of video frames;

[0070] Step S52: Use the pre-trained VGGish audio encoder to process the audio signal, first convert the audio into Mel-frequency cepstral coefficients, and then express the obtained audio features as a vector ;in, is the number of audio frames, ( =1,2,…, ) is the Encoding of audio frames.

[0071] Step S53: Use the pre-trained BERT language model to extract the embedded representation of the text, and the obtained text feature is represented as a vector ,in, is the number of words in the text, ( =1,2,…, ) is the The encoding of a token.

[0072] S6. The production scene video feature vector, the production scene audio feature vector and the user query text feature vector are identified by adopting a multimodal fusion strategy based on an attention mechanism, taking into account the multimodal features of text and audio and video to obtain a user query fusion feature vector.

[0073] like Figure 2 Preferably, the multimodal fusion strategy based on the attention mechanism includes:

[0074] Step S61: Design a multimodal fusion strategy model based on the cross-attention mechanism, using the text feature vector, audio feature vector, and video feature vector extracted in S5 as the three inputs of the model;

[0075] Step S62: Concatenate the video feature vector and the audio feature vector to obtain a new matrix :

[0076] ;

[0077] Step S63: Input the spliced features into the multi-head cross attention mechanism module to learn the relationship between the modalities and achieve the purpose of interactive fusion of text modality features, audio features and video modality features;

[0078] Step S631: Generate query, key, and value using the same set of weights. The attention mechanism will perform a weighted combination of these features to automatically learn the correlation between text, video, and audio features:

[0079] ;

[0080] in, 、 and It is the projection matrix obtained by optimizing the learning through back propagation during the training process.

[0081] Step S632: Query ,key Sum Through different linear transformations, they are projected onto different subspaces, representing different heads, and the dimension of each head is , for each head , calculate the query, key and value separately:

[0082] ;

[0083] ;

[0084] ;

[0085] in, 、 、 It's the head The projection matrix;

[0086] Step S633: For each header , using the query ,key Sum Calculate the attention score and weight the Output of the header:

[0087] ;

[0088] Step S634: splice the outputs of all heads and map the spliced vector back to the original dimension through a linear transformation matrix to obtain the final fusion feature vector ;

[0089] ;

[0090] ;

[0091] S7. Based on the user query fusion feature vector, an external vector database (Pinecone) is searched to obtain knowledge documents that are adapted to the scenario and user needs, and a result document set is obtained.

[0092] S8. Construct a complete SQL statement to generate prompt words based on the schema result set, the new query, and the result document set;

[0093] Preferably, the specific steps of searching the external knowledge base according to the user requirement description include:

[0094] Step S81: Use the fused user query fusion vector to search in an external vector database, calculate the correlation between the query vector and the document based on cosine similarity, filter out relevant documents, and obtain a recalled document set.

[0095] Step S82: Using the RAG rearrangement model, the rearrangement score of each document in the recalled document set is calculated through a graph neural network to obtain a rearrangement score set.

[0096] Step S83: Set a re-ranking score threshold, perform screening in the re-ranking score set, and obtain the result document set; wherein, documents with scores lower than the threshold will be excluded, and documents with scores higher than the threshold will be retained.

[0097] S9. Input the SQL statement generation prompt word into the general large model for reasoning to generate an SQL statement.

[0098] S10. Put the SQL statement into the precision manufacturing database for testing to check whether the generated data meets the user requirements. If the user meets the requirements, the data is output as the final result; if not, the non-compliant SQL corresponding to the data is returned and transferred to S3 for a new round of SQL statement generation and data retrieval.

[0099] The multimodal assisted precision manufacturing database interaction method based on a large model provided by the present invention obtains the spatial characteristics and sound signals of the production process, equipment and facilities, etc. by retrieving the monitoring video of the production process in the precision manufacturing field, and combines the description of the user's query to comprehensively judge the user's data needs in production scheduling, troubleshooting, technological innovation and other work, find the target database, generate high-quality prompt words, perform reasoning in the large model without fine-tuning, generate SQL statements, and execute them in the target database to obtain corresponding data, thereby improving the timeliness and accuracy of data retrieval, helping technicians to achieve efficient control of the database, and solving the data retrieval difficulties caused to technicians by massive data and unknown data; thereby promoting the efficient completion of tasks such as optimizing production line layout and product technological innovation.

[0100] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the description and implementation methods. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A multimodal assisted precision manufacturing database interaction method based on a large model, characterized in that: include: Step 1: Obtain surveillance video of precision manufacturing production scenes and obtain user queries; Step 2: rewrite the user query into a new query that complies with the large model language specification; perform a schema search in the precision manufacturing database based on the new query, obtain relevant data tables and fields, and obtain a schema set; Step 3: Perform feature extraction on the surveillance video to obtain a production scene video feature vector and a production scene audio feature vector; perform feature extraction on the new query to obtain a user query text feature vector; Step 4: Perform multimodal fusion on the production scene video feature vector, the production scene audio feature vector, and the user query text feature vector based on the attention mechanism to obtain a user query fusion feature vector; Step 5: Search an external vector database using the user query fusion feature vector to obtain a search result document set; Step 6: Construct an SQL statement to generate prompt words based on the new query, the search result document set, and the schema set; Step 7: Input the SQL statement generation prompt word into the large model for reasoning to generate the SQL statement; Step 8: Put the SQL statement into the precision manufacturing database for execution to generate search data as the search result; In the second step, the user query is rewritten into a new query that conforms to the large model language specification, including the following steps: Step 1: Use the embedding model to convert the user query into a vector. Call an external vector database and retrieve query samples stored in the external vector database based on similarity retrieval rules. Use the retrieved query samples as reference examples. Input the reference examples and the user query into the big model. Input the query classification rules into the big model and use the big model to classify the user query to obtain the category of the user query. Step 2: Using the rewriting model, after decomposing the target task into one or more subtasks according to the user query category, the user query is converted into one or more rewritten subqueries to obtain the new query.

2. The multimodal assisted precision manufacturing database interaction method based on a large model according to claim 1 is characterized in that: Also includes: The user determines whether the search data meets the requirements. If so, the search data is output as the final search result. If it does not meet the requirements, return to step 2 for a new round of data retrieval.

3. The method for interacting with a multimodal assisted precision manufacturing database based on a large model according to claim 2 is characterized in that: There are four types of user queries, namely explicit fact queries, implicit fact queries, explainable reasoning queries and hidden reasoning queries.

4. The method for interacting with a database of multimodal assisted precision manufacturing based on a large model according to claim 3 is characterized in that: In step 2, the method for obtaining the schema set includes the following steps: Step a: breaking down the new query into words or phrases, so that the entire new query is divided into different semantic keywords; Step b: using embedding technology and matching algorithm, matching the semantic keywords with tables and fields in the precision manufacturing database, and using the matching results as a schema set.

5. The method for interacting with a database of multimodal assisted precision manufacturing based on a large model according to claim 4 is characterized in that: In step b, the method for matching the semantic keywords with the tables and fields in the precision manufacturing database is: Using embedding technology to convert the semantic keywords and the table names and field names in the precision manufacturing database into vectors; Calculate the cosine similarity between the semantic keyword vector and the table and field vectors to obtain a similarity result set; Set a similarity threshold and perform schema pruning based on the similarity threshold, retaining tables and fields above the threshold as matching results.

6. The method for interacting with a database of multimodal assisted precision manufacturing based on a large model according to claim 5 is characterized in that: In step 4, the method for obtaining the user query fusion feature vector is: Inputting the production scene video feature vector, the production scene audio feature vector and the user query text feature vector into a multimodal fusion strategy model; The multimodal fusion strategy model splices the production scene video feature vector and the production scene audio feature vector to obtain a spliced feature vector; A multi-head cross attention mechanism is used to perform a weighted combination of the concatenated feature vector and the user query text feature vector to obtain a user query fusion feature vector.

7. The method for interacting with a large model-based multimodal assisted precision manufacturing database according to claim 5 or 6, characterized in that: In step 5, the method for obtaining the search result document set is: Using the user query fusion feature vector to search in an external vector database, calculating the correlation between the user query fusion feature vector and the document based on cosine similarity, screening out documents with a correlation greater than 0, and obtaining a recalled document set; using the RAG rearrangement model, calculating the rearrangement score of each recalled document in the recalled document set, and obtaining a rearrangement score set; Set a re-ranking score threshold and retain the recalled documents above the threshold as the retrieval result document set.

8. According to the large model-based multimodal assisted precision manufacturing database interaction method according to claim 7, the external vector database adopts the Pinecone vector database.

Images