Electric welding vertical field knowledge question-answering method based on multi-modal retrieval enhancement

Abstract

The invention discloses an electric welding vertical field knowledge questioning and answering method based on multi-modal retrieval enhancement. The method comprises the following steps of obtaining electric welding field multi-modal data and performing preprocessing; the method comprises the following steps: carrying out textualized interpretation by a multi-modal large model Qwen 2.5 VL-32B according to input multi-modal data, and aligning data contents to a text format to obtain textualized multi-modal knowledge interpretation of electric welding; carrying out vectorization processing on the obtained textualized multi-modal knowledge explanation of the electric welding to obtain an electric welding knowledge vector, establishing an electric welding knowledge vector-text point tracing record, and uniformly storing the electric welding knowledge vector-text point tracing record in a localized line; vectorization processing is carried out on query input by a user, the query is compared with locally stored electric welding knowledge vectors, and the first K most relevant electric welding knowledge vectors are selected by calculating the similarity; according to the first K most relevant electric welding knowledge vectors, original corresponding multi-mode electric welding data are pulled through the recording anchor points and put into the multi-mode large model, and the multi-mode large model answers according to electric welding relevant knowledge in the multi-mode electric welding data.

Description

Technical Field

[0001] This invention belongs to the technical field of multimodal knowledge base question answering, specifically involving a knowledge question answering method for the vertical field of welding based on multimodal retrieval enhancement. Background Technology

[0002] Electric welding technology, as a crucial process in metal processing and manufacturing, is widely used in industrial fields such as shipbuilding, bridge construction, machining, and pressure vessel manufacturing. With the development of industrial automation and intelligence, the complexity of electric welding processes is constantly increasing, involving not only welding materials, welding process parameters, and equipment operation methods, but also weld inspection, welding defect analysis, and safety regulations.

[0003] In practical applications, knowledge in the field of welding comes from diverse sources, including textbooks, technical documents, process demonstrations in the form of practical images or worker instruction videos, and data such as temperature, current, and voltage collected by sensors in welding equipment. This information exhibits significant multimodal characteristics, and in production settings, operators often need to quickly obtain accurate technical answers. However, existing knowledge-based question-and-answer methods in the welding field are mostly built upon single textual data, failing to effectively utilize multimodal information, resulting in insufficient accuracy and professionalism in the answers.

[0004] While existing multimodal information processing technologies can identify and analyze image, video, and sensor data, they still have the following problems in knowledge-based question-answering scenarios: 1. The lack of a unified representation for multimodal information makes it difficult to efficiently associate different modalities; 2. Retrieval mechanisms rely mainly on keyword matching, which lacks semantic understanding capabilities, especially in terms of professional terminology and complex process descriptions, which can easily lead to retrieval bias; 3. Existing multimodal retrieval methods lack knowledge enhancement capabilities in the welding vertical field, and cannot efficiently integrate professional domain knowledge with multimodal data into the answer generation process.

[0005] Furthermore, with the development of large-scale model technology, general-purpose multimodal large-scale models have emerged that can be used for multimodal knowledge question answering in general domains. For example, the Qwen2.5-VL-32B model, developed by Bai Shuai et al., was published in the arXiv preprint platform in 2025 as the "Qwen2.5-VL Technical Report". Inputting images or videos lasting several hours along with corresponding task prompts, it can perform multimodal information text processing, multimodal information analysis, and question answering, obtaining textualized multimodal information and multimodal question answers. However, due to the universality of the training data, general-purpose multimodal large-scale models are limited in their performance in providing professional answers in the field of welding, and cannot provide accurate and professional answers to specialized welding questions.

[0006] Therefore, there is an urgent need for a knowledge-based question-answering method for the vertical field of electric welding that can uniformly convert multimodal data into semantic representations and combine high-precision vector retrieval with knowledge enhancement mechanisms to improve the accuracy of retrieval and the professionalism of answers, thereby better meeting the actual needs of electric welding technicians in production, training and fault diagnosis. Summary of the Invention

[0007] The main objective of this invention is to overcome the problems of low semantic relevance, insufficient retrieval accuracy, and limited knowledge enhancement capabilities in existing knowledge-based question-answering methods for the welding field when processing multimodal information. This invention provides a knowledge-based question-answering method for the welding vertical domain based on multimodal retrieval enhancement. This method can uniformly textualize multimodal data, construct a high-precision semantic vector index, and significantly improve the accuracy and professionalism of the answers by utilizing a multimodal knowledge enhancement mechanism. To achieve the above objectives, the present invention adopts the following technical solution: A knowledge-based question-answering method for the welding vertical domain, enhanced by multimodal retrieval, is described below: S1. Acquire multimodal data in the field of electric welding, including electric welding images, worker operation instruction videos, electric welding knowledge audio, and structured process parameters; S2. Preprocess the multimodal data in the field of electric welding, including denoising, format unification, key frame extraction and sensitive information desensitization, to obtain preprocessed multimodal data; S3. The preprocessed multimodal data is interpreted using the multimodal large model Qwen 2.5 VL-32B, aligning non-textual modal content into a unified textual description to obtain textualized multimodal knowledge. The Qwen2.5-VL-32B model is based on the paper "Qwen2.5-VL Technical Report" published on the arXiv preprint platform in 2025 by BaiShuai et al. Input images or videos lasting several hours and corresponding task prompts to achieve functions such as multimodal information textualization, multimodal information analysis and question answering, and obtain textualized multimodal information and answers to multimodal questions.

[0008] Among them, the Qwen 2.5 VL-32B model allows users to input multimodal information such as images or videos. and input text commands Afterwards, the analyzed answer will be obtained. The process is represented as follows:

[0009] in This represents the response function of the Qwen 2.5 VL-32B model, which combines multimodal information and text commands.

[0010] S4. Vectorize the textualized multimodal knowledge description to obtain a high-dimensional semantic knowledge vector, and establish an index between the semantic knowledge vector and the original multimodal data; S5. Localize and store the semantic knowledge vectors and their indexes to obtain a local semantic knowledge vector index library for the welding field. S6. Receive the query input by the user, obtain the query vector by vectorizing the query, and perform semantic similarity matching between the query vector and the semantic knowledge vector in the local welding field semantic knowledge vector index library to retrieve the top K most relevant semantic knowledge vectors. S7. Based on the anchor point record, the corresponding multimodal data is retrieved from the top K most relevant semantic knowledge vectors through an index, and used as context-related knowledge; S8. Input the context-related knowledge and user query into the multimodal large model to generate an answer based on welding domain knowledge.

[0011] Furthermore, the multimodal data includes at least one of images, videos, and text information. The image data is captured by professional welders during actual welding operations, with text annotations added to the images; the videos are obtained from various self-media platforms by searching for welding keywords; and the text information includes welding-related teaching materials.

[0012] Furthermore, text interpretation of multimodal data in the field of electric welding can uniformly convert multimodal information into text information, facilitating local storage and rapid retrieval. The process is represented as follows:

[0013]

[0014] The multimodal dataset M = {M1, M2, ..., M} i ,…,M n}, Let i represent the i-th multimodal data point, i = 1, 2, ..., n, where n is the total number of multimodal data points. The response function represents the Qwen 2.5 VL-32B multimodal large model, and here it represents the textualization mapping process. These are keywords for the abstract task. This represents the text interpretation generated from the i-th multimodal data point.

[0015] Furthermore, the semantic vector construction for text interpretation and the representation of semantic knowledge vectors are as follows:

[0016]

[0017] in, This represents a semantic encoding function that maps text to vectors. Represents the correspondence of dimension d The semantic knowledge vector of text interpretation, the semantic knowledge vector set V={v1,v2,…,v i ,…,v n}, This represents the semantic knowledge vector of the i-th multimodal data item. Anchor index function, representing localized records. and A one-to-one correspondence can be recorded and then input using this function. Find the corresponding By pre-vectorizing the knowledge in the knowledge base and storing it locally, the time spent on interpreting and vectorizing massive amounts of text after user input will be saved, thus speeding up the response to user queries.

[0018] Furthermore, in step S6, the user queries... Vectorization and query vector With semantic knowledge vectors similarity The calculations are as follows:

[0019]

[0020] Based on the above definition, cosine similarity is calculated for each semantic knowledge vector in the semantic knowledge vector set, and then... Functions for retrieving and querying vectors The formula for calculating the top K semantic knowledge vectors with the highest similarity is:

[0021] in This indicates that for the query vector The top K semantic knowledge vectors with the highest similarity scores in the semantic knowledge vector set V. Where j=1,2,...,K. Through the above steps, the semantic knowledge vectors stored locally can be retrieved in batches with the current user query, and the semantic knowledge vectors most relevant to the user query in the local knowledge base can be obtained, which prepares for the subsequent steps of the large model to answer the question by combining professional welding knowledge.

[0022] Furthermore, in step S7, the anchor index function is used. ,pass Obtain raw multimodal information :

[0023] in, The original multimodal information obtained from the retrieval represents the information obtained for the query vector. The j-th multimodal data point with the highest similarity score in the multimodal dataset M. Using pre-stored local index relationships, the top K most relevant original multimodal information points to the current user query are retrieved to assist the large multimodal model Qwen 2.5 VL-32B in answering user queries related to welding knowledge, thereby improving the model's professional answering capabilities in the welding vertical field.

[0024] Compared with the prior art, the present invention has the following advantages and beneficial effects: 1. Compared with existing text-based welding knowledge retrieval and question answering methods, this method proposes a welding vertical domain knowledge question answering method based on multimodal retrieval enhancement. It utilizes existing channels for text-based processing of multimodal data such as images and videos in the welding field, effectively aligning multimodal information with text-based retrieval methods. This enriches the welding knowledge base, which can only process text knowledge, into a knowledge base that can process multimodal information, breaking through the original limitations on the types of data that can be processed. In this way, a richer and more diverse knowledge base can be constructed to provide users with more objective and reliable answers.

[0025] 2. Unlike typical multimodal file retrieval methods, multimodal files are transmitted as links to enable fast and efficient user interaction. These links open in a browser after the user clicks on them. This avoids the delays caused by long-term transmission of large multimodal files, providing users with a seamless interactive experience. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart illustrating the specific process of the knowledge-based question-answering method for the vertical field of electric welding based on multimodal retrieval enhancement of the present invention. Figure 2 This is a system structure diagram of the knowledge question answering method for the vertical field of electric welding based on multimodal retrieval enhancement of the present invention. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0029] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0030] Example 1 Figure 1 This is a flowchart of a knowledge-based question-answering method for the vertical field of electric welding, disclosed in this invention, such as... Figure 1 As shown in the figure, the knowledge question answering method for the vertical field of electric welding based on multimodal retrieval enhancement disclosed in this embodiment includes the following steps: acquiring multimodal data in the electric welding field, preprocessing the multimodal data, generating text explanations of the multimodal data, vectorizing the text explanations, vectorizing the textual multimodal knowledge descriptions and constructing an index, retrieving the knowledge vector with the highest similarity to the user query, pulling the corresponding multimodal data, and obtaining a professional answer from a large model, as detailed below: T1. Acquire multimodal data in the field of electric welding, including 480 images of electric welding processes, 289 instructional videos, and knowledge data on electric welding teaching materials provided by welders; T2. Preprocess the multimodal data, including image denoising and size normalization to 1024×1024, and extract 1 frame from the video every 5 seconds to obtain the preprocessed multimodal data; T3. Input the preprocessed multimodal data into the multimodal large model Qwen2.5-VL-32B, and with the prompt "Please use professional welding terminology to describe this content in a structured way, including process type, materials, parameter suggestions and safety points", generate 769 text explanations; T4. Vectorize the text interpretation to obtain a semantic knowledge vector set V={v1,…,v...} n}, and establish the anchor index function Index(·) to record the one-to-one correspondence between vᵢ and the original multimodal data Mᵢ; T5. Vectorize the textualized multimodal knowledge description to obtain a high-dimensional semantic knowledge vector, and establish an index between the semantic knowledge vector and the original multimodal data; T6. Receive user query "How to weld 304 stainless steel sheet?", vectorize it into q using the same method, calculate the cosine similarity s(q,vᵢ) between q and all vectors in V, and select the three semantic knowledge vectors with the highest similarity scores. T7. Retrieve the corresponding three multimodal data points using the Index anchor point function; T8. Input the user query and the three multimodal data obtained in T7 into Qwen2.5-VL-32B to generate a professional answer that includes process suggestions, parameter ranges, safety tips, and links to multimodal attachments.

[0031] This invention was comprehensively evaluated on a private multimodal dataset. As can be seen from the results in Table 1, compared with the text-based knowledge retrieval tool BGE, the multimodal large model Qwen2.5-VL-32B, and the text retrieval enhancement method Langchain-ChatChat, this method achieves the best performance in responding to private datasets in the field of welding, and simultaneously supports multimodal knowledge detection and understanding response capabilities.

[0032] Table 1. Comparison of accuracy and supported features of this method with other methods on private datasets.

[0033] As shown in Table 1, this knowledge-based question-answering method for the welding vertical domain, enhanced by multimodal retrieval, introduces rich multimodal knowledge on top of traditional text-based retrieval enhancement methods by employing a textual interpretation approach for multimodal data. Combined with the general question-answering capabilities of a large multimodal model, this method significantly improves response accuracy compared to traditional retrieval methods like BGE, the question-answering model Qwen2.5-VL-32B, and the text-based retrieval method Langchain-ChatChat. Furthermore, it is compatible with both text and various multimodal welding knowledge. The BGE model, derived from the paper "C-Pack: Packed Resources For General Chinese Embeddings" published by Shitao Xiao et al. on the arXiv preprint platform in 2024, allows users to input questions and retrieve relevant knowledge from a pre-built text knowledge base as output, but it lacks logical analysis and semantic answering capabilities. The LangChain-Chatchat project, authored by Liu Qian and others, was open-sourced on GitHub in 2024. It is a retrieval enhancement generation method based on a local knowledge base, which can realize document loading, text slicing, vectorized retrieval and question-answer generation. It can be used to retrieve information related to user questions in the local knowledge base and generate semantic answers, but it does not have multimodal information retrieval capabilities.

[0034] Example 2 Based on the multimodal retrieval-enhanced knowledge question answering method for the welding vertical domain disclosed in Embodiment 1, this embodiment continues to refer to steps T1 to T8 of the multimodal retrieval-enhanced knowledge question answering method for the welding vertical domain disclosed in Embodiment 1, such as... Figure 1 As shown, the method includes the following steps: T1. Acquire multimodal data in the field of electric welding, including 480 images of electric welding processes, 289 instructional videos, and knowledge data on electric welding teaching materials provided by welders; T2. Preprocess the multimodal data, including image denoising and size normalization to 1024×1024, and extract 1 frame from the video every 5 seconds to obtain the preprocessed multimodal data; T3. Input the preprocessed multimodal data into the multimodal large model Qwen2.5-VL-32B, and with the prompt "Please use professional welding terminology to describe this content in a structured way, including process type, materials, parameter suggestions and safety points", generate 769 text explanations; T4. Vectorize the text interpretation to obtain a semantic knowledge vector set V={v1,…,v...} n}, and establish the anchor index function Index(·) to record the one-to-one correspondence between vᵢ and the original multimodal data Mᵢ; T5. Vectorize the textualized multimodal knowledge description to obtain a high-dimensional semantic knowledge vector, and establish an index between the semantic knowledge vector and the original multimodal data; T6. Receive user queries, vectorize them into q using the same method, calculate the cosine similarity s(q,vᵢ) between q and all vectors in V, and select the three semantic knowledge vectors with the highest similarity scores. T7. Retrieve the corresponding three multimodal data using the anchor index function Index; T8. Input the user query and the three multimodal data obtained in T7 into Qwen2.5-VL-32B to generate a professional answer that includes process suggestions, parameter ranges, safety tips and links to multimodal attachments; In step T3, the multimodal large model Qwen2.5-VL-32B is used to perform text interpretation to obtain textualized explanations. This is done before the user actually asks interactive questions, thereby reducing the huge latency caused by the method processing multimodal information. In order to verify the performance and speed improvement brought by this operation (described here as multimodal textualization and multimodal knowledge-based processing), the impact of this operation is further tested in a private dataset.

[0035] Table 2. Comparison of accuracy and supported features of this method with other methods on private datasets.

[0036] As shown in Table 2, the introduction of multimodal textual and multimodal knowledge-based methods can significantly improve response accuracy on the basis of plain text. However, compared with the traditional plain text retrieval scheme Langchain-ChatChat, there is no significant difference in average response time. This indicates that the introduction of multimodal data to enhance the retrieval and response on the basis of plain text retrieval not only improves response accuracy but also does not significantly prolong the average response time.

[0037] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0038] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A knowledge-based question-answering method for the vertical field of electric welding based on multimodal retrieval enhancement, characterized in that, The process of the knowledge Q&A method for the vertical field of electric welding is as follows: S1. Acquire multimodal data in the field of electric welding, including electric welding images, worker operation instruction videos, electric welding knowledge audio, and structured process parameters; S2. Preprocess the multimodal data in the field of electric welding, including denoising, format unification, key frame extraction and sensitive information desensitization, to obtain preprocessed multimodal data; S3. Use the multimodal large model Qwen 2.5 VL-32B to perform text interpretation on the preprocessed multimodal data, align the non-textual modal content into a unified textual description, and obtain textualized multimodal knowledge. S4. Vectorize the textualized multimodal knowledge description to obtain a high-dimensional semantic knowledge vector, and establish an index between the semantic knowledge vector and the original multimodal data; S5. Localize and store the semantic knowledge vectors and their indexes to obtain a local semantic knowledge vector index library for the welding field. S6. Receive the query input by the user, obtain the query vector by vectorizing the query, and perform semantic similarity matching between the query vector and the semantic knowledge vector in the local welding field semantic knowledge vector index library to retrieve the top K most relevant semantic knowledge vectors. S7. Based on the anchor point record, the corresponding multimodal data is retrieved from the top K most relevant semantic knowledge vectors through indexing, and used as context-related knowledge; S8. Input the context-related knowledge and user query into the multimodal Qwen 2.5 VL-32B to generate an answer based on knowledge of the welding field.

2. The knowledge-based question-answering method for the vertical field of electric welding based on multimodal retrieval enhancement according to claim 1, characterized in that, The multimodal data includes at least one of images, videos, and text information.

3. The knowledge-based question-answering method for the vertical field of electric welding based on multimodal retrieval enhancement according to claim 1, characterized in that, The textual interpretation of the multimodal data in the field of electric welding is as follows: , , The multimodal dataset M = {M1, M2, ..., M} i ,…,M n }, Let i represent the i-th multimodal data point, where i = 1, 2, ..., n, and n is the total number of multimodal data points. The textual mapping function representing the multimodal large model Qwen 2.5 VL-32B is given. These are keywords for the abstract task. This represents the text interpretation generated from the i-th multimodal data point.

4. The knowledge-based question-answering method for the vertical field of electric welding based on multimodal retrieval enhancement according to claim 3, characterized in that, The semantic vector construction for text interpretation and the representation of semantic knowledge vectors are as follows: , , in, This represents a semantic encoding function that maps text to vectors. Represents the correspondence of dimension d The semantic knowledge vector of text interpretation, the semantic knowledge vector set V={v1,v2,…,v i ,…,v n }, This represents the semantic knowledge vector of the i-th multimodal data item. Anchor index function, representing localized records. and The one-to-one correspondence can be recorded and then used as input using this semantic encoding function. Find the corresponding .

5. The knowledge-based question-answering method for the vertical field of electric welding based on multimodal retrieval enhancement according to claim 4, characterized in that, User query in step S6 Vectorization and query vector With semantic knowledge vectors similarity The calculations are as follows: , ， Based on the above definition, cosine similarity is calculated for each semantic knowledge vector in the semantic knowledge vector set, and then... Functions for retrieving and querying vectors The formula for calculating the top K semantic knowledge vectors with the highest similarity is: in This indicates that for the query vector The top K semantic knowledge vectors with the highest similarity scores in the semantic knowledge vector set V. , where j=1,2,..,K.

6. The knowledge-based question-answering method for the vertical field of electric welding based on multimodal retrieval enhancement according to claim 5, characterized in that, In step S7, the anchor index function is used. Through semantic knowledge vectors Obtain raw multimodal information : ， in, The original multimodal information obtained from the retrieval represents the information obtained for the query vector. , is the j-th multimodal data point with the highest similarity score in the multimodal dataset M.

Images