Industrial process multi-mode voice question-answering system and method

Abstract

The invention provides a multi-mode voice question-answering system and method for an industrial process, and relates to the technical field of man-machine interaction, and the method comprises the steps: receiving a natural language query about the industrial process input by a user; performing entity recognition and linking on the natural language query, and determining at least one process entity pointed by the natural language query; based on a preset industrial process knowledge graph, relation traversal and reasoning are carried out on the at least one process entity, at least one associated element implicitly related to the query is obtained through expansion, and the associated element comprises one or more of associated equipment, associated parameters and associated graph numbers; fusing the natural language query with the associated elements obtained by expansion to generate enhanced query; retrieving is performed in parallel using the enhanced query. According to the method, the retrieval precision ratio is relatively high; delivery of pictures and texts in a WYSIWYG mode; the operation cost is zero, and the streaming voice interaction enables a worker to obtain guidance without taking off a glove and typing when operating a machine tool or assembling.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and in particular to a multimodal voice question-and-answer system and method for industrial processes. Background Technology

[0002] With the rapid development of artificial intelligence technology, intelligent question-answering systems based on large language models have been applied in multiple vertical fields. Existing technical solutions, such as patent documents (CN116932911A, a method for constructing an electrical knowledge question-answering assistant based on ChatGPT; CN117633179A, an intelligent question-answering assistant system for the railway industry; and CN119886327A, a method for an intelligent question-answering assistant for urban management based on a large language model), demonstrate a typical process of general retrieval-enhanced generation mode: vectorizing the natural language question input by the user, performing similarity retrieval in a pre-built document vector library, using the retrieved text paragraphs as context input to the large language model, and finally generating a plain text answer.

[0003] However, when the aforementioned general technical solutions are applied to complex, dynamic, and high-noise industrial environments such as assembly, welding, and machining production lines, their inherent defects are amplified, making it difficult to meet actual needs. The following technical shortcomings exist: First, there are issues of insufficient retrieval precision and the semantic gap. Existing technologies primarily rely on semantic similarity of text content or keyword matching for retrieval. However, in industrial scenarios, frontline operators tend to ask questions using short, colloquial language, while process documents use rigorous, professional technical terminology. This semantic gap between colloquial questions and professional documents causes traditional vector retrieval to easily miss technical documents containing key parameters such as tolerances and material specifications, or return a large number of irrelevant results, resulting in low retrieval precision.

[0004] Furthermore, the information presentation is simplistic and lacks key visual content. Industrial process knowledge heavily relies on non-textual information such as structured drawings, assembly diagrams, parameter tables, and 3D models. The solution presented in CN116932911A, a ChatGPT-based power knowledge question-answering assistant construction method, typically only returns answers in plain text format, failing to accurately and reliably link and display process drawings or technical charts directly related to the current process step. This significantly diminishes the information's value for on-site workers who need to follow the drawings.

[0005] Finally, the interaction mode is not well-suited to the constraints of on-site operations. Existing question-and-answer systems mostly rely on text input or non-streaming, single-voice commands. In industrial settings, operators' hands are often occupied by tools and workpieces, and the environment is noisy. Existing interaction methods cannot support hands-free, continuous, and real-time voice dialogue, increasing the cognitive load on operators and the cost of operational interruptions, thus limiting their practicality.

[0006] In other words, there is room for improvement in existing technologies. There is a need to disclose an intelligent question-answering technology and system that can understand industrial slang, accurately retrieve mixed text and image knowledge, and support natural-flowing voice interaction in order to overcome the shortcomings of existing technologies. Summary of the Invention

[0007] This invention overcomes the shortcomings of the prior art and provides an industrial process multimodal voice question answering system and method, which has the advantages of high retrieval precision and WYSIWYG text and image delivery.

[0008] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution: A knowledge graph-enhanced multimodal question answering method for industrial processes includes the following steps: S1: Receive natural language queries from users regarding industrial processes; S2: Perform entity recognition and linking on the natural language query to determine at least one process entity it points to; S3: Based on a preset industrial process knowledge graph, perform relational traversal and reasoning on the at least one process entity to expand and obtain at least one associated element implicitly related to the query. The associated element includes one or more of associated equipment, associated parameters, and associated drawing numbers. S4: Merge the natural language query with the expanded associated elements to generate an enhanced query; S5: Utilize the enhanced query to perform the following retrieval in parallel: Retrieve relevant textual knowledge fragments from structured / unstructured document libraries; From the knowledge graph, based on predefined relationships, at least one visual process resource corresponding to the current query context is determined and obtained; S6: Based on the retrieved text knowledge fragments, generate a text response using a large language model, and align and encapsulate the at least one visualized process resource with the text response to form a multimodal response result; S7: Output the multimodal response results.

[0009] Furthermore, in step S1, receiving user input refers to collecting the user's continuous voice stream at the industrial site through a front-end device; performing real-time voice activity detection and segmentation on the voice stream; and using a streaming automatic speech recognition engine to transcribe it in real time to obtain the text content of the natural language query.

[0010] Furthermore, in step S3, the industrial process knowledge graph takes process node as its core and connects equipment node, material node, parameter node and resource node through relation edges; the relation traversal and reasoning specifically includes: locating the corresponding process node in the knowledge graph based on the identified process entity, and traversing it to obtain closely related node information as the related elements.

[0011] Furthermore, in step S5, obtaining the visualized process resources means directly obtaining the storage identifier or access address of the corresponding process drawings, parameter tables, or 3D models based on the predefined relationship between the process nodes and resource nodes.

[0012] Furthermore, in step S6, aligning and encapsulating the visualized process resources with the text response means that when the large language model generates the text response, it inserts structured placeholders at the locations where the visualized resources need to be referenced. The backend system encapsulates the generated text stream and the visual resource access address obtained according to the resource node corresponding to the placeholder into a unified structured data object with time sequence or paragraph markers.

[0013] Furthermore, the structured data object is in JSON format.

[0014] Furthermore, it also includes an incremental synchronization step S0; specifically including: Monitor updates to upstream process documentation sources; When a document update is detected, the changed content is automatically parsed, the vector index of the document library is updated synchronously, and the relevant node attributes in the industrial process knowledge graph are updated to maintain the consistency of knowledge.

[0015] A voice question-answering system for implementing the aforementioned multimodal question-answering method for industrial processes includes: The interactive access module is used to receive natural language queries input by the user, the queries originating from streaming speech-to-text or text input; The knowledge graph module stores a networked knowledge structure composed of industrial process entities and relationships. The query understanding and enhancement module is used to perform entity recognition on the query and query the knowledge graph module to perform relational reasoning and query expansion, thereby generating an enhanced query. The hybrid retrieval module is used to retrieve text fragments from an external document library and retrieve bound visualization process resources from the knowledge graph module in parallel, based on the enhanced query. The multimodal generation and synthesis module is used to call a large language model to generate a text response based on the retrieved text fragments, and synthesize the text response with the retrieved visualization process resources to form a multimodal response; The output module is used to render and present the multimodal response results to the user.

[0016] Furthermore, the interactive access module further includes: The streaming speech processing unit is used to interface with the front-end microphone to perform speech activity detection, noise reduction, and streaming speech recognition, enabling real-time and continuous transcription of user speech.

[0017] Furthermore, it also includes: The incremental synchronization module is used to monitor updates to process documents and automatically synchronize the updated content to the external document library and the knowledge graph module to ensure the timeliness of system knowledge.

[0018] Compared with the prior art, the beneficial effects of the present invention are: This invention boasts a high retrieval precision rate, overcoming the shortcomings of traditional RAGs in matching industrial terms; its WYSIWYG graphic delivery overcomes the deficiency of existing LLMs in accurately outputting industrial drawings; and its zero-operation-cost streaming voice interaction allows workers to obtain guidance without removing gloves or typing while operating machine tools or assembling. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate the invention and are used together with the embodiments of the invention to explain the invention. They do not constitute a limitation of the invention. In the drawings: Figure 1 This is a schematic diagram of the overall system architecture of the present invention; Figure 2 This is an enhanced query flowchart; Figure 3 This is a schematic diagram of the spectrum structure. Detailed Implementation

[0020] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0021] like Figures 1 to 3 As shown, this invention claims protection for a multimodal voice question-answering method for industrial processes, comprising the following steps: Step S1: Receive natural language queries from the user regarding industrial processes.

[0022] Users input questions about industrial processes through terminal devices such as industrial tablets, smartphones, or smart headsets. Input can be text or voice input captured by a microphone. For example, in a specific application scenario, a welding workshop operator could hold the device and speak into the microphone: "What is the allowable error for the current weld?" This serves as an example of natural language query input. Step S2: Perform entity recognition and linking on the natural language query.

[0023] Upon receiving a query, the system first performs natural language processing to identify the process entities mentioned in the query. For example, for the query above, the entity recognition model would identify the key entity "weld". Then, through entity linking technology, this colloquial description of "weld" is precisely associated with standardized process entity nodes in the knowledge graph, determining that it points to at least one process entity.

[0024] Step S3: Expand the query based on relational traversal and reasoning of the knowledge graph.

[0025] The system accesses a pre-defined industrial process knowledge graph. This knowledge graph is centered around process nodes and connects nodes such as equipment, materials, parameters, and drawing resources through various relational edges.

[0026] Based on the nodes determined in step S2, the system performs relationship traversal and reasoning within the knowledge graph. For example, it finds associated parameter nodes, associated drawing resource nodes, and associated equipment nodes. These expanded associated elements constitute the professional context implicit in the user's original query.

[0027] Step S4: Generate enhanced query.

[0028] like Figure 2 As shown, the system merges the user's original natural language query with related elements obtained from the knowledge graph to generate a richer and more specialized enhanced query. For example, the merged query can include parameters such as the allowable error parameter.

[0029] Step S5: Parallel hybrid retrieval.

[0030] The system utilizes the enhanced query described above to perform two types of searches in parallel: Text retrieval transforms enhanced queries into vectors, and performs similarity searches in a vector database of process documents, including structured and unstructured documents such as process specification PDFs and work instruction Word documents, to retrieve the most relevant text knowledge fragments.

[0031] The system enables visual resource retrieval. Simultaneously, it directly determines and retrieves the storage identifier or access address of the corresponding visual process resource based on the established relationships within the knowledge graph. This process does not rely on the illusionary generation of generative models, but rather on precise invocation based on predefined, deterministic relationships.

[0032] Step S6: Multimodal response generation and encapsulation.

[0033] The system inputs the retrieved text knowledge fragments into the large language model, instructing it to generate text responses that are easy for workers to understand. Simultaneously, the large language model inserts structured placeholders at locations in the response where images need to be referenced.

[0034] The backend system then aligns and encapsulates the text stream generated by the large language model with the visual resource access address obtained in step S5. Specifically, the system replaces placeholders in the text with corresponding resource addresses and encapsulates all information into a unified, tagged structured data object. In this embodiment, this object is in JSON format: Step S7: Output the multimodal response results.

[0035] After receiving the structured response, the front-end application or terminal device simultaneously renders the text content and visual resources. For example, the text response is displayed word by word at the top of the screen, while simultaneously loading and displaying it below the text or at a specified location, delivering a visually appealing and WYSIWYG answer.

[0036] It also includes the specific implementation of streaming voice interaction. In step S1, receiving user input is specifically implemented in the following way: 1. Voice Acquisition and Front-End Processing. Devices worn or held by industrial workers, featuring built-in microphones. When acquiring voice streams, the front-end application integrates voice activity detection algorithms to automatically filter out silent segments from environmental background noise, sending only valid voice segments to the back-end.

[0037] 2. Streaming Speech Recognition: The backend employs a streaming automatic speech recognition engine, such as an end-to-end ASR model based on deep learning. It features real-time performance, dual-output, and text acquisition. Real-time performance means it can recognize ongoing speech and output intermediate results without waiting for the user to finish speaking a complete sentence. For example, if a user says "check it," the screen may immediately display the transcribed text. Dual-output refers to the engine simultaneously outputting partial recognition results for real-time feedback and the complete sentence for final confirmation, used for subsequent processing. This achieves a smooth experience of speaking and transcribing simultaneously, allowing for immediate querying after speaking. Text acquisition is the final step; the system uses the transcribed text as a natural language query and passes it to the subsequent query understanding module. This approach completely frees up workers' hands and adapts to the operational constraints of industrial environments.

[0038] like Figure 3As shown, the industrial process knowledge graph is constructed using an attribute graph model, and its core structure includes: Node types include processes, equipment, materials, parameters, and resources.

[0039] Relation type: Defines semantically meaningful relation edges.

[0040] The process of obtaining visualized process resources in step S5 is deterministic: Once the query locates a specific process node through steps S2-S4, the system directly queries the relationship edges originating from that node. These edges were explicitly established by the process engineer during the knowledge graph construction phase, pointing to specific, approved drawings or forms.

[0041] This method fundamentally eliminates the possibility of illusory graphics or information errors that generative models may produce, ensuring the absolute accuracy of information in industrial scenarios.

[0042] In step S6, the alignment and encapsulation mechanism ensures that the text and images are logically consistent and positionally matched in the final rendering: Regarding placeholder insertion, the system explicitly requires the large language model to insert fixed-format markers at specific positions in the generated answer in its instructions.

[0043] Regarding resource binding, in a knowledge graph, resource nodes themselves may carry semantic tags. After retrieving a resource, the system records its tags.

[0044] Regarding structured encapsulation, the backend system splits the LLM-generated text into paragraphs, scans for placeholder tags, and identifies the bound resource URLs based on semantic labels within those tags. Subsequently, the system encapsulates all information into a well-structured data object. JSON format is preferred due to its cross-platform compatibility and ease of parsing. The encapsulated object not only contains the text and resource list but also specifies the insertion position of each resource within the text, allowing the frontend to perform precise rendering.

[0045] In addition, it includes an incremental synchronization step S0 independent of the question-and-answer process, specifically including: Listening and triggering, that is, the system has a file monitoring service or an interface with the product lifecycle management system to continuously monitor version changes of authoritative upstream process documents.

[0046] Content parsing and index updates: When a document update is detected, the system automatically parses the new document, extracts the text content, and updates the vector index of the document library to ensure that the search results are based on the latest text.

[0047] When the graph attributes are updated, the system simultaneously parses the key information in the document and triggers the update of the corresponding node attributes in the knowledge graph.

[0048] Consistency assurance: This closed-loop synchronization mechanism ensures that the facts stored in the vector retrieval library and the knowledge graph are always consistent, so that the answers given by the question-answering system are always based on the latest process standards, avoiding the risk of information lag caused by untimely document updates.

[0049] This invention also claims protection for an industrial process multimodal voice question-answering system for implementing the above method, comprising: Interactive access module: As the system's front-end interface, it receives user voice streams or text input. Its streaming speech processing unit is responsible for VAD, noise reduction, and ASR transcription.

[0050] Knowledge Graph Module: Stores and maintains a structured knowledge graph of industrial processes.

[0051] The query understanding and enhancement module includes an NLP model responsible for entity recognition and linking, and calls the knowledge graph module for reasoning and query expansion.

[0052] Hybrid retrieval module: Includes a vector retrieval engine, which can query resource associations in the document vector library and knowledge graph in parallel.

[0053] Multimodal generation and synthesis module: calls the large language model API to generate text, and synthesizes and encapsulates the text with the retrieved resources.

[0054] Output module: responsible for rendering the encapsulated multimodal response into a graphical interface visible to the end user.

[0055] Incremental synchronization module: As a background service, it monitors document updates and drives the synchronous updates of the knowledge base.

[0056] The above structure forms a complete, intelligent question-and-answer system suitable for industrial environments, from natural language input to graphical and textual answer output.

[0057] This invention boasts a high retrieval precision rate, overcoming the shortcomings of traditional RAGs in matching industrial terms; its WYSIWYG graphic delivery overcomes the deficiency of existing LLMs in accurately outputting industrial drawings; and its zero-operation-cost streaming voice interaction allows workers to obtain guidance without removing gloves or typing while operating machine tools or assembling.

[0058] Finally, it should be noted that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. However, any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal question-answering method for industrial processes based on knowledge graph enhancement, characterized in that, Includes the following steps: S1: Receive natural language queries from users regarding industrial processes; S2: Perform entity recognition and linking on the natural language query to determine at least one process entity it points to; S3: Based on a preset industrial process knowledge graph, perform relational traversal and reasoning on the at least one process entity to expand and obtain at least one associated element implicitly related to the query. The associated element includes one or more of associated equipment, associated parameters, and associated drawing numbers. S4: Merge the natural language query with the expanded associated elements to generate an enhanced query; S5: Utilize the enhanced query to perform the following retrieval in parallel: Retrieve relevant textual knowledge fragments from structured / unstructured document libraries; From the knowledge graph, based on predefined relationships, at least one visual process resource corresponding to the current query context is determined and obtained; S6: Based on the retrieved text knowledge fragments, generate a text response using a large language model, and align and encapsulate the at least one visualized process resource with the text response to form a multimodal response result; S7: Output the multimodal response results.

2. The industrial process multimodal question-and-answer method according to claim 1, characterized in that, In step S1, receiving user input means collecting the user's continuous voice stream at the industrial site through a front-end device; performing real-time voice activity detection and segmentation on the voice stream; and using a streaming automatic speech recognition engine to transcribe it in real time to obtain the text content of the natural language query.

3. The industrial process multimodal question-and-answer method according to claim 1, characterized in that, In step S3, the industrial process knowledge graph takes process operation nodes as the core and connects equipment nodes, material nodes, parameter nodes and resource nodes through relational edges; the relational traversal and reasoning specifically includes: based on the identified process entity, locating the corresponding process node in the knowledge graph, traversing it, and obtaining closely related node information as the related elements.

4. The industrial process multimodal question-and-answer method according to claim 3, characterized in that, In step S5, obtaining the visualized process resources means directly obtaining the storage identifier or access address of the corresponding process drawings, parameter tables, or 3D models based on the predefined relationship between the process nodes and resource nodes.

5. The industrial process multimodal question-and-answer method according to claim 1, characterized in that, In step S6, aligning and encapsulating the visualized process resources with the text response means that when the large language model generates the text response, it inserts structured placeholders at the locations where the visualized resources need to be referenced. The backend system encapsulates the generated text stream and the visual resource access address obtained according to the resource node corresponding to the placeholder into a unified structured data object with time sequence or paragraph markers.

6. The industrial process multimodal question-and-answer method according to claim 5, characterized in that, The structured data object is in JSON format.

7. The industrial process multimodal question-and-answer method according to claim 1, characterized in that, It also includes incremental synchronization step S0; specifically, it includes: Monitor updates to upstream process documentation sources; When a document update is detected, the changed content is automatically parsed, the vector index of the document library is updated synchronously, and the relevant node attributes in the industrial process knowledge graph are updated to maintain the consistency of knowledge.

8. A voice question-and-answer system for implementing the industrial process multimodal question-and-answer method as described in any one of claims 1 to 7, characterized in that, include: The interactive access module is used to receive natural language queries input by the user, the queries originating from streaming speech-to-text or text input; The knowledge graph module stores a networked knowledge structure composed of industrial process entities and relationships. The query understanding and enhancement module is used to perform entity recognition on the query and query the knowledge graph module to perform relational reasoning and query expansion, thereby generating an enhanced query. The hybrid retrieval module is used to retrieve text fragments from an external document library and retrieve bound visualization process resources from the knowledge graph module in parallel, based on the enhanced query. The multimodal generation and synthesis module is used to call a large language model to generate a text response based on the retrieved text fragments, and synthesize the text response with the retrieved visualization process resources to form a multimodal response; The output module is used to render and present the multimodal response results to the user.

9. The system according to claim 8, characterized in that, The interactive access module further includes: The streaming speech processing unit is used to interface with the front-end microphone to perform speech activity detection, noise reduction, and streaming speech recognition, enabling real-time and continuous transcription of user speech.

10. The system according to claim 8, characterized in that, Also includes: The incremental synchronization module is used to monitor updates to process documents and automatically synchronize the updated content to the external document library and the knowledge graph module to ensure the timeliness of system knowledge.

Images