Industrial internet software intelligent customer service interaction method based on multi-modal deep fusion

Abstract

The invention relates to the technical field of industrial internet, artificial intelligence and human-computer interaction, in particular to an industrial internet software intelligent customer service interaction method based on multi-modal deep fusion. According to the method, an intelligent customer service system integrating natural language processing, a voice technology, computer vision and augmented reality (AR) is constructed aiming at the characteristics of complex operation, multiple user groups, high problem speciality and the like in an industrial scene. The core of the method is that operation questions, interface states and potential faults of engineers in software use are accurately understood through multi-modal input; an accurate solution is generated in combination with the knowledge graph of the industrial software and the real-time data of the equipment; a 3D virtual expert image is driven, and immersive and scene-based guidance is provided by fusing voice explanation, interface marking guidance, AR superposition demonstration and operation demonstration videos. The method has the working condition self-adaptive capability, can identify the professional level (such as a green hand / expert mode) of a user, and adjusts the explanation depth; and cross-region collaboration is supported, a plurality of users are allowed to synchronously share a guidance picture of a virtual expert, and efficient remote collaboration troubleshooting is realized. The method solves the technical problems that traditional industrial software is slow in customer service response, abstract in guidance, difficult to process complex problems on site and the like, and operation and maintenance efficiency and user experience are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of interdisciplinary technology of industrial internet and artificial intelligence. Specifically, it relates to an intelligent customer service method that deeply integrates multimodal interaction, augmented reality and industrial knowledge graph, which is dedicated to improving the technical support and user training experience of complex industrial software (such as MES manufacturing execution system, SCADA monitoring and data acquisition system, PLC programming software, digital twin platform, etc.). Background Technology

[0002] As industrial digital transformation deepens, engineers and maintenance personnel increasingly rely on powerful industrial internet software for monitoring, debugging, and optimization. However, this software is typically complex to operate, has numerous modules, and is tightly coupled with physical equipment, making it difficult for users to quickly resolve problems. Existing technical support methods face significant challenges: 1. Low communication efficiency: Users describe problems by phone or online chat, making it difficult for customer service personnel to intuitively understand the complex software interface status or the on-site situation of the equipment, resulting in repeated communication and frequent misunderstandings.

[0003] 2. Abstract guidance methods and high learning costs: Traditional text or voice guidance is extremely inefficient for scenarios requiring the location of specific buttons, interpretation of complex curves, or on-site operation. Sending operation manuals or recording videos is cumbersome and lacks personalization.

[0004] 3. Lack of on-site experience and collaborative capabilities: Remote customer service cannot "see" the user's equipment in person, making it difficult to provide accurate guidance. When multiple departments collaborate on troubleshooting, information is asynchronously transmitted between different personnel via screenshots and emails, resulting in low efficiency and easy loss of context.

[0005] 4. Difficulty in knowledge transfer and retention: A large amount of valuable troubleshooting experience is scattered in the minds of individual experts or in isolated email records, failing to be structured and retained, and cannot be effectively reused by new employees or used for systematic product improvement.

[0006] 5. Significant differences in user skill levels make personalized service difficult: Novice users need basic operation guidance, while expert users may require in-depth parameter logic or secondary development guidance. Traditional customer service models cannot dynamically differentiate and adapt to these differences.

[0007] Therefore, there is an urgent need for a new generation of intelligent customer service solutions that can deeply understand industrial scenarios, provide intuitive visual guidance, and support efficient collaboration, in order to ensure the stable operation of industrial software and accelerate the improvement of personnel skills. Summary of the Invention

[0008] The technical problem this invention aims to solve is to provide an immersive, scenario-based, and intelligent interactive support method to address the core pain points in industrial internet software customer service, such as communication barriers, abstract guidance, difficulties in collaboration, and insufficient knowledge accumulation.

[0009] To address the aforementioned technical challenges, this invention proposes an intelligent customer service interaction method for industrial internet software based on multimodal deep fusion. The core idea of ​​this method is to construct a three-in-one intelligent perception and guidance system integrating "software interface - field equipment - personnel knowledge." By integrating multi-dimensional user input (operational questions, interface status, field video), and with the support of industrial knowledge graphs and real-time data, problems are accurately diagnosed. Then, a "virtual expert" responds in a multimodal manner most suited to industrial communication habits (professional explanation + interface annotation + AR overlay + animation demonstration). Furthermore, real-time collaboration technology expands single-point support to multi-point, cross-regional team collaboration, while transforming the entire interaction process into reusable structured knowledge. Another objective of this invention is to provide a system for implementing the above method, achieving seamless integration with industrial internet software.

[0010] The beneficial technical effects of this invention are as follows: 1. Significantly improves problem diagnosis and resolution efficiency: The customer service system can "see" what the user sees, and combined with data context, quickly and accurately locates the problem, greatly shortening the average resolution time.

[0011] 2. Provide an immersive and unambiguous guidance experience: By using AR to directly overlay guidance information onto real devices and using interface annotations to precisely point to specific controls, the guidance is as if an expert is on-site, greatly reducing the understanding threshold and the error rate.

[0012] 3. Enables efficient remote collaboration and training: Supports multiple parties to access the same guidance scenario in real time, share videos, annotate in real time, and conduct voice discussions, enabling efficient use of expert resources across factories and departments, and also serving as a powerful remote training tool.

[0013] 4. Achieve automated accumulation and reuse of industrial knowledge: The problems, diagnostic processes, solutions, and multimedia guidance materials of each successful interactive session can be structured and saved to the knowledge base. After desensitization and review, they can be retrieved and studied by other users, forming a cycle of knowledge accumulation.

[0014] 5. Achieve personalized and adaptive services: The system can identify the user's skill level and provide matching guidance depth. For common questions, it can even provide proactive prompts to answer questions before they are asked, significantly improving the satisfaction of users at different levels. Attached Figure Description

[0015] Appendix Figure 1This is a schematic diagram of the overall architecture of an industrial internet software intelligent customer service system in one embodiment of the present invention.

[0016] Appendix Figure 2 This is a core workflow diagram of the intelligent customer service interaction method described in one embodiment of the present invention.

[0017] Appendix Figure 3 This is an interactive schematic diagram of a multi-user collaborative AR guidance scenario in one embodiment of the present invention. Detailed Implementation To make the objectives, technical solutions, and advantages of this invention clearer, the following detailed description is provided with reference to the accompanying drawings and embodiments, taking the troubleshooting of a certain MES system's reporting function as an example.

[0018] I. Overall System Architecture refer to Figure 1 This system adopts a "cloud-edge-device" collaborative architecture and is deeply integrated into the industrial internet platform.

[0019] Terminal layer (engineer's side): Embed a smart customer service plugin in the MES system PC client or mobile app used by engineers. This plugin provides a "one-click help" button, can capture a screenshot of the current problem interface, record the operation process, and start a microphone.

[0020] Engineers may wear AR glasses or use tablets to photograph on-site equipment (such as work reporting terminals and machine tools).

[0021] Edge / cloud service layer: Multimodal access gateway: Receives multiple audio and video streams, screenshots, and metadata (such as user ID and software module) from the terminal.

[0022] Industrial Knowledge Augmented Analysis Engine: Core Module. It uses visual models to identify error pop-ups and blank data areas in screenshots; it understands engineers' voice descriptions through ASR and NLU; and it retrieves real-time status and reporting history of relevant equipment from MES and SCADA systems using device identifiers.

[0023] Digital twin and AR services: Manage 3D models of devices and provide spatial positioning and rendering services for AR guidance.

[0024] Collaborative Session Management: When remote expert support is requested, this module is responsible for establishing and maintaining multi-person audio and video calls, shared whiteboards (annotation interfaces), and synchronized virtual expert views.

[0025] Knowledge Data Layer: Industrial software knowledge graph: stores the logical relationships between various functional modules of MES, common alarm cause chains, and standard operating procedures.

[0026] Case study library and multimedia resource library: Stores historical failure cases, standard operation videos, and equipment 3D animations.

[0027] User profiles and skill models: Record engineers' processing preferences, modules they are good at, and their historical learning records.

[0028] II. Core Process of Intelligent Customer Service Interaction Method refer to Figure 2 In a typical troubleshooting request, this method operates according to the following process: Phase 1: Multimodal Problem Capture and Enhanced Diagnosis 1. Scenario Trigger: Workshop operator Zhang is unable to submit data on the MES work reporting interface and clicks the "Assist" button. The system automatically captures a screenshot of the current error interface (displaying "PLC communication timeout") and starts recording.

[0029] 2. Multi-source information input: Zhang said to Kefeng, "Wang, my machine tool (equipment ID: CNC-03) keeps failing to report work, indicating a PLC communication problem. It was working fine just now." At the same time, he used the tablet's camera to scan the machine tool control cabinet.

[0030] 3. Enhanced understanding through industrial knowledge: Voice / Text Understanding: The system identifies key entities: User Character = Operator, Equipment = CNC-03, Fault Phenomenon = PLC Communication Timeout, Time Characteristic = Sudden.

[0031] Visual understanding of the interface: The error code "ERR-2021" in the screenshot was identified and associated with the knowledge graph, and it was initially determined to belong to the category of "workshop network - equipment layer communication interruption".

[0032] Working condition data correlation: The system immediately queries the real-time status of the CNC-03 equipment: the last update time of the PLC signal (2 minutes ago) and the current machine tool operating mode (stopped). At the same time, the workshop network monitoring system is checked, and it is found that the network segment where the equipment is located has a brief fluctuation record 5 minutes ago.

[0033] User skill assessment: Based on Mr. Zhang's historical records, the system determines that he is a skilled operator but not a network expert.

[0034] Integrated Diagnosis: Based on the above information, the reason with the highest confidence level in the system diagnosis is: the connection between the PLC and the MES server of CNC-03 was interrupted due to fluctuations in the workshop network. The network connection status of this device needs to be checked.

[0035] Phase Two: Generation of Personalized and Contextualized Response Strategies Based on the diagnostic results and user profile (skilled operator), the system generates a response strategy: 1. Solution: Provide step-by-step guidance on "on-site inspection of network physical connections".

[0036] 2. Voice strategy: Use clear, affirmative, and directive language, and avoid excessive explanations of network principles.

[0037] 3. Image Strategy: The virtual expert appears as a capable engineer, using instructive gestures such as "check" and "confirm".

[0038] 4. Visual strategies: AR guidance: On the screen of the machine tool control cabinet captured by Zhang Gong's tablet camera, a flashing red arrow is superimposed, pointing to the PLC's Ethernet port.

[0039] Interface annotation: On the shared screenshot of the MES software error pop-up, highlight the "Retry" button and add the annotation "Please complete the physical check before clicking".

[0040] Push notification: Link and push a 30-second standard operating procedure video on "How to check the network cable connection of a machine tool".

[0041] Phase 3: Multimodal Collaborative Guidance and Execution 1. Initiating a collaborative session: The system prompts Mr. Zhang whether he needs to invite equipment maintenance worker Mr. Li (network expert) to join the session. After Mr. Zhang confirms, Mr. Li receives the invitation on his PC and joins.

[0042] 2. Multimodal synchronization guidance: The virtual expert spoke to Mr. Zhang via voice: "Mr. Zhang, please follow the AR arrow to find the PLC network port of CNC-03 and check if the network cable indicator light is flashing normally." On Zhang's tablet, the AR arrow steadily pointed to the network port. He reported, "The light isn't on!" At this moment, Engineer Li, who was remotely monitoring the situation, saw the same thing on the shared AR video screen. Engineer Li directly drew a circle next to the network port on his own screen with a stylus and added in voice: "Engineer Zhang, check if the other end of the network cable is loose from the switch. You can trace back along the network cable." The virtual expert provides synchronized guidance, and a highlighted virtual cable extends from the network port onto the AR screen, pointing towards the switch.

[0043] 3. Problem Solving and Confirmation: Mr. Zhang discovered that the network cable at the switch end was loose. After tightening it, the indicator light returned to normal. He clicked "Retry" in the MES interface and successfully reported the problem. The system automatically recorded the problem resolution status.

[0044] Phase Four: Knowledge Accumulation and System Optimization 1. Automatic Case Generation: The entire process of this interaction (original input, diagnostic analysis, solution, AR annotation record, participants) is automatically generated into a structured case draft and stored in the case library. Keywords include: "PLC communication timeout", "network loose", "CNC-03", "AR guidance".

[0045] 2. Model and Strategy Optimization: The effectiveness of the successful "AR-guided network cable check" strategy has been recorded. When similar network problems are triggered again by novice users, the system can prioritize recommending this strategy.

[0046] 3. User profile update: The performance of Engineer Zhang and Engineer Li in this collaboration was recorded, enriching their skill profiles.

[0047] Through the above process, a complex field device communication problem can be solved in just a few minutes through intuitive AR guidance and efficient remote collaboration, and the experience can be saved for future users to learn from.

Claims

1. A method for intelligent customer service interaction in industrial internet software based on multimodal deep fusion, characterized in that, Includes the following steps: S1: Through the multimodal interface integrated into the industrial internet software client, it can simultaneously capture the user's problem description in voice or text, screenshots or screen recordings of the current software interface, and optional on-site video streams of actual equipment. S2: Utilizing an industrial knowledge-enhanced multimodal understanding module, the system performs joint analysis of the problem description, interface status, and on-site video. By combining software operation logs and real-time equipment data, it accurately diagnoses the root cause of the problem and identifies the user's identity and proficiency. S3: Based on the problem diagnosis results and user profiles, the multimodal response engine retrieves or dynamically generates a solution strategy containing step descriptions and expected results from the solution library, and plans the expression methods and auxiliary media of the virtual expert; S4: In the user software interface, launch and drive a 3D virtual expert avatar, and simultaneously perform the following: provide explanations with voice and level of detail adapted to the user's avatar; add visual annotations such as highlighting and circling on the software interface screenshots; push relevant operation videos or 3D animation demonstrations; and on AR-enabled devices, overlay operation instructions onto the real device view. S5: During the guidance process, it supports multi-user video synchronization and interaction, allowing remote experts to intervene or users in different positions (such as operators and process engineers) to share the same guidance session and conduct real-time annotation and discussion, forming a collaborative troubleshooting capability.

2. The method according to claim 1, characterized in that, The "industrial knowledge-enhanced multimodal understanding module" in step S2 specifically includes: The industrial terminology recognition and disambiguation submodule accurately understands the technical terms such as equipment models, process parameters, and alarm codes involved in user descriptions. The interface element recognition submodule uses computer vision technology to identify controls, charts, data tables, and alarm information statuses in user-provided software interface screenshots. The operating condition context association submodule connects to a real-time database to obtain the current operating parameters, historical alarm records and production batch information of relevant equipment, providing data context for problem diagnosis. The user skills assessment submodule dynamically evaluates a user's professional level based on their historical query records, operational complexity, and current questioning method, classifying them into modes such as novice, skilled worker, or expert.

3. The method according to claim 1, characterized in that, The "solution strategy planning" in step S3 includes: Based on the type of problem diagnosed, standard processing procedures, contingency plans, or best practices are matched from the structured industrial knowledge graph; Based on the user's skill assessment results, the granularity of the solution explanation is automatically adjusted: providing detailed basic operation steps and principle explanations for beginners; and focusing on key parameter adjustments and advanced troubleshooting approaches for experts. For complex or high-risk operations, automatically associate and push key safety specifications, standard operating procedure video clips, or historical similar case reports.

4. The method according to claim 1, characterized in that, The "multimodal response presentation" in step S4 specifically includes: The virtual expert avatar-driven submodule calls up a library of professional gestures (such as pointing, observing, thinking, etc.) that match the engineer's image, based on the content being explained and the industrial scenario. The Augmented Reality Registration and Rendering submodule accurately overlays guidance information such as 3D arrows, disassembly animations, and data labels onto the corresponding components of the physical device when the user points the AR glasses or mobile device camera at the actual device. The interface collaborative annotation submodule allows virtual experts and users (or remote experts) to draw and write on a shared software interface or device image in real time, and the annotation content of all participants is visible in real time.

5. An industrial internet software intelligent customer service system for implementing the method of any one of claims 1-4, characterized in that, include: The client-side enhanced interaction plugin is embedded in industrial internet software, providing screen capture, voice input, AR vision, and multi-user conversational interface. The cloud-based multimodal intelligent analysis platform includes the aforementioned industrial knowledge-enhanced multimodal understanding module, solution knowledge graph, and multimodal response engine; An industrial digital twin model library stores 3D models of key equipment, standard operation animations, and fault simulation animations. The collaborative session management server is responsible for creating and maintaining multi-user guidance sessions, synchronizing voice and video streams, annotation information, and virtual expert status to enable immersive collaborative work across regions.

Images