Device trouble shooting method, device, device and program product

Abstract

The application discloses a device troubleshooting method, device, equipment and program product, solves the problems of insufficient information utilization, fixed diagnosis process and inability to automatically execute long-chain reasoning in the existing troubleshooting scheme. The scheme includes: performing device entity recognition on the multi-modal description information of the user input device for reporting a fault, obtaining the device attribute information of the device for reporting a fault. Retrieve the standard working state information corresponding to the device attribute information from the device knowledge base, and compare the multi-modal description information with the standard working state information to obtain the abnormal description information of the device for reporting a fault. Retrieve the diagnosis knowledge corresponding to the abnormal description information from the fault diagnosis knowledge base and / or the fault diagnosis historical case base to generate the diagnosis process of the abnormal description information. Configure the diagnosis process as a diagnosis chain cascaded by at least two diagnosis task nodes, execute the diagnosis task nodes in the diagnosis chain in path order, obtain all diagnosis results, generate and output a fault diagnosis report for the device for reporting a fault.

Description

Technical Field

[0001] This application relates to the field of equipment fault detection technology, and in particular to a method, apparatus, equipment and program product for troubleshooting equipment. Background Technology

[0002] Current troubleshooting technologies primarily include human customer service, rule-based or keyword-based question-and-answer systems, and machine learning models that rely on single-modal data (such as text or images only). However, these methods all have significant limitations: human customer service is inefficient, heavily reliant on the personal experience of customer service personnel, and difficult to scale and standardize; rule-based or keyword-based question-and-answer systems are limited by preset logic, unable to accurately understand the complex semantics in user descriptions, and even less able to support multi-step, process-oriented troubleshooting; while single-modal machine learning models lack the ability to fuse multi-source information, making it difficult to handle fault descriptions interwoven with images and text in real-world scenarios, and their single-point judgment mechanism cannot support cross-step, reasoning-based systematic diagnosis. Summary of the Invention

[0003] This application proposes a troubleshooting method, apparatus, equipment, and program product, aiming to solve the problems of insufficient information utilization, rigid diagnostic processes, and inability to automate long-chain reasoning in existing troubleshooting solutions. Accordingly, the technical solution of this application is as follows: In a first aspect, embodiments of this application provide a device troubleshooting method applied to large-scale models, including: The device entity recognition is performed on the multimodal description information of the reported fault device input by the user to obtain the device attribute information of the reported fault device; Retrieve standard operating status information corresponding to the device attribute information from the device knowledge base, and compare the multimodal description information with the standard operating status information to obtain the abnormal description information corresponding to the faulty device; Retrieve diagnostic knowledge corresponding to the anomaly description information from the fault diagnosis knowledge base and / or fault diagnosis historical case base to generate a diagnostic process corresponding to the anomaly description information; The diagnostic process is configured as a diagnostic chain consisting of at least two cascaded diagnostic task nodes, so that the diagnostic task nodes in the diagnostic chain are executed in the order of the path to obtain the diagnostic results of at least two diagnostic task nodes. Based on all the diagnostic results obtained, a fault diagnosis report for the reported faulty device is generated and output.

[0004] Secondly, embodiments of this application provide a device for troubleshooting equipment applied to large models, comprising: The device attribute recognition module is used to perform device entity recognition on the multimodal description information of the reported fault device input by the user, and obtain the device attribute information of the reported fault device; The anomaly description determination module retrieves standard operating status information corresponding to the device attribute information from the device knowledge base, and compares the multimodal description information with the standard operating status information to obtain the anomaly description information corresponding to the faulty device. The diagnostic process determination module is used to retrieve diagnostic knowledge corresponding to the abnormal description information from the fault diagnosis knowledge base and / or the fault diagnosis historical case base, so as to generate the diagnostic process corresponding to the abnormal description information. The diagnostic task execution module is used to configure the diagnostic process as a diagnostic chain consisting of at least two cascaded diagnostic task nodes, so as to execute the diagnostic task nodes in the diagnostic chain in the path order and obtain the diagnostic results of at least two diagnostic task nodes. The diagnostic report generation module is used to generate and output a fault diagnosis report for the reported faulty device based on all the obtained diagnostic results.

[0005] Thirdly, embodiments of this application provide an electronic device, including: a processor; and a memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the method described in the first aspect.

[0006] Fourthly, embodiments of this application provide a computer program product, the computer program product including a computer-readable storage medium storing a computer program operable to cause a computer to perform the method described in the first aspect.

[0007] In this embodiment of the application, the solution firstly performs device entity recognition by fusing multi-source information such as text and images of the reported faulty device input by the user, overcoming the limitations of single-modal analysis and achieving accurate identification of device attribute information. Next, it retrieves standard operating status information corresponding to the device attribute information from the device knowledge base and compares it with the multi-modal description information to obtain structured abnormal description information of the reported faulty device, thereby replacing manual experience-based judgment. Then, it retrieves diagnostic knowledge corresponding to the abnormal description information from the fault diagnosis knowledge base and / or fault diagnosis historical case database, and constructs a diagnostic process reflecting the diagnostic path accordingly. This approach addresses the issue that traditional fixed diagnostic processes cannot adapt to complex and ever-changing fault scenarios. Instead, it configures the diagnostic process as a cascaded chain of at least two diagnostic task nodes, executing these nodes sequentially to obtain the corresponding diagnostic results. This execution process allows for path jumps based on intermediate diagnostic results, essentially simulating human chain-like reasoning to automate the complex multi-step, multi-branch troubleshooting process. Finally, it integrates all diagnostic results to generate a fault diagnosis report for the reported device, completing an intelligent closed loop from multimodal information input to complete diagnostic conclusion output, significantly improving troubleshooting accuracy and efficiency. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 This is a schematic diagram of the first process of the device troubleshooting method according to an embodiment of this application.

[0010] Figure 2 This is a schematic diagram of the second process of the device troubleshooting method according to an embodiment of this application.

[0011] Figure 3 The architecture intent for performing diagnostics in the device troubleshooting method of this application embodiment.

[0012] Figure 4 This is a schematic diagram of the device troubleshooting apparatus according to an embodiment of this application.

[0013] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0014] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0015] Current equipment troubleshooting technologies primarily include human customer service, rule-based or keyword-matching question-and-answer systems, and machine learning models that rely on single-modal data (such as text or images only). However, these methods all have significant limitations: human customer service is inefficient, highly dependent on the personal experience and real-time status of customer service personnel, making it difficult to achieve scalable and standardized services, and incurring high training costs and slow response times; rule-based or keyword-matching question-and-answer systems are limited by preset logic and fixed templates, making it difficult to accurately understand the complex semantics, contextual intent, or information evolution in multi-turn dialogues in user descriptions, and even more difficult to support multi-step, process-oriented troubleshooting, resulting in poor adaptability and low fault tolerance in dynamically changing real-world fault scenarios; while single-modal machine learning models lack the ability to fuse multi-source information, making it difficult to cope with complex fault representations in real-world scenarios where text and images are intertwined and states and descriptions coexist, and their "single-point judgment" mechanism cannot support cross-step, reasoning-based systematic diagnosis, often only able to complete isolated classification or matching tasks under limited conditions, and unable to form a coherent and interpretable troubleshooting logic chain.

[0016] In view of this, embodiments of this application provide a device troubleshooting method, apparatus, equipment, and program product, aiming to systematically solve a series of key problems in existing troubleshooting solutions, such as insufficient utilization of multimodal information, rigid and inflexible diagnostic processes, and the inability to automate long-chain reasoning with conditional branches and state evolution. The technical solutions provided by the various embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0017] One embodiment of this application provides a device troubleshooting method based on a large model (intelligent agent). Its core lies in leveraging the powerful multimodal understanding, semantic reasoning, and sequence generation capabilities of the large model. The system can deeply integrate multi-source information such as text and images, simulating the diagnostic thinking of human experts, transforming static knowledge into a dynamic and executable reasoning chain, thereby achieving end-to-end intelligent fault diagnosis for complex faults. Figure 1 This is a flowchart illustrating the troubleshooting method for this equipment, including: S101, perform device entity recognition on the multimodal description information of the reported fault device input by the user to obtain the device attribute information of the reported fault device.

[0018] The core significance of this step lies in using a large-scale model to accurately identify the specific device reported by the user, thus providing a crucial identity index for all subsequent diagnostic steps. In complex scenarios such as those involving telecom operators, facing various service types like broadband, television, and smart home, as well as a massive number of terminal devices of different brands and models, the rules, status standards, and solutions relied upon for fault diagnosis all strictly correspond to specific device entities. In this step, the large-scale model accurately identifies the device's service type, device type, brand, and model, essentially anchoring the entire troubleshooting process to a single correct diagnostic target. This identification result is the fundamental prerequisite for the correct execution of subsequent steps: only by clearly identifying "which device" can the system retrieve matching "standard operating status information" from the device knowledge base for comparison, and recall targeted "diagnostic knowledge" from the fault diagnosis knowledge base or historical case database to construct the process. If the device identification is incorrect or ambiguous, subsequent fault comparisons will lack a benchmark, knowledge retrieval will lose direction, and the entire diagnostic reasoning chain will be built on an incorrect foundation, inevitably leading to misjudgment or falling into an inefficient cycle. Therefore, this step, by transforming unstructured multimodal inputs into standardized device attributes, achieves a crucial leap from "unknown" to "known" in diagnostics, laying an indispensable foundation for subsequent accurate and efficient automated diagnostics.

[0019] In terms of specific implementation, this embodiment relies on a large model to execute the entire process of device entity recognition. Before performing deep fusion analysis on the multimodal description information input by the user, the large model proactively performs compliance checks on the images of reported faulty devices contained therein, as a prerequisite for initiating deep recognition analysis. This compliance check is an inherent part of the large model's workflow and mainly includes at least one of brightness detection, sharpness detection, and content compliance detection. Brightness detection analyzes the brightness statistical characteristics of image pixels (such as mean and standard deviation) and compares them with a preset reasonable range to determine whether the image is overexposed or underexposed; sharpness detection typically uses methods such as gradient calculation to assess the degree of image blur; and content compliance detection quickly screens whether the image contains illegal or sensitive content unrelated to fault diagnosis.

[0020] If the large-scale model determines that the image of the reported device fails the aforementioned compliance check, it will temporarily suspend the execution of deep device entity recognition and generate specific reasons for the failure and clear guidance information for re-collection based on the detection results, which will be fed back to the user through the user terminal. For example, the large-scale model may generate and output a prompt: "The light in the photo you uploaded is too dark, and the indicator light cannot be seen clearly. Please adjust the lighting and retake a photo of the front of the device." The significance of this mechanism is that the large-scale model actively verifies the input quality, ensuring the effectiveness of the visual information on which it bases its recognition and reasoning from the source. Low-quality or irrelevant images will directly affect the accuracy of the large-scale model's feature extraction and understanding, thereby threatening the reliability of the entire troubleshooting process. By internalizing compliance verification as a pre-process of the large-scale model's workflow and giving it interactive guidance capabilities, this embodiment ensures that the large-scale model only performs high-precision device entity recognition and subsequent diagnosis after obtaining high-quality, highly relevant visual input, thus laying a reliable data foundation for the entire automated troubleshooting process in the initial stage and avoiding reasoning failures or incorrect conclusions due to input quality issues.

[0021] S102, retrieve the standard operating status information of the corresponding equipment attribute information from the equipment knowledge base, and compare the multimodal description information with the standard operating status information to obtain the abnormal description information corresponding to the faulty equipment. The significance of this step is to establish an objective and quantifiable fault judgment benchmark for the specific equipment entities identified in S101, thereby transforming the user's subjective and general fault description into anomalies that the system can accurately analyze and process. During troubleshooting, simply knowing "what equipment it is" is insufficient to locate the problem; the key is to clearly define "what specific deviations the current state of the equipment has from its normal state." This embodiment introduces pre-stored, authoritative "standard operating status information" from the equipment knowledge base as a reference benchmark, enabling the large model to perform standardized comparison operations rather than relying on vague experience-based judgments. Through this comparison, the system can automatically and structurally output anomaly description information, clearly indicating which component and attribute has experienced what type of anomaly. This provides direct and reliable input for subsequent diagnostic processes, representing a crucial transformation from "phenomenon perception" to "problem definition."

[0022] Here, the standard operating status information referred to in this article refers to the standardized descriptions and reference data pre-built in the device knowledge base and strictly bound to specific device attribute information (such as brand and model). It defines the state that the device should be in when there are no faults. For example, for a specific model of optical modem, its standard operating status information may include: the normal color and status (solid green / off) of each indicator light (such as PON light and LOS light), a standard image of the intact device casing, and a connection specification description of key interfaces (such as fiber optic interfaces). The anomaly description information, on the other hand, is a structured difference report generated by automatically comparing the user-provided multimodal description information (such as a real-life image showing a solid red light) with the above standard operating status information. It systematically describes the inconsistencies between observed values ​​and standard values. For example, in the device indicator light area, the LOS light is abnormal; the standard state should be off, but the observed state is solid red. This structured anomaly description provides precise targets for fault location.

[0023] In terms of specific implementation, this embodiment mainly relies on a multimodal large model to complete the comparison and generation tasks. First, the system uses the device attribute information obtained in S101 as the retrieval key to recall the corresponding standard operating status information from the device knowledge base. This information typically includes structured knowledge such as standard status images and standard status text description templates. Subsequently, the large model takes the multimodal description information (including images and text) provided by the user and the retrieved standard operating status information as input. Through its visual and semantic understanding capabilities, the large model performs fine-grained cross-modal comparative analysis. For example, it compares the visual features of the device images uploaded by the user with standard images at the level of specified components (such as indicator lights and interfaces), while combining the user's text description for semantic verification and supplementation. This embodiment guides the large model to strictly follow the structured framework of "entity-component-attribute-standard value-observation value-difference description" for analysis and output by designing specific prompt word templates, ensuring that the generated abnormal description information has both machine-readable standardization and human-understandable clarity. The entire comparative diagnostic process essentially involves fusing and reasoning the prior standards in the equipment knowledge base with the real-time observation data provided by the user in a large model, ultimately outputting an accurate and structured anomaly report, laying the factual foundation for subsequent processes.

[0024] S103, retrieve diagnostic knowledge corresponding to the anomaly description information from the fault diagnosis knowledge base and / or fault diagnosis historical case base to generate the diagnostic process corresponding to the anomaly description information.

[0025] The core significance of this step lies in dynamically transforming the static factual description of where the device malfunctions, output by S102, into an executable action plan guiding the system on "how to troubleshoot and resolve this malfunction step by step." A single malfunction description alone cannot directly lead to a solution; it needs to be placed within specific troubleshooting logic and context. This embodiment retrieves diagnostic knowledge highly relevant to the current malfunction from the existing knowledge system and utilizes the generation and logical organization capabilities of a large model to construct a structured diagnostic process, thereby achieving a leap from "problem definition" to "solution path planning." This avoids the inefficiency and lag of manually writing fixed procedures for each new fault scenario, enabling the system to automatically and intelligently generate targeted diagnostic solutions based on existing knowledge.

[0026] Here, the fault diagnosis knowledge base refers to a database that stores systematic knowledge such as structured diagnostic rules, expert experience, and equipment parameter thresholds. For example, it might contain rule-based diagnostic manuals such as "If the LOS light on the optical modem is red, check the fiber optic connection, restart the device, and check the area's optical path status in sequence." The fault diagnosis historical case database stores past fault work order records that have actually occurred and been resolved. Each case typically includes a description of the fault phenomenon (similar to anomaly description information), the sequence of troubleshooting steps performed, and the final solution and result. "Diagnostic knowledge" is a collective term for all information fragments retrieved from the aforementioned knowledge base or case database that are related to the current anomaly description information. This can be structured rule clauses or unstructured text cases. For example, for the anomaly "LOS light is constantly red," the retrieved diagnostic knowledge might include a rule text from the knowledge base: "Check if the fiber optic connector is loose or bent," and a record from the historical case database: "Case ID: X, similar phenomenon, previously resolved by re-plugging and unplugging the user-end fiber optic connector."

[0027] In terms of specific implementation, this embodiment adopts a retrieval-enhanced generation technology framework. First, the system converts the structured anomaly description information obtained in S102 into a query vector and performs semantic retrieval in the fault diagnosis knowledge base and the fault diagnosis historical case database. The retrieval process is achieved by calculating the similarity between the query vector and the knowledge entry vector in the database, aiming to recall the set of diagnostic knowledge fragments most relevant to the current anomaly. These retrieved knowledge fragments, which may come from different sources and have loose formats, are then input into the large model along with the original anomaly description information. This embodiment guides the large model to play the role of a "troubleshooting process planning expert" by designing specific prompts. Based on its deep understanding of natural language and troubleshooting logic, the large model identifies, integrates, sorts, and logically connects the retrieved diagnostic knowledge, ultimately generating a structured, step-by-step diagnostic process. This process is usually presented in the form of a list or node diagram, clearly defining the judgment content, expected results, and possible branching relationships for each step, thereby decomposing a complex diagnostic problem into a series of operable tasks that can be executed sequentially or conditionally, preparing for the next step to transform it into an executable diagnostic chain.

[0028] S104, configure the diagnostic process as a diagnostic chain consisting of at least two cascaded diagnostic task nodes, so as to execute the diagnostic task nodes in the diagnostic chain in the path order and obtain the diagnostic results of at least two diagnostic task nodes.

[0029] The significance of this step is to transform the diagnostic process generated in S103 as a static action plan into a computational logic structure that can be dynamically driven and executed by the system, thereby achieving complete automation of the troubleshooting process. A diagnostic process described in text cannot truly replace manual intervention if it cannot be understood and executed step by step by the system. Therefore, this embodiment introduces a diagnostic chain as a structured execution carrier, instantiating each logical step in the process as an independent diagnostic task node, and organizing these nodes through preset cascading relationships (such as sequential connections or conditional branches). This allows the system to proceed along the path defined by the diagnostic chain like a strict automaton, executing node tasks one by one, acquiring and recording diagnostic results, and dynamically determining subsequent paths based on intermediate results. This fundamentally solves the problem that fixed scripts or question-answering robots cannot handle complex reasoning processes with multiple steps and branches, realizing the operational simulation of expert chain thinking.

[0030] Here, a diagnostic task node is the smallest independently executable logical unit in the diagnostic chain. It encapsulates a specific diagnostic action or judgment, such as "determining if the fiber optic connector is loose" or "restarting the device and observing changes in its status." Each node typically contains the content to be executed, available tool interfaces, and rules for determining its completion status and subsequent steps. Cascading describes the connection relationships and execution order logic between these nodes. It includes not only simple linear sequences but, more importantly, supports conditional branch jumps based on the execution results of the previous node, thus forming branch paths that can handle different troubleshooting results. A diagnostic chain is a directed, executable task graph composed of at least two such diagnostic task nodes connected by cascading relationships. For example, for a diagnostic process of "optical modem LOS light red," its configured diagnostic chain might start at node 1: "ask the user if they have recently moved the device." If the user answers "yes," it cascades to node 2: "instruct the user to check if the fiber optic cable is bent"; if the answer is "no," it cascades to node 3: "query the optical path alarm information for this area." Each node will generate a clear "diagnostic result" (such as "user confirms no fiber optic bends" or "area optical path status is normal") after execution, serving as an intermediate conclusion in the chain-like reasoning.

[0031] In terms of specific implementation, this embodiment uses a process execution engine to configure and drive the diagnostic chain. First, the engine parses the structured diagnostic process description generated by S103, identifying the steps, judgment logic, and branch conditions. Then, it creates a corresponding diagnostic task node object for each step and establishes a cascading relationship between nodes based on the logic defined in the process, using fields such as node ID and jump conditions, thereby constructing the diagnostic chain data structure in memory. When the diagnostic chain starts execution, the engine starts from the initial node and calls the large model to analyze the task objective of the current node. For nodes that require interaction with the outside world (such as querying the user or querying the database), the large model generates specific interaction content (such as natural language questions or query instructions) and executes it through system calls to the corresponding tool interfaces; for purely judgmental nodes, the large model performs analysis and reasoning based on existing context information. After a node is executed, the engine obtains its output as the diagnostic result of that node and matches this result with the preset branch conditions of the current node to determine the next node to be executed. This iterative process continues until a termination state is reached where no further transition to a new node is required (such as outputting the final fault point or solution), thus completing the entire diagnostic chain and collecting the diagnostic results from all nodes along the way. This provides a complete record of the reasoning process and a chain of evidence for generating the final report.

[0032] S105, based on all the diagnostic results obtained, generates and outputs a fault diagnosis report for the reported faulty device.

[0033] The significance of this step lies in summarizing, integrating, and ultimately outputting the entire chain-like reasoning process automatically executed in the preceding steps, forming a complete, coherent, and actionable troubleshooting conclusion, thereby completing the end-to-end closed loop from fault reporting to solution delivery. The execution results of each diagnostic task node in the diagnostic chain are scattered, phased intermediate conclusions and evidence. This embodiment uses a large model to perform global analysis, logical connection, and language organization of all diagnostic results, integrating them into a structured fault diagnosis report. This report not only clearly presents the user with the final determined root cause of the fault, suggested solutions, and operational guidelines, but also provides a standardized record containing the complete reasoning path and evidence chain for possible subsequent manual review or work order archiving, representing the ultimate embodiment of the value output of intelligent troubleshooting.

[0034] In terms of specific implementation, the fault diagnosis report generation in this embodiment is a natural termination and organic component of the diagnostic chain execution process. The entire execution process dynamically embodies two key technical mechanisms. First, when executing and determining the diagnostic results of non-tail-level target diagnostic task nodes in the diagnostic chain, the system automatically determines the next level of diagnostic task node to be executed based on all currently obtained diagnostic results and the preset branch conditions of that target diagnostic task node. For example, when the diagnostic result of the node "Check if the optical fiber is bent" is "No", according to its branch conditions, the system will automatically jump to the next node "Query the area optical path alarm status", instead of executing other irrelevant steps. Second, when executing any target diagnostic task node in the diagnostic chain, the large model will actively analyze and determine whether the information on which the execution of that node depends is complete; if there is missing information (such as needing the user to confirm whether the device has been restarted), the large model will generate a natural query statement or call the corresponding query interface to actively initiate interaction with the user or external system to obtain the missing information, ensuring that the diagnosis of each node is performed under the premise of sufficient information. When the diagnostic chain is completed, the diagnostic results of all nodes have been obtained. At this point, the large model is invoked again, its input being the execution trajectory of the entire diagnostic chain, the diagnostic results of each node, and the original device attributes and anomaly information. Guided by specific prompts, the large model summarizes, refines, and formats this information to generate a final report. The report typically includes: an overview of the faulty device, a summary of the anomaly, a brief description of the diagnostic reasoning process, the identified root cause of the fault, and specific solutions or operational steps.

[0035] Furthermore, this embodiment incorporates a continuous learning feedback and optimization loop after generating and outputting the fault diagnosis report. The system can receive feedback from users or maintenance personnel regarding the fault diagnosis report. If the feedback confirms the accuracy of the diagnosis report or that the reported equipment problem has been effectively resolved, this embodiment will automatically generate a new, structured fault diagnosis history case based on the key data generated throughout the troubleshooting process—including equipment entity attribute information, anomaly description information, diagnostic process, and the final fault diagnosis report. This new case fully encapsulates all elements from the phenomenon to the solution. Subsequently, the system automatically adds this new fault diagnosis history case to the fault diagnosis history case library. This mechanism allows the system's knowledge base to continuously enrich and evolve as more cases are processed, enabling subsequent searches for similar faults to retrieve more accurate and relevant cases, thus achieving continuous self-enhancement of diagnostic capabilities.

[0036] Figure 2 This embodiment illustrates the device troubleshooting process. The entire process is driven and coordinated by an intelligent agent (large model scheduler) as the central controller, which calls upon the large model as the core capability module to execute specific cognitive and generative tasks. Specifically, the intelligent agent first guides the user to input multimodal descriptive information including images and text, then actively calls an image detection tool to complete compliance checks and standardized preprocessing. After ensuring input quality, the intelligent agent schedules the multimodal large model to perform device entity recognition to determine the specific attributes of the reported device. It then calls upon the large model again to perform a comparative diagnosis between the user's image and a standard image, generating a structured anomaly description. Based on this, the intelligent agent retrieves relevant knowledge bases and constructs an executable troubleshooting process node table through the planning and generation capabilities of the large model. In the core reasoning phase, the intelligent agent dynamically schedules the semantic understanding and dialogue generation capabilities of the large model according to the process nodes, actively polling the user to supplement information and progressively advancing chain-like reasoning. Finally, the intelligent agent organizes the large model to integrate all interactive information and retrieved knowledge, generating and outputting a complete diagnostic report. In this collaborative architecture, the intelligent agent is responsible for process control, tool invocation, and status judgment, serving as the decision-making and scheduling center of the system; while the large model provides core intelligence such as cross-modal understanding, logical reasoning, and content generation, serving as the system's cognitive and execution engine. The two work closely together to automate and intelligentize the troubleshooting process.

[0037] Figure 3This embodiment illustrates the specific implementation architecture of the intelligent agent's dynamic driving and hierarchical management of the troubleshooting process based on a thought chain. The process begins with the intelligent agent's process management module, which breaks down the overall troubleshooting task into four core stages: "multimodal information acquisition, multimodal information processing, troubleshooting reasoning, and diagnosis and resolution," and is responsible for the scheduling and status monitoring of the entire chain. At the task decision layer, taking the troubleshooting reasoning stage as an example, the intelligent agent first calls a large model to generate a diagnostic process based on retrieved diagnostic knowledge, and analyzes and compares the multimodal description information with standard working status information to obtain abnormal description information, thereby deciding on the specific diagnostic task node to be executed. If the information is insufficient, it dynamically supplements and corrects the information by identifying missing information and generating guiding text. At the task execution layer, the intelligent agent schedules tools such as retrieval enhancement generation technology, the multimodal large model, a diagnostic knowledge query module for retrieving information from the knowledge base, and a front-end module for user interaction to execute specific instructions issued by the decision layer, such as retrieving knowledge from the fault diagnosis knowledge base, generating user queries to obtain missing information, or performing device entity identification and status comparison. Finally, at the task evaluation and feedback layer, the agent evaluates the diagnostic results of the diagnostic task nodes according to preset task completion rules: if the task is determined to be completed, feedback is sent to the process control module to proceed to the next stage; if not completed, the process is guided to continue iterating in the current stage until the stage goal is achieved. This layered, feedback-based dynamic control mechanism ensures the flexibility and reliability of complex troubleshooting processes.

[0038] In summary, the method in this embodiment first performs device entity recognition by fusing multi-source information, such as text and images of the reported faulty device input by the user, overcoming the limitations of single-modal analysis and achieving accurate identification of device attribute information. Next, it retrieves standard operating status information corresponding to the device attribute information from the device knowledge base and compares it with the multi-modal description information to obtain structured abnormal description information of the reported faulty device, thus replacing manual experience-based judgment. Then, it retrieves diagnostic knowledge corresponding to the abnormal description information from the fault diagnosis knowledge base and / or fault diagnosis historical case database, and constructs a diagnostic process reflecting the diagnostic path accordingly. This approach addresses the issue that traditional fixed diagnostic processes cannot adapt to complex and ever-changing fault scenarios. Instead, it configures the diagnostic process as a cascaded chain of at least two diagnostic task nodes, executing these nodes sequentially to obtain the corresponding diagnostic results. This execution process allows for path jumps based on intermediate diagnostic results, essentially simulating human chain-like reasoning to automate the complex multi-step, multi-branch troubleshooting process. Finally, it integrates all diagnostic results to generate a fault diagnosis report for the reported device, completing an intelligent closed loop from multimodal information input to complete diagnostic conclusion output, significantly improving troubleshooting accuracy and efficiency.

[0039] To enable the multimodal large language model relied upon in this embodiment to master the complete obstacle-clearing capabilities from multimodal information understanding to chain-based reasoning and decision-making, it needs to undergo systematic and professional training. This embodiment adopts a two-stage training strategy that combines multi-task joint supervised fine-tuning with reinforcement learning optimization, aiming to build a unified, accurate, and reliable intelligent agent.

[0040] Phase 1: Multi-task joint supervision and fine-tuning The large model first learns the basic mapping and generation capabilities of each step in the troubleshooting process through joint supervised fine-tuning of six core tasks. These six tasks share some model parameters, and joint training promotes knowledge transfer and capability synergy.

[0041] The first task (device entity recognition): its training data includes multimodal descriptions of samples and corresponding ground truth entity attribute information and supervision labels. Let the training dataset be... Each sample is a triple. I is the input image. For natural language problems, (The following output will be a consistent representation of the true answer, including attributes such as business type, equipment type, brand, and model.) The supervised fine-tuning loss function for this task... The goal of minimizing the difference between the model's predicted answer and the true answer can be expressed as: Where θ represents the model parameters, and the summation iterates through the dataset. .

[0042] The second task (anomaly description information generation): its training data includes sample anomaly description information and corresponding ground truth anomaly description information, along with supervision labels. This task aims to train the model to compare the standard state with the observed state and generate structured difference descriptions. Let the training dataset be... Each sample contains a standard state image. Device attribute text User input image User issues and truth value answer (Includes status description) (and anomaly summary E). Its loss function Defined as: .

[0043] The third task (diagnostic chain generation): its training data includes sample diagnostic procedures and corresponding ground truth diagnostic chain supervision labels. Let the dataset be... The sample input contains a structured state description. Summary of Abnormal States Troubleshooting Knowledge Base Content Historical Case References and images The output truth value is the diagnostic process. Its loss function for: .

[0044] The fourth task (execution of diagnostic task nodes): its training data includes sample diagnostic task nodes and corresponding ground truth diagnostic results supervision labels. Let the training dataset be... The sample input includes the current node. List of available tools Intermediate information and user issues The output truth value is the decision or result. Its loss function .

[0045] The fifth task (Missing Information Guided Text Generation): Its training data includes sample missing information and corresponding ground truth guided text supervision labels, used to train the model to generate proactive queries. Let the training dataset be... The sample input is a question or context. The output truth value is the guiding statement. Its loss function .

[0046] The sixth task (diagnostic report generation): its training data includes sample diagnostic results and corresponding ground truth diagnostic report supervision labels. Let the training dataset be... The model is based on the summary information output by the aforementioned tasks. Generate final report Its loss function .

[0047] Total loss function of multi-task joint supervision and fine-tuning The weighted sum of the loss functions for the six tasks mentioned above is: By simultaneously optimizing this total loss, the model can establish a collaborative understanding and coherent generation capability across tasks and stages.

[0048] Phase Two: Reinforcement Learning Optimization After supervised fine-tuning lays the foundation, the large model is further optimized through reinforcement learning to improve the accuracy, logical compliance, and overall task completion of its output. At this stage, the model is treated as a policy network. .

[0049] The core of reinforcement learning is designing a comprehensive reward signal. This signal is determined based on the matching degree between the model's output on tasks one through six and their corresponding ground truth supervision labels. The matching degree is typically derived from semantic similarity. Consistency with output structure A comprehensive evaluation is conducted from multiple dimensions, including [specific dimensions]. For the task... ( =1 to 6), let the model output be The standard answer is The reward amount for this task is... It can be defined as: ,in and This refers to the weighting coefficients. The final overall reward. It is an aggregation of all related task rewards.

[0050] The goal of reinforcement learning is to tune model parameters. To maximize expected reward Training is typically performed using algorithms such as near-end policy optimization, and its policy gradient loss function... This can be formally represented as: Through iterative optimization, the model is driven to produce more accurate, reliable, and professionally compliant outputs in complex sequence decision-making and generation tasks.

[0051] In addition, corresponding to Figure 1 The method shown in this embodiment, in another embodiment, also provides a device troubleshooting apparatus applied to large models. Among them, Figure 4 This is a structural diagram of the obstacle removal device 400, including: The device attribute recognition module 410 is used to perform device entity recognition on the multimodal description information of the reported fault device input by the user, and obtain the device attribute information of the reported fault device; The anomaly description determination module 420 retrieves standard working status information corresponding to the device attribute information from the device knowledge base, and compares the multimodal description information with the standard working status information to obtain the anomaly description information corresponding to the faulty device. The diagnostic process determination module 430 is used to retrieve diagnostic knowledge corresponding to the abnormal description information from the fault diagnosis knowledge base and / or the fault diagnosis historical case base, so as to generate the diagnostic process corresponding to the abnormal description information. The diagnostic task execution module 440 is used to configure the diagnostic process as a diagnostic chain consisting of at least two cascaded diagnostic task nodes, so as to execute the diagnostic task nodes in the diagnostic chain in the path order and obtain the diagnostic results of at least two diagnostic task nodes. The diagnostic report generation module 450 is used to generate and output a fault diagnosis report for the reported faulty device based on all the obtained diagnostic results.

[0052] Optionally, the diagnostic task execution module 440 is further configured to: when executing any target diagnostic task node in the diagnostic chain, determine the missing information on which the execution of the target diagnostic task node depends, so as to obtain the missing information from the user terminal or an external system.

[0053] Optionally, the non-final-level diagnostic task nodes in the diagnostic chain are configured with branch conditions that guide at least one next-level diagnostic task node based on the current diagnostic result; the diagnostic task execution module 440 is further configured to: when executing and determining the diagnostic result of the non-tail-level target diagnostic task node in the diagnostic chain, determine the next-level diagnostic task node to be executed based on all currently obtained diagnostic results and the branch conditions set by the target diagnostic task node.

[0054] Optionally, the large model is obtained through joint training of the following tasks: a first task of performing device entity recognition, wherein the training data for the first task includes sample multimodal description information and supervision labels for indicating ground truth entity attribute information of the sample multimodal description information; a second task of performing anomaly description information generation, wherein the training data for the second task includes sample anomaly description information and supervision labels for indicating ground truth anomaly description information of the sample anomaly description information; a third task of performing diagnostic chain generation, wherein the training data for the third task includes sample diagnostic process and supervision labels for indicating ground truth diagnostic chain of the sample diagnostic process; and a fourth task of performing diagnostic task node execution. The training data for the tasks includes sample diagnosis task nodes and supervisory labels for indicating the truth value diagnosis results of the sample diagnosis task nodes; a fifth task is to generate missing information guidance text, the training data of which includes sample missing information and supervisory labels for indicating the sample missing information; a sixth task is to generate a diagnosis report, the training data of which includes sample diagnosis results and supervisory labels for indicating the sample diagnosis results; wherein, the sample multimodal description information, the sample anomaly description information, the sample diagnosis process, the sample diagnosis task nodes, the sample missing information, and the sample diagnosis results all correspond to the same device.

[0055] Optionally, the large model is further obtained through reinforcement learning optimization training, wherein the comprehensive reward signal optimized by reinforcement learning is determined based on the matching degree between the output results of the first task, the second task, the third task, the fourth task, the fifth task and the sixth task and their respective corresponding supervision labels.

[0056] Optionally, after generating and outputting a fault diagnosis report for the reported device, the diagnostic report generation module 450 is further configured to: receive feedback information regarding the fault diagnosis report; if the feedback information confirms that the fault diagnosis report is correct or that the problem of the reported device has been resolved, then based on the device entity attribute information, the anomaly description information, the diagnostic process, and the fault diagnosis report, generate a new fault diagnosis historical case; and add the new fault diagnosis historical case to the fault diagnosis historical case library.

[0057] Optionally, the multimodal description information includes an image of the reporting device; before performing device entity recognition on the multimodal description information of the reporting device input by the user, the device attribute recognition module 410 is further configured to: perform compliance detection on the image of the reporting device, the compliance detection including at least one of brightness detection, sharpness detection, and content compliance detection; if the image of the reporting device fails the compliance detection, the user terminal is prompted with the reason for the failure of the compliance detection and guidance information for re-collecting the image of the reporting device.

[0058] It should be noted that the equipment troubleshooting device in this embodiment can be used as... Figure 1 The execution body of the method shown is therefore able to achieve... Figure 1 The steps and functions of the method shown are illustrated.

[0059] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Please refer to it. Figure 5 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for other business operations.

[0060] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0061] Memory is used to store computer programs. Specifically, a computer program may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides the computer program to the processor.

[0062] Specifically, the processor reads the corresponding computer program from non-volatile memory into memory and then runs it, forming the above-mentioned logical structure. Figure 4 The device shown is a troubleshooting device. Correspondingly, the processor executes the program stored in the memory and specifically performs the following operations: The device entity recognition is performed on the multimodal description information of the reported fault device input by the user to obtain the device attribute information of the reported fault device.

[0063] The standard operating status information corresponding to the device attribute information is retrieved from the device knowledge base, and the multimodal description information is compared with the standard operating status information to obtain the abnormal description information corresponding to the faulty device. Retrieve diagnostic knowledge corresponding to the anomaly description information from the fault diagnosis knowledge base and / or fault diagnosis historical case base to generate a diagnostic process corresponding to the anomaly description information.

[0064] The diagnostic process is configured as a diagnostic chain consisting of at least two cascaded diagnostic task nodes, so that the diagnostic task nodes in the diagnostic chain are executed in the order of the path to obtain the diagnostic results of at least two diagnostic task nodes.

[0065] Based on all the diagnostic results obtained, a fault diagnosis report for the reported faulty device is generated and output.

[0066] The above is as described in this instruction manual. Figure 1The device troubleshooting method disclosed in the illustrated embodiments can be applied to a processor and implemented by the processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0067] Of course, in addition to software implementation, the electronic device described in this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0068] Furthermore, embodiments of this application also propose a computer program product, including a computer-readable storage medium storing one or more computer programs, the one or more computer programs including instructions.

[0069] When the aforementioned instructions are executed by a portable electronic device that includes multiple applications, they enable the portable electronic device to perform... Figure 1 The steps in the method shown include: The device entity recognition is performed on the multimodal description information of the reported fault device input by the user to obtain the device attribute information of the reported fault device.

[0070] The standard operating status information corresponding to the device attribute information is retrieved from the device knowledge base, and the multimodal description information is compared with the standard operating status information to obtain the abnormal description information corresponding to the faulty device. Retrieve diagnostic knowledge corresponding to the anomaly description information from the fault diagnosis knowledge base and / or fault diagnosis historical case base to generate a diagnostic process corresponding to the anomaly description information.

[0071] The diagnostic process is configured as a diagnostic chain consisting of at least two cascaded diagnostic task nodes, so that the diagnostic task nodes in the diagnostic chain are executed in the order of the path to obtain the diagnostic results of at least two diagnostic task nodes.

[0072] Based on all the diagnostic results obtained, a fault diagnosis report for the reported faulty device is generated and output.

[0073] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0074] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0075] The above are merely embodiments of this specification and are not intended to limit the scope of this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification. Furthermore, all other embodiments obtained by those skilled in the art without inventive effort should fall within the protection scope of this document.

Claims

1. A method for troubleshooting equipment, characterized in that, When applied to large models, the method includes: The device entity recognition is performed on the multimodal description information of the reported fault device input by the user to obtain the device attribute information of the reported fault device; Retrieve standard operating status information corresponding to the device attribute information from the device knowledge base, and compare the multimodal description information with the standard operating status information to obtain the abnormal description information corresponding to the faulty device; Retrieve diagnostic knowledge corresponding to the anomaly description information from the fault diagnosis knowledge base and / or fault diagnosis historical case base to generate a diagnostic process corresponding to the anomaly description information; The diagnostic process is configured as a diagnostic chain consisting of at least two cascaded diagnostic task nodes, so that the diagnostic task nodes in the diagnostic chain are executed in the order of the path to obtain the diagnostic results of at least two diagnostic task nodes. Based on all the diagnostic results obtained, a fault diagnosis report for the reported faulty device is generated and output.

2. The method according to claim 1, characterized in that, Also includes: When executing any target diagnostic task node in the diagnostic chain, determine the missing information on which the execution of the target diagnostic task node depends, and obtain the missing information from the user terminal or external system.

3. The method according to claim 1, characterized in that, The diagnostic task nodes that are not at the final level in the diagnostic chain are configured with branching conditions that guide at least one next-level diagnostic task node based on the current diagnostic result; the method further includes: When executing and determining the diagnostic results of the non-tail-level target diagnostic task nodes in the diagnostic chain, the next level diagnostic task node to be executed is determined based on all the currently obtained diagnostic results and the branch conditions set by the target diagnostic task node.

4. The method according to claim 1, characterized in that, The large model is obtained through joint training on the following tasks: The first task of performing device entity recognition includes training data for sample multimodal description information and supervision labels for indicating truth entity attribute information of the sample multimodal description information. The second task is to generate anomaly description information. The training data for the second task includes sample anomaly description information and supervision labels for indicating the ground truth anomaly description information of the sample anomaly description information. A third task is performed to generate a diagnostic chain. The training data for the third task includes a sample diagnostic process and supervision labels for a truth diagnostic chain that indicates the sample diagnostic process. The fourth task is to perform diagnostic task nodes, and the training data for the fourth task includes sample diagnostic task nodes and supervision labels used to indicate the ground truth diagnostic results of the sample diagnostic task nodes. The fifth task is to generate missing information-guided text. The training data for the fifth task includes sample missing information and supervision labels for indicating the truth-guided text of the sample missing information. The sixth task is to generate diagnostic reports. The training data for the sixth task includes sample diagnostic results and supervision labels for ground truth diagnostic reports that indicate the sample diagnostic results. The sample multimodal description information, the sample anomaly description information, the sample diagnosis process, the sample diagnosis task node, the sample missing information, and the sample diagnosis result all correspond to the same device.

5. The method according to claim 4, characterized in that, The large model is also obtained through reinforcement learning optimization training, wherein the comprehensive reward signal optimized by reinforcement learning is determined based on the matching degree between the output results of the first task, the second task, the third task, the fourth task, the fifth task and the sixth task and their respective corresponding supervision labels.

6. The method according to any one of claims 1 to 5, characterized in that, After generating and outputting a fault diagnosis report for the reported faulty device, the method further includes: Receive feedback information regarding the fault diagnosis report; If the feedback information confirms that the fault diagnosis report is correct or that the problem of the reported device has been resolved, then a new fault diagnosis history case is generated based on the device entity attribute information, the anomaly description information, the diagnosis process, and the fault diagnosis report. The new fault diagnosis history case is added to the fault diagnosis history case library.

7. The method according to any one of claims 1 to 5, characterized in that, The multimodal description information includes an image of the reporting device; before performing device entity recognition on the multimodal description information of the reporting device input by the user, the method further includes: The images from the fault-reporting device are subjected to compliance testing, which includes at least one of brightness testing, sharpness testing, and content compliance testing. If the image of the reporting device fails the compliance test, the user terminal will be prompted with the reason for the failure and guidance information on how to re-collect the image of the reporting device.

8. A device for troubleshooting equipment, characterized in that, Applied to large models, the device includes: The device attribute recognition module is used to perform device entity recognition on the multimodal description information of the reported fault device input by the user, and obtain the device attribute information of the reported fault device; The anomaly description determination module retrieves standard operating status information corresponding to the device attribute information from the device knowledge base, and compares the multimodal description information with the standard operating status information to obtain the anomaly description information corresponding to the faulty device. The diagnostic process determination module is used to retrieve diagnostic knowledge corresponding to the abnormal description information from the fault diagnosis knowledge base and / or the fault diagnosis historical case base, so as to generate the diagnostic process corresponding to the abnormal description information. The diagnostic task execution module is used to configure the diagnostic process as a diagnostic chain consisting of at least two cascaded diagnostic task nodes, so as to execute the diagnostic task nodes in the diagnostic chain in the path order and obtain the diagnostic results of at least two diagnostic task nodes. The diagnostic report generation module is used to generate and output a fault diagnosis report for the reported faulty device based on all the obtained diagnostic results.

9. An electronic device, comprising: processor; And a memory arranged to store computer-executable instructions, characterized in that, when executed, the executable instructions cause the processor to perform the method as described in any one of claims 1 to 7.

10. A computer program product, the computer program product comprising a computer-readable storage medium storing a computer program, characterized in that, The computer program is operable to cause the computer to perform the method as described in any one of claims 1 to 7.

Images