WSN fault diagnosis and maintenance method based on visual big language model

Abstract

The invention provides a WSN fault diagnosis and maintenance method based on a visual large language model. The method comprises the following steps: (1) multi-modal data acquisition; (2) protocol automatic analysis; (3) visual feature extraction; (4) carrying out multi-modal feature fusion; and (5) fault matching and reasoning. The method can be operated on a cloud, a gateway or a mobile device so as to meet real-time diagnosis requirements in different scenes.

Description

Technical Field

[0001] This invention relates to the fields of wireless communication network operation and maintenance and artificial intelligence multimodal fusion technology, and in particular to a method for WSN fault diagnosis and maintenance based on a visual large language model. Background Technology

[0002] Wireless sensor networks (WSNs) have been widely used in many important scenarios. Composed of numerous low-power sensor nodes, they are deployed long-term in complex environments and are prone to various operational problems such as node offline, link instability, data anomalies, and physical damage. To ensure network reliability and data integrity, existing technologies mainly rely on operational data actively reported by nodes and centralized analysis by network management platforms. Core solutions include periodic status reporting, heartbeat monitoring, link quality assessment, and mobile sink sniffing of broadcast data. Simultaneously, they parse heterogeneous protocols from different vendors using manually written templates or regular expressions.

[0003] However, existing technologies have significant limitations: First, they heavily rely on node reporting capabilities. When a node cannot send data due to insufficient power supply or firmware failure, it becomes a "silent node," and physical status information such as node damage, indicator light alarms, and abnormal posture cannot be obtained, resulting in significant diagnostic blind spots. Second, heterogeneous WSN protocol formats are scattered, and manually writing parsers for adaptation is inefficient and error-prone, making it difficult to meet the needs of large-scale expansion. Third, diagnostic methods are mostly based on single-modal data, lacking the ability to jointly reason from visual, message, environmental, and historical knowledge, and cannot form a complete closed loop of "fault identification → cause analysis → maintenance strategy generation." For example, "diagnostic methods that rely solely on message data" can complete "fault identification" (capturing abnormal messages), but lack "cause analysis" (unable to determine whether it is hardware damage or signal interference based on the node's physical status), and it is even more difficult to generate accurate "maintenance strategies" (not knowing whether to replace equipment or adjust the channel). Another example is "diagnostic methods that rely solely on environmental data": they can assist in "cause analysis" (e.g., detecting excessive temperature and humidity), but "fault identification" is inaccurate (unable to locate specific faulty nodes) and cannot provide targeted "maintenance strategies." Summary of the Invention

[0004] This invention provides a WSN fault diagnosis and maintenance method based on a visual large language model. The core process includes four key steps: First, through a fixed gateway, mobile sniffing nodes, and visual inspection devices, the wireless packets (heartbeat packets, alarm packets, status packets, etc.), appearance images, indicator light status, and environmental information of the WSN nodes are collected from multiple dimensions, and a time synchronization mechanism is used to ensure data alignment. Second, the visual large language model is used to automatically parse heterogeneous protocols, generate structured semantic features, and extract visual features related to the physical state of the nodes. Next, a multimodal feature set integrating packet features, visual features, topological features, and historical knowledge is constructed and matched with a preset structured fault rule base. Finally, through a hybrid mechanism of rule matching and large model semantic reasoning, the fault type is accurately identified, the root cause is analyzed, and actionable maintenance suggestions are output, forming a fully automated closed loop from data collection and feature parsing to fault diagnosis and maintenance decision-making.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: The WSN fault diagnosis and maintenance method based on visual large language model includes the following steps: (1) Multimodal data acquisition: Through a fixed gateway, a mobile sniffing node and a visual inspection device equipped with a camera unit, wireless packets, node images, channel quality indicators and topology association information of each node in the wireless sensor network are collected. At the same time, a time synchronization mechanism is established between the fixed gateway, the mobile sniffing node and the visual inspection device to map the collected multi-source data to a unified time axis and package it. (2) Automatic protocol parsing: The collected raw wireless packets are input into the Visual Large Language Model (VLM), and combined with multimodal context information, the VLM automatically infers the packet structure, field meaning and fault semantics, and generates standardized protocol fields and corresponding semantic annotation information; (3) Visual feature extraction: The collected node images are input into the VLM, and the model is guided to focus on the key areas of the nodes through multiple rounds of interactive text prompts. Visual features related to indicator light status, appearance integrity, posture and power supply component status are extracted and converted into structured visual feature vectors. (4) Multimodal feature fusion: Construct a multimodal fault feature vector that includes the standardized protocol fields, structured visual feature vector, channel quality indicators, topology association information and historical knowledge base features; (5) Fault matching and reasoning: The multimodal fault feature vector is matched with the pre-built fault feature library. Combined with the reasoning capability of the VLM, the fault type, root cause and corresponding maintenance suggestions of the node are output to form an automated closed loop of fault diagnosis and maintenance.

[0006] In this specification, the wireless messages in step (1) include heartbeat packets, alarm packets, status packets and other wireless broadcast messages sent by the node, and the node images include indicator light status images, node appearance images, power supply component images and images of the surrounding environment of the node.

[0007] In this specification, the mobile sniffing node described in step (1) has passive listening capability, and can enter weak signal areas and obstructed areas where the fixed gateway coverage is insufficient, capture residual broadcasts, retransmission frames or handshake requests sent by nodes that cannot report data normally, and supplement the acquisition of the node's communication behavior characteristics.

[0008] In this manual, the visual information collected by the visual inspection device in step (1) includes: the color and flashing mode of the node indicator light, whether the shell is damaged or water has entered, whether there is any obstruction or tilting, the physical status of power supply components such as batteries or solar panels, and the lighting, water accumulation and obstruction conditions of the on-site environment.

[0009] In this specification, the multimodal context information mentioned in step (2) includes image slices of the protocol document, natural language descriptions of known related protocols, and system prompts for guiding model parsing. The semantic annotation information output by the VLM includes the original field value, parsed value, unit information, and fault association relationship.

[0010] In this specification, the multi-round interactive text prompts in step (3) are constructed according to hierarchical topics and are used to inquire about the overall scene of the node, appearance integrity, indicator status and power supply component status in turn. The VLM generates a corresponding natural language description based on each prompt and then converts it into a structured visual feature vector through text encoding or rule post-processing.

[0011] In this specification, the fault feature library mentioned in step (5) is composed of structured rules. Each rule includes a feature condition set, a fault category, and a maintenance measure set. The feature condition set is a protocol parsing field, a visual extraction attribute, a network layer indicator, or a logical combination thereof. The maintenance measure set is a specific operation suggestion for the corresponding fault category.

[0012] These structured rules mainly come from three types of criteria: Industry WSN Fault Knowledge Base: Integrates standardized correspondences of characteristic conditions, fault categories, and maintenance measures for common faults (such as battery aging and signal interference) that have been verified in the field of wireless sensor networks. VLM analyzes historical multimodal data: By performing correlation analysis on historically collected message, visual, and channel multimodal data, new fault feature combination rules are extracted (such as "voltage < 3V + battery appearance swelling" corresponding to "battery aging"). Iterative optimization in actual operation and maintenance scenarios: In the actual implementation of fault diagnosis, verify the effectiveness of existing rules and supplement the rules corresponding to newly emerging fault feature combinations.

[0013] In this specification, the reasoning process described in step (5) adopts a hybrid mechanism: first, the multimodal fault feature vector is precisely matched or the confidence threshold is matched with the rule conditions in the fault feature library to activate candidate faults and maintenance measures; when there is a rule conflict or no fully matching rule, the VLM is called to perform semantic reasoning to complete conflict resolution, fuzzy inference, fault root cause analysis and maintenance measure optimization.

[0014] In this specification, the time synchronization mechanism described in step (1) is implemented through local clock calibration, network time protocol or timestamp embedding to ensure that wireless messages, node images and other collected data are aligned in the time dimension, thereby ensuring the accuracy of multimodal feature fusion.

[0015] In this specification, step (4) introduces a multimodal feature attention fusion algorithm, and step (5) introduces a hybrid inference collaborative optimization algorithm, forming a new process as follows: (1) Multimodal data acquisition: Through fixed gateways, mobile sniffing nodes and visual inspection devices equipped with camera units, wireless messages, node images, channel quality indicators and topology association information of each node in the wireless sensor network are collected. The wireless messages include heartbeat packets, alarm packets, status packets and other wireless broadcast messages sent by the nodes. The node images include indicator status images, node appearance images, power supply component images and images of the surrounding environment of the nodes. A time synchronization mechanism is established through local clock calibration, network time protocol or timestamp embedding to map the collected multi-source data to a unified time axis and package it. (2) Automatic protocol parsing: The collected raw wireless packets are input into the Visual Large Language Model (VLM) and combined with multimodal context information, which includes image slices of the protocol document, natural language descriptions of known related protocols, and system prompts to guide the model parsing. The VLM automatically infers the packet structure, field meanings, and fault semantics, and generates standardized protocol fields and corresponding semantic annotation information. The semantic annotation information includes the original value, parsed value, unit information, and fault association relationship of the field. The standardized protocol fields and semantic annotation information serve as one of the core inputs of multimodal fusion. (3) Visual feature extraction: The collected node images are input into VLM. The model is guided to focus on the key areas of the node by multi-round interactive text prompts constructed according to the hierarchical theme. The multi-round interactive text prompts are used to inquire about the overall scene of the node, appearance integrity, indicator status and power supply component status in turn. VLM generates a corresponding natural language description based on each prompt, and then converts it into a structured visual feature vector through text encoding or rule post-processing. (4) Multimodal feature fusion: Receive the standardized protocol fields from step (2), the structured visual feature vectors from step (3), the collected channel quality indicators, topology association information, and the historical knowledge base features from pre-stored WSN fault cases, maintenance records, and equipment operation rule bases. Calculate the real-time reliability score and inference feedback weight of each modality using the multimodal feature attention fusion algorithm. The real-time reliability score is determined by the integrity and consistency of each modality's data. The inference feedback weight is generated by the feature adjustment signal output by the hybrid inference collaborative optimization algorithm. Dynamically allocate the fusion weight of each modality based on the real-time reliability score and inference feedback weight. Generate a multimodal fault feature vector by weighted summation to improve the correlation between features and faults. (5) Fault matching and reasoning: The multimodal fault feature vector from step (4) is matched with the pre-built fault feature library, which consists of structured rules containing feature condition sets, fault categories and maintenance measure sets. The weights of rule matching and VLM semantic reasoning are dynamically adjusted by a hybrid reasoning collaborative optimization algorithm. The rule matching weight is determined by the sum of the fusion contributions of protocol features, channel features and topology features, and the VLM semantic reasoning weight is determined by the sum of the fusion contributions of visual features and historical knowledge base features. Candidate faults and maintenance measures are activated first by precise matching or confidence threshold matching. When there is a rule conflict or no fully matching rule, VLM is called to perform semantic reasoning to complete conflict resolution, fuzzy inference, fault root cause analysis and maintenance measure optimization. At the same time, feature adjustment feedback signals are output and transmitted back to VLM and multimodal feature attention fusion algorithm to optimize the subsequent feature extraction weights and fusion weights respectively. Finally, the fault type, root cause and corresponding maintenance suggestions of the node are output to form an automated closed loop of fault diagnosis and maintenance.

[0016] There are three ways to set the confidence threshold: Based on fault type, the basic threshold is set as follows: For high-impact faults such as hardware damage and communication interruption, the basic confidence threshold is set to 0.6 (to reduce the risk of missed detection); for low-impact faults such as signal fluctuations and temporary message delays, the basic threshold is set to 0.8 (to reduce false positives). Real-time reliability adjustment based on multimodal features: Referencing the real-time reliability score of "data integrity × 0.6 + data consistency × 0.4" output by the multimodal feature attention fusion algorithm, if the data integrity of a certain modality is < 0.7, the corresponding threshold is increased by 0.1 (to improve the matching rigor). Iterative optimization based on historical diagnostic results: For every 1000 valid diagnostic data points accumulated, the correspondence between "threshold and accuracy" for this type of fault is automatically calculated, and the threshold with a false positive rate > 5% is finely adjusted by ±0.05.

[0017] In summary, the present invention has at least the following beneficial effects: Improved diagnostic comprehensiveness and accuracy: By combining mobile sniffing and visual recognition, it effectively identifies silent nodes and physical faults that cannot be detected by traditional methods, filling diagnostic blind spots and significantly improving the comprehensiveness and accuracy of fault diagnosis.

[0018] Protocol adaptation efficiency optimization: Based on a visual large language model, automatic parsing and semantic understanding of heterogeneous WSN protocols are achieved, eliminating the need for manual parser writing, significantly reducing the manpower and time costs of protocol adaptation, and improving the adaptation flexibility for multi-vendor and multi-protocol scenarios.

[0019] Improved automation level of operation and maintenance: Construct a multimodal fusion intelligent reasoning mechanism to automatically associate messages, vision, topology and historical knowledge, generate root causes of failures and targeted maintenance suggestions, realize an automated closed loop from diagnosis to maintenance, reduce manual intervention and reduce operation and maintenance complexity.

[0020] Network reliability assurance: Through precise fault location and efficient maintenance guidance, the duration of network failures is reduced, and the long-term stability of WSN operation and the integrity of data transmission are improved. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the structure of the solution involved in this invention.

[0022] Figure 2 This is a schematic diagram of the workflow involved in this invention.

[0023] Figure 3 This is a schematic diagram illustrating the working principle involved in this invention. Detailed Implementation

[0024] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0025] like Figure 1 , Figure 2 and Figure 3 As shown, this embodiment provides a WSN fault diagnosis and maintenance method based on a visual large language model, including the following steps: (1) Multimodal data acquisition: Through a fixed gateway, a mobile sniffing node and a visual inspection device equipped with a camera unit, wireless packets, node images, channel quality indicators and topology association information of each node in the wireless sensor network are collected. At the same time, a time synchronization mechanism is established between the fixed gateway, the mobile sniffing node and the visual inspection device to map the collected multi-source data to a unified time axis and package it. (2) Automatic protocol parsing: The collected raw wireless packets are input into the Visual Large Language Model (VLM), and combined with multimodal context information, the VLM automatically infers the packet structure, field meaning and fault semantics, and generates standardized protocol fields and corresponding semantic annotation information; (3) Visual feature extraction: The collected node images are input into the VLM, and the model is guided to focus on the key areas of the nodes through multiple rounds of interactive text prompts. Visual features related to indicator light status, appearance integrity, posture and power supply component status are extracted and converted into structured visual feature vectors. (4) Multimodal feature fusion: Construct a multimodal fault feature vector that includes the standardized protocol fields, structured visual feature vector, channel quality indicators, topology association information and historical knowledge base features; (5) Fault matching and reasoning: The multimodal fault feature vector is matched with the pre-built fault feature library. Combined with the reasoning capability of the VLM, the fault type, root cause and corresponding maintenance suggestions of the node are output to form an automated closed loop of fault diagnosis and maintenance.

[0026] In some embodiments, step (4) introduces a multimodal feature attention fusion algorithm, and step (5) introduces a hybrid inference collaborative optimization algorithm, forming a new process as follows: (1) Multimodal data acquisition: Through fixed gateways, mobile sniffing nodes and visual inspection devices equipped with camera units, wireless messages, node images, channel quality indicators and topology association information of each node in the wireless sensor network are collected. The wireless messages include heartbeat packets, alarm packets, status packets and other wireless broadcast messages sent by the nodes. The node images include indicator status images, node appearance images, power supply component images and images of the surrounding environment of the nodes. A time synchronization mechanism is established through local clock calibration, network time protocol or timestamp embedding to map the collected multi-source data to a unified time axis and package it. (2) Automatic protocol parsing: The collected raw wireless packets are input into the Visual Large Language Model (VLM) and combined with multimodal context information, which includes image slices of the protocol document, natural language descriptions of known related protocols, and system prompts to guide the model parsing. The VLM automatically infers the packet structure, field meanings, and fault semantics, and generates standardized protocol fields and corresponding semantic annotation information. The semantic annotation information includes the original value, parsed value, unit information, and fault association relationship of the field. The standardized protocol fields and semantic annotation information serve as one of the core inputs of multimodal fusion. (3) Visual feature extraction: The collected node images are input into VLM. The model is guided to focus on the key areas of the node by multi-round interactive text prompts constructed according to the hierarchical theme. The multi-round interactive text prompts are used to inquire about the overall scene of the node, appearance integrity, indicator status and power supply component status in turn. VLM generates a corresponding natural language description based on each prompt, and then converts it into a structured visual feature vector through text encoding or rule post-processing. (4) Multimodal feature fusion: Receive the standardized protocol fields from step (2), the structured visual feature vectors from step (3), the collected channel quality indicators, topology association information, and the historical knowledge base features from pre-stored WSN fault cases, maintenance records, and equipment operation rule bases. Calculate the real-time reliability score and inference feedback weight of each modality using the multimodal feature attention fusion algorithm. The real-time reliability score is determined by the integrity and consistency of each modality's data. The inference feedback weight is generated by the feature adjustment signal output by the hybrid inference collaborative optimization algorithm. Dynamically allocate the fusion weight of each modality based on the real-time reliability score and inference feedback weight. Generate a multimodal fault feature vector by weighted summation to improve the correlation between features and faults. (5) Fault matching and reasoning: The multimodal fault feature vector from step (4) is matched with the pre-built fault feature library, which consists of structured rules containing feature condition sets, fault categories and maintenance measure sets. The weights of rule matching and VLM semantic reasoning are dynamically adjusted by a hybrid reasoning collaborative optimization algorithm. The rule matching weight is determined by the sum of the fusion contributions of protocol features, channel features and topology features, and the VLM semantic reasoning weight is determined by the sum of the fusion contributions of visual features and historical knowledge base features. Candidate faults and maintenance measures are activated first by precise matching or confidence threshold matching. When there is a rule conflict or no fully matching rule, VLM is called to perform semantic reasoning to complete conflict resolution, fuzzy inference, fault root cause analysis and maintenance measure optimization. At the same time, feature adjustment feedback signals are output and transmitted back to VLM and multimodal feature attention fusion algorithm to optimize the subsequent feature extraction weights and fusion weights respectively. Finally, the fault type, root cause and corresponding maintenance suggestions of the node are output to form an automated closed loop of fault diagnosis and maintenance.

[0027] The technical concept of this invention is as follows: This invention aims to overcome the problems of unobservable silent nodes, difficulty in parsing heterogeneous protocols, and limited diagnostic information in existing wireless sensor network (WSN) fault diagnosis, and proposes a multimodal fault diagnosis and maintenance method and system based on a visual large language model. Specific objectives include: 1) Enhance the ability to diagnose the physical status of nodes and silent nodes: Identify physical information such as node indicator lights, appearance damage, and abnormal posture through a visual large language model, making up for the shortcomings of traditional methods that rely on node reporting and cannot diagnose silent nodes.

[0028] 2) Achieve automatic parsing and semantic understanding of heterogeneous WSN protocols: Utilize a large language model to automatically infer message structure, field meaning, and fault semantics, solving the problems of existing protocols requiring manual parsing and having low adaptation efficiency.

[0029] 3) Construct an intelligent fault diagnosis and maintenance mechanism that integrates vision, messages, and knowledge: Integrate wireless messages and visual information collected by mobile sniffing, and generate fault types and maintenance suggestions through large model reasoning to achieve an automated closed loop from diagnosis to decision-making.

[0030] The overall technical solution of this invention consists of key modules such as multimodal data acquisition, automatic protocol parsing, visual feature extraction, fault feature fusion, fault identification, and maintenance strategy generation. The solution architecture is as follows: Figure 1 As shown, the workflow is as follows Figure 2 As shown, the working principle is as follows: Figure 3 As shown, the overall process is as follows: First, operational information of each node in the wireless sensor network is acquired through fixed gateways and mobile sniffing nodes, including over-the-air broadcast packets, service messages, indicator light status images, and node appearance images. Mobile sniffing nodes are used to enhance data availability, especially when nodes are unable to report messages normally; they can passively listen to obtain characteristics of the node's communication behavior.

[0031] Secondly, the collected raw wireless data packets are input into the Visual Large Language Model (VLM). Through the model's structural inference and semantic understanding capabilities, the message format is automatically parsed to generate standardized protocol fields and fault-related semantic information. At the same time, the Visual Large Language Model (VLM) processes the node images to identify visual features such as indicator light flashing patterns, appearance damage, occlusion, tilting, and power abnormalities, thereby achieving automatic extraction of the physical state.

[0032] Next, the system constructs a multimodal fault feature vector that includes communication features, visual features, protocol field features, and historical knowledge base features, and matches it with a pre-built fault feature library. The fault feature library is generated from typical fault scenarios, operating rules, and maintenance experience, and has a scalable index structure to support fast retrieval and comparison.

[0033] Finally, based on the inference capabilities of VLM, the system outputs the fault type, root cause, and corresponding maintenance recommendations according to the matching results, forming an automated closed loop from fault identification and cause explanation to maintenance measure recommendation. This technical solution is applicable to both method implementation and system deployment, and can run on the cloud, gateway, or mobile devices to meet real-time diagnostic needs in different scenarios.

[0034] Multimodal data acquisition subsystem: The mobile sniffing node possesses flexible mobility and passive monitoring capabilities, allowing it to enter areas with insufficient coverage from fixed base stations. Through its radio frequency front-end, it captures heartbeat packets, status packets, alarm packets, and other radio broadcast messages sent by the node in real time, as shown in Table 1-3. Because it can change its acquisition location spatially, it can supplement communication information in weak signal areas, obstructed areas, or areas where nodes cannot report normally. Especially when a node is offline, has insufficient power, or is unable to complete normal reporting due to firmware abnormalities, the sniffing node may still be able to hear residual broadcasts, retransmission frames, or handshake requests attempted by the node, thus significantly improving the observability of "silent nodes."

[0035] A mobile inspection device equipped with a camera unit is used to collect images of the node's appearance, indicator light flashing sequences, power supply component status, and surrounding environmental information. Visual data can reflect physical conditions that communication data cannot cover, such as hardware alarms corresponding to LED flashing patterns, power supply problems caused by battery swelling or solar panel obstruction, decreased sensing capabilities due to node tilting or being covered by foreign objects, and equipment damage caused by broken casings or water ingress. In addition, the visual acquisition device can also record external factors such as ambient light, water accumulation, and obstructions, providing richer contextual information for subsequent diagnosis.

[0036] To ensure that multimodal data is aligned in the time dimension, this invention establishes a unified time synchronization mechanism among mobile sniffing nodes, fixed gateways, and visual acquisition devices. This mechanism maps data from different sources onto a unified timeline through methods such as local clock calibration, network time protocols, or timestamp embedding. After data acquisition, the multimodal data is packaged according to a predefined structure, including wireless data packets, visual images, channel quality indicators, topology association information, and environmental context. This data is then uploaded to a server or edge computing node via wireless or wired links for further processing.

[0037] The typical data collected includes various types of messages such as heartbeat packets (0x02), alarm packets (0x22), and status packets (0x04), with the following structure: 1) The format of the alarm data packet (0x22) field is shown in Table 1: Table 1. Alarm Data Packet (0x22) Field Format 2) The format of the heartbeat data packet (0x02) field is shown in Table 2: Table 2. Heartbeat Data Packet (0x02) Field Format 3) The format of the status data packet (0x04) field is shown in Table 3. Table 3. Field Format of Status Data Packet (0x04) ; The protocol automatic parsing module based on a large visual language model: The core of this module is to introduce a large visual language model as an intelligent parsing engine, transforming the raw, unstructured wireless message data stream into a structured description rich in semantic information. Its basic idea is to model protocol parsing as a probabilistic mapping and reasoning problem from low-dimensional byte sequences to high-level semantic structures.

[0038] Suppose the captured raw message is a byte sequence of length L. ,in ∈[0,255]. This module uses a general analytical function parameterized by a large visual language model. The function's input includes not only the raw message P, but also multimodal context information C, which may include image slices of the protocol document. Natural language description of known relevant protocols and guiding system prompts Therefore, the parsing process can be formally represented as: Here, Θ represents the model parameters of the VLM, and S* is the structured parsing result with rich semantic annotations output by the model inference. This process essentially leverages the pattern recognition and commonsense reasoning capabilities gained by the VLM during training on large-scale cross-modal data. Through joint analysis of the input sequence and context, the model automatically identifies fixed frame headers, length fields, delimiters, and data segments with specific statistical distributions (such as sensor reading ranges and increasing sequence numbers) in the message, thereby inferring the message's hierarchical structure.

[0039] The parsing process consists of three stages. First, in the data representation and prompt construction stage, the system converts the binary message P into a text representation such as a hexadecimal string, and embeds it along with the multimodal context C into a structured prompt template, forming an input sequence X that the model can understand. Next, in the VLM inference and sequence generation stage, the model calculates the conditional probability P(Y|X;Θ) of generating the optimal parsed description sequence Y based on its internal representation. This process can be decomposed into autoregressive sequence generation: ,in, t is the t-th word in the output sequence, and M is the length of the output sequence. The output Y of the model is a natural language description, such as: "This message is an alarm type, command word 0x22; the timestamp field (BCD encoding) is parsed as UTC time 2023-10-26 14:30:05; the following byte 0x01 represents the radio frequency fault code, with a value of 1, corresponding to the semantic state of 'weak signal strength'." Finally, in the structured output and knowledge solidification stage, the system uses a post-processing program to automatically convert the natural language description Y generated by the model into a unified, machine-readable JSON or XML format S*. This structured result includes not only the original and parsed values ​​of the fields, but also semantic annotations provided by VLM, unit information, and possible fault associations.

[0040] Visual Large Language Model Feature Extraction Module: Given an image or video frame captured by a mobile inspection device that contains target sensor nodes. And possibly multiple rounds of interactive text prompts (For example: "Describe the status of the device indicator lights in the image", "Check if the device casing is damaged or abnormal"). The task of this module is to combine image and text information to generate a comprehensive and structured visual feature description. .

[0041] This process can be formally represented as the conditional probability generation of VLM given an input: Where Φ is a parameter of the VLM. It constitutes a feature description The k-th semantic unit (word or phrase) is given, where K is the length of the description. The model uses its multimodal attention mechanism to focus on key areas in the image related to the fault (such as LED lights, antenna interfaces, battery compartments, and mounting brackets), and uses its prior knowledge (such as "a solid red light may indicate a serious alarm" and "a crack in the casing may cause water ingress") to reason and finally output a natural language description.

[0042] The workflow of this module includes the following key steps: 1. Visual Data Preprocessing and Prompt Engineering: The system receives raw images or video streams and performs necessary preprocessing (such as scaling and cropping to highlight key nodes). Simultaneously, based on the diagnostic task requirements, it automatically constructs a series of structured, multi-round visual question-and-answer prompts. These prompts are designed to extract visual information in a layered and thematic manner. For example, the first round of prompts might request an overall scene description, the second round focuses on the integrity of the device's appearance, and the third round specifically inquires about the color and flashing pattern of indicator lights.

[0043] 2. VLM Multi-Turn Interactive Feature Extraction: Preprocessed images and a series of prompts are input into VLM. The model responds to each prompt sequentially in a conversational manner, generating detailed textual responses. For example, for the prompt "Please describe the condition of the device casing in detail," VLM might output: "The main casing of the device has no obvious cracks or damage, but there is a mud stain approximately 5 cm long attached to the bottom; there are slight traces of rust at the antenna interface; the device mounting bracket is tilted to the left at approximately 15 degrees." This interactive approach allows the system to proactively and specifically "question" the image to obtain all the key visual details needed for diagnosis.

[0044] 3. Visual Feature Structuring and Vectorization: The multi-turn natural language description generated by VLM is transformed into a structured visual feature vector v through a lightweight text encoder or a keyword and rule-based post-processing pipeline. This vector contains multiple dimensions, each corresponding to a specific visual attribute and its... Status rating. For example: v=[ Red flash mob 0.2, 15° 0.7, :0.8,...]^T, where continuous values ​​may represent confidence scores or quantified estimates of the degree of anomaly. This structured vector v is the output of the module, and its format is compatible with the semantic features output by the protocol parsing module, facilitating subsequent cross-modal fusion.

[0045] The structured visual feature vector *v* output by this module, together with the semantic protocol feature vector *s* output by the automatic protocol parsing module, constitutes a multimodal representation of the node state. In the subsequent fault diagnosis and reasoning module, the system fuses *v* and *s* at the feature level or decision level. For example, when the protocol feature indicates a "low battery voltage" fault code, while the visual feature simultaneously indicates "solar panels are covered with a large amount of dust," the system can more accurately infer that the root cause is "insufficient power supply due to obstructed energy acquisition," rather than simply battery aging.

[0046] Rule base and reasoning mechanism: (1) Construction of a structured rule base: The rule base constructed in this invention is essentially a formalized and structured product of domain expert knowledge, historical maintenance experience, and device protocol specifications. A wireless sensor network can be formally represented as WSN=(D,Q,BS), where For a set of sensor nodes, Let be a set of relay nodes, and BS be a base station. To achieve systematic diagnosis, we constructed a structured knowledge base consisting of a series of diagnostic rules, denoted as . .

[0047] Each diagnostic rule Define a triple as follows: .in: This represents the feature condition set, a set of feature attribute assertions that must be satisfied to trigger the rule. These features can be derived from protocol parsing fields (such as...). <3.0V), visual extraction attributes (such as solar_panel_dirt_coverage>0.6), network layer metrics (such as packet_loss_rate>20%), or logical combinations thereof.

[0048] Indicates the fault category, i.e., when the condition is... When the conditions are met, the specific fault type determined by the system (such as "insufficient power supply" or "weak radio frequency link").

[0049] This represents a set of maintenance measures, which are designed to address faults. The recommended series of specific maintenance operations (such as "cleaning the solar panel" and "checking and adjusting the antenna angle").

[0050] The specific rule base is shown in Table 4: Table 4. Examples of WSN Fault Diagnosis Rule Base ; The rule base is organized hierarchically and modularly, covering typical failure modes across dimensions such as hardware (e.g., sensors, batteries, memory), communication (RF links, network connections), software (program execution, protocol consistency), and environment (physical obstruction, contamination). For example, a composite rule for "solar power supply system failure" has the following conditions: It may simultaneously include the "solar cell voltage consistently below the threshold" from the protocol and the "solar panel is severely covered by dust or physically obstructed" from the visual characteristics, thus accurately distinguishing whether the charging abnormality is caused by a battery malfunction or environmental factors.

[0051] (2) Multimodal Feature-Driven Hybrid Inference Mechanism: The inference mechanism of this module receives the structured semantic feature vector s from the protocol parsing module and the visual feature vector v from the visual feature extraction module. After fusion, it performs matching and inference with the rule base R. Its core function can be expressed as: ; The reasoning process specifically includes the following steps: Feature fusion and unified representation: Align and fuse s and v to form a unified multimodal feature set that describes the all-round state of the current node. The fusion process needs to address the differences in confidence level, dimensions, and time scale among different modal features.

[0052] Deterministic rule matching: With each rule in the rule base conditions Perform fast matching. The matcher supports exact matching, range matching, and confidence threshold matching. When A rule is fully or highly satisfied. When this rule is activated, its corresponding This will be used as a candidate output.

[0053] Semantic reasoning and arbitration based on a large model: In complex scenarios, multiple rules may be partially activated, or no rule may be able to completely match. In such cases, the system will invoke the embedded large language model for deep semantic reasoning. Taking the natural language description, all partial matching rules, and their contextual information as input, perform the following tasks: Conflict resolution: When multiple rules conflict, analyze the primary and secondary characteristics and causal relationships, and select the most likely failure.

[0054] Fuzzy inference: When features do not fully conform to any predefined rules, analogical reasoning is performed based on general fault knowledge to generate reasonable fault hypotheses.

[0055] Root cause analysis: Combining multimodal characteristics to infer the root cause of the failure (e.g., determining whether the low battery voltage is due to aging or solar panel failure causing the inability to charge).

[0056] Optimization of measures: Adjust standard maintenance measures according to specific environmental characteristics (such as site weather and equipment accessibility). Refine or sort the instructions to generate more actionable guidance.

[0057] Diagnostic result generation and output: The reasoning mechanism ultimately outputs the determined fault category. and a detailed list of maintenance recommendations The output includes a confidence score and key feature descriptions of the reasoning basis to ensure the interpretability of the diagnostic results.

[0058] The innovation of this invention is mainly reflected in the following four core technologies, which are the key areas of protection of this invention: (1) Automatic parsing method of wireless protocol based on visual large language model: This invention uses visual large language model to automatically infer the structure and interpret the semantics of the original binary message, so as to adapt to WSN heterogeneous protocols of different manufacturers and formats without the need for manual parsing, and significantly improve the automation and universality of protocol parsing.

[0059] (2) Node physical status diagnosis mechanism based on visual semantic understanding: By analyzing node images through a visual big language model, physical features such as indicator light status, appearance damage, obstruction, tilting, and abnormal power supply components can be automatically identified, which makes up for the shortcomings of traditional methods that rely solely on communication data to identify "silent nodes" and physical faults.

[0060] (3) Multimodal fusion fault feature representation and reasoning mechanism: Construct a multimodal feature vector composed of message features, visual features, topological features and historical knowledge, and combine rule reasoning and large model reasoning to form a hybrid diagnostic framework to achieve accurate identification of complex, multi-factor coupled faults and generation of maintenance strategies.

[0061] (4) Enhanced observability system based on mobile sniffing: By using mobile sniffing nodes to supplement the capture of residual broadcasts and weak signal behaviors of weak coverage areas and abnormal nodes, the system can effectively detect silent nodes and use them in conjunction with visual inspection results to improve the overall diagnostic coverage and accuracy.

[0062] This invention significantly improves the fault diagnosis capabilities of wireless sensor networks by introducing a large visual language model and a multimodal fusion mechanism. Firstly, the combination of mobile sniffing and visual recognition enables the system to identify silent nodes and physical faults that traditional methods cannot detect, improving diagnostic coverage and accuracy. Secondly, the large-model-based automatic protocol parsing technology eliminates the need for manually written parsers, significantly reducing protocol adaptation costs and improving the efficiency of handling heterogeneous protocols from multiple vendors. Thirdly, the multimodal feature fusion and intelligent reasoning methods can automatically generate fault causes and maintenance suggestions, achieving an automated closed loop from diagnosis to maintenance, significantly reducing operation and maintenance costs and improving the long-term reliability of the network.

[0063] In some embodiments, the Visual Large Language Model (VLM) adopts the BLIP-2 base model, with a ViT-L / 14 visual encoder and a Flan-T5-XL text decoder. It supports running via API calls or local deployment, with the input format being "image data + text prompts" and the output format being structured natural language descriptions or JSON data.

[0064] In some embodiments, the system prompt template for guiding model parsing in step (2) is: "It is known that the wireless sensor network message is in binary format and may contain fields such as frame header, timestamp, and fault code. Please parse the following hexadecimal message structure, meaning of each field, unit and fault association: [hexadecimal message content], refer to the protocol document image slice information: [image description], known related protocol description: [natural language description]."

[0065] In some embodiments, the specific content of the multi-round interactive text prompts in step (3) is as follows: the first round "Please describe the overall deployment scenario of the sensor nodes and the relationship between the device locations"; the second round "Please check whether the node shell is damaged, water-infiltrated, mud-stained or rusted, whether the bracket is tilted, and the approximate tilt angle"; the third round "Please describe the color, flashing mode (constant on / fast flashing / slow flashing) and number of the node indicator lights"; the fourth round "Please observe the physical state of the power supply components such as batteries and solar panels, and whether there is swelling, obstruction, damage or abnormal voltage interface".

[0066] In some embodiments, the dimensions of the structured visual feature vector in step (3) include LED status, degree of shell damage, bracket tilt angle, degree of mud coverage, corrosion level, and power supply component status. The degree of damage, degree of coverage, and corrosion level are represented by confidence scores between 0 and 1, the tilt angle is represented by degrees, and the LED status and power supply component status are represented by classification labels, for example, the vector format is [LED_Status:Red flashing,Enclosure_Damage:0.2,Tilt_Angle:15°,Dirt_Coverage:0.7,Corrosion_Level:0.8,Power_Component_Status:Normal].

[0067] In some embodiments, the multimodal feature fusion in step (4) adopts a weighted fusion method, wherein the weight of the standardized protocol field is 0.4, the weight of the structured visual feature vector is 0.3, the weight of the channel quality index is 0.2, the weight of the topology association information is 0.05, and the weight of the historical knowledge base feature is 0.05. Before fusion, the modal features are unified to the [0,1] interval by the Min-Max normalization method to ensure that the dimensions are consistent.

[0068] In some embodiments, the structured rules of the fault feature library in step (5) also include composite rules. For example, the feature condition set of a composite rule is "the solar cell voltage <3.2V lasts for 60 seconds in the protocol parsing field and the confidence level of solar panel shading in the visual extraction attribute is >0.6", the corresponding fault category is "solar panel shading leads to insufficient power supply", and the maintenance measures set is "clean the surface of the solar panel of the obstruction and adjust the solar panel orientation to the direction of sufficient light".

[0069] In some embodiments, the confidence threshold of the hybrid inference mechanism in step (5) is set to 0.8. When the matching degree between the multimodal fault feature vector and a certain rule condition set is ≥0.8, it is determined to be a complete match; when the matching degree is between 0.5 and 0.8, it is determined to be a partial match; when the matching degree is <0.5, it is determined to be no matching rule.

[0070] In some embodiments, the specific process of resolving conflicts in VLM semantic reasoning in step (5) is as follows: input the multimodal fault feature vector, all partially matched rule conditions and candidate fault categories into VLM, and obtain the reasoning result by prompting "The following are the multimodal features of the sensor node: [feature description], the currently matched candidate fault and rule conditions are: [rules and fault description], please resolve the conflict based on feature correlation and fault occurrence probability priority, and determine the optimal fault type, root cause and appropriate maintenance measures".

[0071] In some embodiments, the listening frequency of the mobile sniffing node in step (1) is 10Hz, the signal capture threshold is RSSI≥-100dBm, it supports capturing wireless packets in the 2.4GHz / 5GHz band, and its movement path is planned according to the coverage blind zone map of the fixed gateway, prioritizing traversing weak signal areas and historically high fault areas.

[0072] In some embodiments, the time synchronization mechanism in step (1) is implemented using the Network Time Protocol (NTP), with a synchronization period of 30 seconds and a synchronization error controlled within 50ms. By embedding a timestamp field in the data packets of each acquisition device, the alignment accuracy of wireless messages, node images, and channel quality indicators in the time dimension is ensured to meet the requirements of multimodal fusion.

[0073] In some embodiments, the hardware parameters of the visual inspection device in step (1) are: camera resolution ≥ 4K, shooting distance controlled between 0.5-1 meter, shooting angle with the vertical direction of the node ≤ 30°, supporting automatic focusing to the key area of ​​the node (indicator light, power supply components, shell interface), and the image acquisition frequency is consistent with the message capture frequency of the mobile sniffing node, both being 1 frame per second.

[0074] In some embodiments, the criteria for determining a silent node are: "no data is reported to the fixed gateway for three consecutive heartbeat cycles (each heartbeat cycle is 60 seconds), and the mobile sniffing node listens for 3 minutes in the deployment area of ​​the node and does not capture any residual broadcasts, retransmission frames or handshake requests." At this time, the visual inspection device is triggered to focus on collecting data from the node, and the number of collections is no less than 3 times to ensure that a clear physical state image is obtained.

[0075] In some embodiments, taking battery fault diagnosis of a smart city street light wireless sensor network as an example, the specific implementation process is as follows: ① Multimodal data acquisition: The fixed gateway has not received a heartbeat packet (command word 0x02) from a certain street light node for more than 3 cycles. The mobile sniffing node listens and captures 1 frame of residual broadcast message (hexadecimal: 0x02 16 01 0203 04 05 03 -10 05...) in the node area. The visual inspection device takes 3 node images and achieves time synchronization through the NTP protocol with a synchronization error of 30ms; ② Automatic protocol parsing: The residual broadcast message is input into the BLIP-2 model. Combined with the protocol document image slices and the known heartbeat packet protocol description, the battery voltage field value is parsed to be 28 (corresponding to 2.8V), the battery remaining power field value is 20 (corresponding to 20%), and the semantic annotation is "Battery voltage is lower than the normal threshold of 3.0V, and the remaining power is insufficient"; ③ Visual feature extraction: The image is analyzed through multi-round prompting guidance model to extract the structured visual feature vector [LE D_Status: Yellow slow flash, Enclosure_Damage: 0.1, Tilt_Angle: 5°, Dirt_Coverage: 0.3, Corrosion_Level: 0.2, Power_Component_Status: Battery no swelling, no obstruction]; ④ Multimodal feature fusion: Weighted fusion protocol features (weight 0.4), visual features (weight 0.3), channel quality index (RSSI=-92dBm, weight 0.2), topology association information (parent node MAC address: 0x00) 00 02 CA 16 04 00 03, weight 0.05), historical knowledge base features (the battery at this node has been used for 2 years, weight 0.05), forming a multimodal fault feature vector; ⑤ Fault matching and reasoning: the matching degree with rule R8 (voltage <3.0V for >60 seconds, shutdown >3 times per day) in the fault feature base is 0.85, which is judged as a complete match. The output fault type is "battery fault", the root cause is "battery aging leading to insufficient voltage", and the maintenance suggestion is "replace with the same type of lithium battery, and perform power-on calibration after replacement".

[0076] In some embodiments, a multimodal feature attention fusion algorithm is introduced in step (4), and a hybrid inference collaborative optimization algorithm is introduced in step (5).

[0077] I. The Visual Large Language Model (VLM), the multimodal feature attention fusion algorithm, and the hybrid reasoning collaborative optimization algorithm form a closed-loop collaborative system of "feature extraction → fusion enhancement → reasoning decision → two-way feedback": 1. The VLM output protocol parsing features and visual features serve as the core input for the multimodal feature attention fusion algorithm; 2. The fusion algorithm outputs an optimized multimodal fault feature vector, which serves as the input to the hybrid inference collaborative optimization algorithm; 3. The inference algorithm outputs fault diagnosis results (fault type, confidence level, conflict signal), which are then fed back to the VLM and fusion algorithm. Transmit a "feature focusing signal" to the VLM to adjust the weights of visual feature extraction (e.g., prioritize strengthening features in fault-related regions). The "feature importance feedback" is passed to the fusion algorithm to dynamically adjust the attention weights (such as increasing the weight of high-confidence fault-related features). 4. Based on the feedback-adjusted feature extraction results, VLM inputs them again into the fusion algorithm to achieve dynamic interaction and adaptive optimization among the three.

[0078] II. Module 1: Visual Large Language Model (VLM) VLM undertakes the dual tasks of "protocol semantic parsing" and "visual-physical feature extraction", outputting standardized protocol feature vectors and structured visual feature vectors, while receiving feedback signals from the hybrid inference algorithm and dynamically adjusting the feature extraction weights.

[0079] 1. Model Construction: Assume that the VLM outputs two types of basic features after processing, and simultaneously receives inference feedback signals to adjust the extraction strategy: The normalized protocol feature vector output by VLM. For protocol feature dimensions, For the first Normalized semantic feature values ​​of each protocol field (such as the semantic confidence of the fault code); : The structured visual feature vector output by VLM For visual feature dimensions, For the first Normalized feature values ​​of visual attributes (such as confidence level of casing damage, intensity of indicator light alarm); The feature adjustment signal vector fed back by the hybrid inference algorithm. For the feedback signal dimension, For the first The adjustment coefficient of each feature (the larger the value, the more relevant the feature is to the fault, and the more it needs to be extracted). The dynamic weights for feature extraction in the VLM protocol are corrected by inference feedback signals; The dynamic weights for VLM visual feature extraction are corrected by inference feedback signals, satisfying the following conditions: ; VLM feedback adjustment coefficient (empirical value 0.8), which controls the degree of influence of the feedback signal on the extracted weights.

[0080] ①Inference feedback-driven extraction weight adjustment formula: VLM based on the feedback signal output by the hybrid inference algorithm The extraction weights of protocol features and visual features are dynamically adjusted to ensure that key features are enhanced. (1) (2) In formula (1), This is the average value of the feedback signal corresponding to the protocol characteristics, reflecting the degree of correlation between the protocol characteristics and the fault; The mean of the feedback signal corresponding to the visual feature reflects the degree of correlation between the visual feature and the fault; core logic: if the inference feedback shows that the protocol features are more critical ( ),but Increasing the VLM value enhances the semantic parsing accuracy of protocol fields; conversely, decreasing it enhances visual feature extraction.

[0081] ② Adjusted VLM feature output formula: VLM outputs optimized basic features based on the corrected weights (strengthening / weakening is achieved through the product of the weights and the original features): (3) (4) The optimized standardized protocol feature vector, The optimized structured visual feature vectors are used as inputs to the multimodal feature attention fusion algorithm.

[0082] 2. Model Training Process (1) Training data preparation: Collect 8000 sets of WSN fault samples. Each set of samples includes: original wireless messages (heartbeat packets, alarm packets, status packets) and corresponding manually labeled protocol semantics; node images and corresponding manually labeled visual attributes (indicator status, shell damage, etc.); fault diagnosis result labels (such as battery fault, radio frequency fault) and feature association labels (labeling which protocol / visual features are most critical for diagnosis).

[0083] (2) Training objective: Minimize VLM feature extraction error (MSE loss between original features and manually labeled features), and optimize the adaptability of the feedback adjustment mechanism (ensure that the correlation between features and faults is improved after weight adjustment).

[0084] (3) Training steps: 1. Initialize the VLM base model (based on the BLIP-2 pre-trained model, with the visual encoder being ViT-L / 14 and the text decoder being Flan-T5-XL); 2. Fix The feature extraction capability of VLM was trained using 6000 sets of samples, and the model parameters were optimized. ,make and Approximating manually labeled features; 3. Simulating the feedback process using the remaining 2000 samples: manually generating feedback signals. Training VLM is based on Adjust weights and The ability, the goal is to make the adjusted and The correlation with fault labels was improved by more than 15%; 4. After 30 rounds of iterative training, the VLM model parameters were finally determined. and feedback adjustment coefficient .

[0085] 3. Model Application Process: 1. The VLM receives the original wireless messages and node images, and initially outputs... and 2. Receive feedback from the hybrid inference algorithm Calculated using formulas (1) and (2) and 3. Output using formulas (3) and (4) and 4. If the confidence of the subsequent inference result is lower than the threshold (e.g., 0.7), then VLM is triggered to re-extract features (based on the updated feedback signal).

[0086] 4. Core Functions: Provides high-quality, structured basic features to solve the problems of semantic ambiguity and lack of physical state in traditional feature extraction; receives inference feedback to dynamically adjust the extraction strategy, ensuring the correlation between features and faults, and providing accurate input for subsequent fusion and inference.

[0087] Module 2: Multimodal Feature Attention Fusion Algorithm The algorithm receives the basic features optimized by VLM, integrates auxiliary features such as channel quality, topology correlation, and historical knowledge, and receives feedback from the hybrid inference algorithm to dynamically adjust the attention weights and output the optimized multimodal fault feature vector.

[0088] 1. Model Construction: Assume the fusion algorithm processes multiple features and simultaneously receives inference feedback to adjust the attention weights. : The optimized protocol feature vector output by VLM (from Equation 3); : The optimized visual feature vector output by VLM (from Equation 4); Channel quality feature vector For channel feature dimensions, For the first Normalized values ​​of channel metrics (such as RSSI, packet loss rate); : Topological association feature vector For topological feature dimension, For the first The normalized value of a topological attribute (such as the stability of parent node connections); : Feature vector of historical knowledge base As a historical characteristic dimension, For the first Normalized values ​​of historical attributes (such as the recurrence probability of similar faults); : Feature-adjusted signal vector fed back by the hybrid inference algorithm (consistent with that received by the VLM); Attention weights for protocol features Attention weights for visual features Attention weights for channel features Attention weights for topological features Attention weights for historical features, satisfying ; Initial importance coefficients for each modality (determined through training). ); Real-time reliability score for each modality (calculated based on data integrity; if the message is intact, then...). If the image is blurry... ); : Feedback adjustment coefficient of the fusion algorithm (empirical value 0.75), which controls the impact of inference feedback on attention weights.

[0089] ① Attention Weight Calculation (Integrated Inference Feedback): Attention weight is determined by the combined effect of three factors: initial importance coefficient, real-time reliability score, and inference feedback signal. The formula is as follows: (5) In formula (5), The mean of the corresponding modal feedback signal (e.g.) ), This is a feedback enhancement factor; the stronger the feedback signal, the larger this factor, and the higher the weight of the corresponding mode. Core logic: It not only considers the importance and reliability of the modes themselves, but also strengthens the weight of fault-related modes by combining inference feedback, thereby improving the targeting of fused features.

[0090] ② Multimodal feature fusion formula: The optimized multimodal fault feature vector is obtained by weighting and summing the features of each modality with their corresponding attention weights. : (6) For dimensionally unified fusion feature vectors (dimensions) Low-dimensional features (completed with zero-padding) serve as input to the hybrid inference collaborative optimization algorithm; simultaneously, the fusion algorithm calculates the fusion contribution of each modality's features. ( (for Pearson correlation coefficient), and Feedback is given to VLM to help it adjust the extraction weights (the higher the contribution, the more VLM needs to strengthen the extraction of that modality feature).

[0091] 2. Model Training Process (1) Training data preparation: reuse 8000 sets of samples trained by VLM, and supplement each sample with channel quality, topology association, historical knowledge data and manually labeled fusion features (label the optimal fusion feature vector).

[0092] (2) Training objective: Minimize the MSE loss of the fused features and the manually labeled fused features, while ensuring that the attention weights are sensitive to the feedback signal (the weights change by no less than 5% when the feedback signal changes by 10%).

[0093] (3) Training steps: 1. Initialize the initial importance coefficients Feedback adjustment coefficient 2. Train the attention weight calculation logic using 6000 sets of samples and optimize it. , to enable fusion features Approximating manually labeled fusion features; 3. Simulating inference feedback using 2000 sets of samples, manually generating... The sensitivity of training weights to feedback, adjustment 4. Iterate through the training for 25 rounds to ensure effective feedback, and finally determine the optimal value. , .

[0094] 3. Model application process: 1. Receive VLM output and and the collected , , 2. Calculate the real-time reliability score for each mode. (like Message integrity Image clarity); 3. Receiving hybrid inference algorithm Attention weights are calculated using formula (5). ;4. Generate using formula (6) 5. Calculate the fusion contribution of each modality and pass it to the hybrid inference collaborative optimization algorithm. Feedback is sent to the VLM to assist in its adjustment. and .

[0095] 4. Core Functions: It solves the problem of fixed weights in traditional fusion methods that cannot adapt to dynamic scenarios. It dynamically optimizes weights through attention mechanisms and inference feedback; it achieves deep collaboration of multimodal features, strengthens fault-related features and weakens irrelevant features, providing high-quality input for inference algorithms; it feeds back the fusion contribution to the VLM, forming a two-way interaction between the "VLM-fusion algorithm" to improve the overall feature quality.

[0096] IV. Module 3: Hybrid Inference Collaborative Optimization Algorithm The algorithm receives the multimodal fault feature vector output by the fusion algorithm, combines it with the fault feature library to achieve the synergy of rule matching and VLM semantic reasoning, outputs the fault diagnosis result, and feeds back adjustment signals to VLM and the fusion algorithm to form a closed-loop optimization.

[0097] 1. Model Construction: The inference algorithm combines rule matching and VLM semantic reasoning, while simultaneously outputting feedback signals. : The fused feature vector output by the multimodal feature attention fusion algorithm (from formula 6); Fault feature library, containing Structured rules, each rule ( For feature condition set, Fault category, (for maintenance measures) Weights for rule matching The weights of VLM semantic reasoning satisfy... ; : Integration of features and rules The matching degree (calculated by cosine similarity); :rule The corresponding fault confidence level; Faults in VLM semantic reasoning output Confidence level Let be the conditional probability inference function of VLM; :Fault The final confidence level; The final diagnosed fault type (the fault with the highest confidence level). The difference between the highest confidence level and the second highest confidence level (used to determine whether there is a conflict); The inference feedback signal vector (consistent with the previous definition) is calculated from the correlation between the fused features and the fault category. ,in This is the one-hot encoded vector corresponding to the fault category.

[0098] ① Dynamic adjustment formula for inference weights (feedback from fusion algorithm): The inference algorithm receives the modal fusion contribution output by the fusion algorithm. Dynamically adjust the weights of rule matching and VLM semantic reasoning: (7) (8) In formula (7), rule matching depends on protocol (S), channel (C), and topology (T) features (these features are easier to formalize into rules), so its weight is determined by the sum of the fusion contributions of these three modalities; VLM semantic reasoning depends on visual (V) and historical (H) features (these features require more semantic understanding), so its weight is determined by the sum of the fusion contributions of the remaining modalities; the core logic is that the modality with the higher fusion contribution has a higher weight for the corresponding reasoning method (rule / VLM), thus achieving dynamic adaptation of "fusion effect → reasoning method".

[0099] ② Formula for fault conflict resolution and feedback signal generation: When ( When the threshold for conflict is reached, a fault conflict is determined to exist. VLM semantic reasoning must be invoked to resolve the conflict, and the feedback signal must be updated. (9) In formula (9), To provide feedback on the update step size, for The gradient of the fused features (the gradient direction indicates how adjusting the features can increase the confidence difference); the updated Feedback is provided to the fusion algorithm and VLM to guide them in adjusting the feature fusion weights and extraction weights, thereby resolving conflicts.

[0100] 2. Model Training Process (1) Training data preparation: reuse 8,000 sets of samples from the first two modules, supplement the fault feature library rules (including 50 rules covering hardware, communication, software and environmental faults) and manually labeled fault diagnosis results.

[0101] (2) Training objectives: maximize the fault diagnosis accuracy (target ≥ 92%), minimize the conflict occurrence rate (target ≤ 5%), and ensure the effectiveness of feedback signals for feature adjustment (conflict resolution rate after feedback ≥ 90%).

[0102] (3) Training steps: 1. Initialization , Conflict threshold Feedback update step size 2. Train the collaborative logic of rule matching and VLM semantic reasoning using 6000 sets of samples, and optimize it. and 3. A dynamic adjustment mechanism to improve diagnostic accuracy; 4. Using 2000 sets of samples to simulate fault conflict scenarios, train the conflict resolution logic, and adjust... The accuracy was reduced to 0.05 to ensure that the conflict resolution rate met the target after feedback; 4. After 20 rounds of iterative training, the final diagnostic accuracy reached 93.5%, the conflict occurrence rate was 4.2%, and the conflict resolution rate was 91.8%.

[0103] 3. Model Application Process: 1. Receive the output of the fusion algorithm. Contribution to modal fusion 2. Calculate using formulas (7) and (8) and 3. Calculate the matching degree between the fusion features and each rule. ,get 4. Inputting VLM for semantic reasoning yields... 5. Calculation ,Sure and maintenance measures 6. Calculation ,like Generate by formula (9) Feedback is provided to the fusion algorithm and VLM; 7. The fusion algorithm and VLM adjust the weights based on the feedback, regenerate features, and input them into the inference algorithm until the conflict is resolved. ).

[0104] 4. Core Functions: To achieve collaborative optimization of rule matching and VLM semantic reasoning, overcoming the limitations of single reasoning methods (rule matching struggles to handle fuzzy faults, and VLM reasoning lacks structured constraints); to provide feedback adjustment signals to VLM and fusion algorithms, forming a closed loop of "reasoning → feature → reasoning", improving diagnostic accuracy and conflict resolution capabilities; and to output clear fault types and maintenance suggestions, completing an automated closed loop from diagnosis to maintenance.

[0105] V. Interaction Process and Example Verification of the Three Major Components 1. Complete interaction process 1. Initial Feature Extraction: VLM processes the original message and image, and outputs... , Initial weight extraction , ; 2. Initial Feature Fusion: The fusion algorithm receives... , ,as well as , , ;calculate , , , , Initial feedback The weights are calculated using formula (5): Molecular part: Similarly, the values ​​for other modal molecules were calculated: 1.342, 1.183, 1.127, and 1.065. Weight: , , , , ; Fusion characteristics: ; 3. Initial Inference: The inference algorithm receives... Calculate the rule matching degree ( (For battery fault rules), VLM semantic inference confidence ; Integration contribution , , , , ; Calculated using formula (7) , Final confidence level Second highest confidence level , No conflict, output fault type "battery fault", maintenance suggestion "replace with the same model lithium battery"; 4. Feedback Adjustment: Inference Algorithm Calculation This feedback is then fed back to the fusion algorithm and VLM; 5. Dynamic optimization: Fusion algorithm reception Recalculate the weights , Output new VLM receiver and ,calculate , We can obtain the result through formula (1). , Output new ; 6. Final Inference: The inference algorithm receives new... ,calculate , The confidence level of the diagnostic results increases, and the final conclusion is output.

[0106] 2. Core Conclusions of the Interaction The three components achieve bidirectional interaction through a closed loop of "feature → fusion → inference → feedback → feature", rather than one-way transmission; VLM and the fusion algorithm interact through "extracting weights - fusion contribution", the fusion algorithm and the inference algorithm interact through "fusion features - inference feedback", and VLM and the inference algorithm interact through "feature adjustment signals", thus satisfying the pairwise interaction requirements; After interaction, the confidence level of fault diagnosis increased from 0.806 to 0.85, verifying the effectiveness of collaborative optimization.

[0107] VI. Summary of Core Contributions 1. A closed-loop collaborative system of "VLM-fusion algorithm-inference algorithm" was constructed, which solved the problem that feature extraction, fusion and inference are independent in traditional technology and improved the adaptive capability of diagnosis; 2. A two-way feedback mechanism was designed, and the three were dynamically adjusted through formulas (1), (5), (7) and (9) to ensure that features and inference always focus on the core of the fault.

[0108] VII. Complete Steps After Incorporating Three Complete Algorithms Step (1) Multimodal data acquisition: Through fixed gateways, mobile sniffing nodes and visual inspection devices, wireless packets (heartbeat packets, alarm packets, etc.), node images (indicator lights, appearance, etc.), channel quality indicators, and topology association information are collected. The multi-source data are aligned and packaged through a time synchronization mechanism to provide basic data for subsequent algorithms.

[0109] Step (2) Automatic Protocol Parsing (VLM-driven): The collected raw wireless packets are input into the VLM. Combined with multimodal context information such as protocol document image slices and known protocol descriptions, the VLM automatically infers the packet structure, field meanings and fault semantics, and generates standardized protocol fields and semantic annotation information, which serve as one of the core inputs for multimodal fusion.

[0110] Step (3) Visual feature extraction (VLM driven): Input the collected node images into the VLM, guide the VLM to focus on key areas of the node (indicator lights, power supply components, etc.) through multi-round interactive text prompts of hierarchical topics, extract the visual features related to the physical state, convert them into visual feature vectors through structured processing, and together with the protocol features in step (2) constitute the basic feature set.

[0111] Step (4) Multimodal feature fusion (multimodal feature attention fusion algorithm): Receive the standardized protocol features from step (2), the visual feature vector from step (3), as well as the collected channel quality indicators, topology association information, and historical knowledge base features. Calculate the real-time reliability score and inference feedback weight of each modality through the multimodal feature attention fusion algorithm, dynamically allocate fusion weights, and generate a multimodal fault feature vector to improve the correlation between features and faults.

[0112] Step (5) Fault matching and reasoning (hybrid reasoning collaborative optimization algorithm): The multimodal fault feature vector from step (4) is matched with the fault feature library. The weights of rule matching and VLM semantic reasoning are dynamically adjusted through the hybrid reasoning collaborative optimization algorithm. Candidate faults are activated first through rule matching, and then VLM semantic reasoning is used to resolve conflicts and analyze root causes. At the same time, feature adjustment feedback signals are output and transmitted in reverse to the VLM and multimodal feature attention fusion algorithm to optimize the subsequent feature extraction and fusion process. Finally, the fault type, root cause and maintenance suggestions are output to form an automated closed loop.

[0113] In some embodiments, the core reason for choosing the BLIP-2 basic model as the Visual Large Language Model (VLM) is that it has strong visual-language alignment capabilities, can accurately support fine-grained visual question answering tasks, can effectively parse fault-related visual features such as the flashing pattern of node indicator lights, shell damage, and power supply component status, and its text decoder's semantic reasoning capabilities can meet the structural inference and fault semantic association requirements of heterogeneous WSN protocols, and adapt to multimodal context information fusion parsing scenarios.

[0114] In some embodiments, the VLM adaptation process for WSN fault diagnosis tasks includes: supplementing the training data with domain visual concept annotations for sensor node hardware (indicator lights, batteries, antennas, housings), fine-tuning the target to focus on the dual tasks of "accurate extraction of visual features + semantic parsing of protocol fields"; freezing the underlying parameters of the visual encoder (ViT-L / 14) during training, and only fine-tuning the parameters of the top-level network and text decoder (Flan-T5-XL) to balance the model's generalization ability and WSN domain adaptability, ensuring that the accuracy of protocol parsing and the accuracy of visual feature extraction are both improved by more than 15%.

[0115] In some embodiments, the real-time reliability score (r_attn,modal) in the multimodal feature attention fusion algorithm is calculated using quantitative indicators: Data integrity = number of valid fields in the modality / total number of fields in the protocol / task definition (e.g., wireless message integrity = number of successfully parsed protocol fields / total number of fields in the protocol definition, image integrity = percentage of clear pixels in key areas such as indicator lights and power supply components / total number of pixels in key areas); Data consistency = 1 - (feature variance of multi-frame data in the modality / preset variance threshold), the smaller the variance, the higher the consistency; finally, r_attn,modal = data integrity × 0.6 + data consistency × 0.4, with the value range constrained between [0,1].

[0116] In some embodiments, the initial importance coefficient (α_attn) of the multimodal feature attention fusion algorithm is determined jointly by "historical data statistics + domain expert experience": First, based on 1000 sets of WSN historical fault samples, the contribution of each modal feature (protocol, vision, channel, topology, and historical knowledge) to the fault diagnosis results is statistically analyzed; then, adjustments are made in conjunction with the domain expert's empirical judgment that "protocol / channel / topology features are easier to formalize into rules, while visual / historical features rely more on semantic understanding"; finally, through 5 rounds of cross-validation optimization, the optimal value of α_attn=[0.32,0.26,0.19,0.14,0.09] is determined.

[0117] In some embodiments, the feedback adjustment coefficients (β_VLM, γ_attn, γ_inf) range from [0.5, 1]. The specific values ​​are calibrated through simulation experiments: when β_VLM (VLM feedback adjustment coefficient) is 0.8, the correlation between VLM feature extraction and fault category is optimally improved; when γ_attn (fusion algorithm feedback adjustment coefficient) is 0.75, the sensitivity of attention weights to inference feedback signals meets the requirement that "when the feedback signal changes by 10%, the weight change is not less than 5%"; when γ_inf (inference algorithm feedback update step size) is set to 0.05, system oscillations caused by excessive feedback adjustment can be avoided, ensuring a stable improvement in fault diagnosis confidence.

[0118] In some embodiments, the rules for the fault feature library are constructed from two sources: first, the experience of domain experts on WSN hardware faults (sensors, batteries, Flash, RTC), communication faults (RF, link, protocol), and software faults (program execution, firmware); second, the mining and analysis of more than 10,000 historical fault cases, by associating multimodal features with fault results, automatically extracting high-frequency, high-confidence feature combinations as rule condition sets, ensuring that the rule library covers more than 85% of typical fault scenarios.

[0119] In some embodiments, the fault feature library supports a self-learning update mechanism: when VLM semantic reasoning successfully resolves a new type of fault without matching rules, the system automatically records the multimodal feature conditions, root causes, and effective maintenance measures of the fault, and includes them in the rule library after manual review and confirmation; at the same time, the rule matching success rate is calculated every 3 months, and rules with a matching success rate of less than 50% are optimized and adjusted, and invalid rules that are repeatedly triggered are deleted to ensure the timeliness and adaptability of the rule library.

Claims

1. A method for WSN fault diagnosis and maintenance based on a visual large language model, characterized in that, include: (1) Multimodal data acquisition: Through a fixed gateway, a mobile sniffing node and a visual inspection device equipped with a camera unit, wireless packets, node images, channel quality indicators and topology association information of each node in the wireless sensor network are collected. At the same time, a time synchronization mechanism is established between the fixed gateway, the mobile sniffing node and the visual inspection device to map the collected multi-source data to a unified time axis and package it. (2) Automatic protocol parsing: The collected raw wireless packets are input into the Visual Large Language Model (VLM), and combined with multimodal context information, the VLM automatically infers the packet structure, field meanings and fault semantics, and generates standardized protocol fields and corresponding semantic annotation information; (3) Visual feature extraction: The collected node images are input into the VLM, and the model is guided to focus on the key areas of the nodes through multiple rounds of interactive text prompts. Visual features related to indicator light status, appearance integrity, posture and power supply component status are extracted and converted into structured visual feature vectors. (4) Multimodal feature fusion: Construct a multimodal fault feature vector that includes the standardized protocol fields, structured visual feature vector, channel quality indicators, topology association information and historical knowledge base features; (5) Fault matching and reasoning: The multimodal fault feature vector is matched with the pre-built fault feature library. Combined with the reasoning capability of the VLM, the fault type, root cause and corresponding maintenance suggestions of the node are output to form an automated closed loop of fault diagnosis and maintenance.

2. The WSN fault diagnosis and maintenance method based on a visual large language model according to claim 1, characterized in that, The wireless messages in step (1) include heartbeat packets, alarm packets, status packets and other wireless broadcast messages sent by the node, and the node images include indicator light status images, node appearance images, power supply component images and images of the surrounding environment of the node.

3. The WSN fault diagnosis and maintenance method based on a visual large language model according to claim 1, characterized in that, The mobile sniffing node described in step (1) has passive listening capabilities and can enter weak signal areas and obstructed areas where the fixed gateway is not covered, capture residual broadcasts, retransmission frames or handshake requests sent by nodes that cannot report data normally, and supplement the acquisition of the node's communication behavior characteristics.

4. The WSN fault diagnosis and maintenance method based on a visual large language model according to claim 1, characterized in that, The visual information collected by the visual inspection device in step (1) includes: the color and flashing pattern of the node indicator light, whether the shell is damaged or water has entered, whether there is any obstruction or tilting, the physical status of the power supply components, and the lighting, water accumulation and obstruction conditions of the on-site environment.

5. The WSN fault diagnosis and maintenance method based on a visual large language model according to claim 1, characterized in that, The multimodal context information mentioned in step (2) includes image slices of the protocol document, natural language descriptions of known related protocols, and system prompts for guiding model parsing. The semantic annotation information output by the VLM includes the original field value, parsed value, unit information, and fault association relationship.

6. The WSN fault diagnosis and maintenance method based on a visual large language model according to claim 1, characterized in that, The multi-round interactive text prompts in step (3) are constructed according to hierarchical topics and are used to inquire about the overall scene of the node, appearance integrity, indicator status and power supply component status in turn. The VLM generates a corresponding natural language description based on each prompt and then converts it into a structured visual feature vector through text encoding or rule post-processing.

7. The WSN fault diagnosis and maintenance method based on a visual large language model according to claim 1, characterized in that, The fault feature library mentioned in step (5) consists of structured rules. Each rule includes a feature condition set, a fault category, and a maintenance measure set. The feature condition set is a protocol parsing field, a visual extraction attribute, a network layer indicator, or a logical combination thereof. The maintenance measure set is a specific operation suggestion for the corresponding fault category.

8. The WSN fault diagnosis and maintenance method based on a visual large language model according to claim 1, characterized in that, The reasoning process in step (5) adopts a hybrid mechanism: first, the multimodal fault feature vector is precisely matched or the confidence threshold is matched with the rule conditions in the fault feature library to activate candidate faults and maintenance measures; when there is a rule conflict or no fully matching rule, the VLM is called to perform semantic reasoning to complete conflict resolution, fuzzy inference, fault root cause analysis and maintenance measure optimization.

9. The WSN fault diagnosis and maintenance method based on a visual large language model according to claim 1, characterized in that, The time synchronization mechanism described in step (1) is implemented through local clock calibration, network time protocol or timestamp embedding to ensure that wireless messages, node images and other collected data are aligned in the time dimension, thus ensuring the accuracy of multimodal feature fusion.

10. The WSN fault diagnosis and maintenance method based on a visual large language model according to claim 1, characterized in that, Step (4) introduces a multimodal feature attention fusion algorithm, and step (5) introduces a hybrid inference collaborative optimization algorithm, forming a new process as follows: (1) Multimodal data acquisition: Through fixed gateways, mobile sniffing nodes and visual inspection devices equipped with camera units, wireless messages, node images, channel quality indicators and topology association information of each node in the wireless sensor network are collected. The wireless messages include heartbeat packets, alarm packets, status packets and other wireless broadcast messages sent by the nodes. The node images include indicator status images, node appearance images, power supply component images and images of the surrounding environment of the nodes. A time synchronization mechanism is established through local clock calibration, network time protocol or timestamp embedding to map the collected multi-source data to a unified time axis and package it. (2) Automatic protocol parsing: The collected raw wireless packets are input into the Visual Large Language Model (VLM) and combined with multimodal context information, which includes image slices of the protocol document, natural language descriptions of known related protocols, and system prompts to guide the model parsing. The VLM automatically infers the packet structure, field meanings, and fault semantics, and generates standardized protocol fields and corresponding semantic annotation information. The semantic annotation information includes the original value, parsed value, unit information, and fault association relationship of the field. The standardized protocol fields and semantic annotation information serve as one of the core inputs of multimodal fusion. (3) Visual feature extraction: The collected node images are input into VLM. The model is guided to focus on the key areas of the node by multi-round interactive text prompts constructed according to the hierarchical theme. The multi-round interactive text prompts are used to inquire about the overall scene of the node, appearance integrity, indicator status and power supply component status in turn. VLM generates a corresponding natural language description based on each prompt, and then converts it into a structured visual feature vector through text encoding or rule post-processing. (4) Multimodal feature fusion: Receive the standardized protocol fields from step (2), the structured visual feature vectors from step (3), the collected channel quality indicators, topology association information, and the historical knowledge base features from pre-stored WSN fault cases, maintenance records, and equipment operation rule bases. Calculate the real-time reliability score and inference feedback weight of each modality using the multimodal feature attention fusion algorithm. The real-time reliability score is determined by the integrity and consistency of each modality's data. The inference feedback weight is generated by the feature adjustment signal output by the hybrid inference collaborative optimization algorithm. Dynamically allocate the fusion weight of each modality based on the real-time reliability score and inference feedback weight. Generate a multimodal fault feature vector by weighted summation to improve the correlation between features and faults. (5) Fault matching and reasoning: The multimodal fault feature vector from step (4) is matched with the pre-built fault feature library, which consists of structured rules containing feature condition sets, fault categories and maintenance measure sets. The weights of rule matching and VLM semantic reasoning are dynamically adjusted by a hybrid reasoning collaborative optimization algorithm. The rule matching weight is determined by the sum of the fusion contributions of protocol features, channel features and topology features, and the VLM semantic reasoning weight is determined by the sum of the fusion contributions of visual features and historical knowledge base features. Candidate faults and maintenance measures are activated first by precise matching or confidence threshold matching. When there is a rule conflict or no fully matching rule, VLM is called to perform semantic reasoning to complete conflict resolution, fuzzy inference, fault root cause analysis and maintenance measure optimization. At the same time, feature adjustment feedback signals are output and transmitted back to VLM and multimodal feature attention fusion algorithm to optimize the subsequent feature extraction weights and fusion weights respectively. Finally, the fault type, root cause and corresponding maintenance suggestions of the node are output to form an automated closed loop of fault diagnosis and maintenance.

Images