A method and system for intelligent failure detection for a head-mounted device

Abstract

The application provides a fault intelligent detection method for a head-mounted device, which is applied to fault diagnosis of a high-speed periodic packaging device with multiple cooperative motion stations. The method collects multi-modal physical data such as vibration and image through the head-mounted device to generate a fault performance vector. The fault type and subset are determined through comparison with a threshold value and cluster analysis. The fault root cause and physical position are determined by matching historical data. The head ring of the head-mounted device is driven to generate a tactile guidance signal according to the physical position. The application solves the problems of accurate tracing and rapid physical positioning of concurrent faults by combining software and hardware, and significantly improves the maintenance efficiency of complex industrial sites.

Description

Technical Field

[0001] This invention relates to the field of industrial equipment maintenance and automation technology, and in particular to a method and system for intelligent fault detection of head-mounted devices. Background Technology

[0002] Packaging equipment is a critical part of modern production lines, and its fault detection mainly relies on manual inspections, which are prone to misjudgment, and fixed sensors with limited monitoring range. These traditional methods are difficult to comprehensively collect multi-dimensional data such as visual and vibration data, and cannot effectively integrate and analyze the data to accurately pinpoint the root cause of the fault.

[0003] Especially for high-speed, cyclical packaging equipment with multiple coordinating motion stations, existing technologies have two major drawbacks:

[0004] First, there is the challenge of tracing the root cause of concurrent failures. In such equipment, multiple workstations operate in a strongly coupled manner, and a minor anomaly in one upstream workstation often triggers a cascading failure downstream. Current technology lacks effective means to accurately trace the initial "root cause" from complex mixed signals, resulting in maintenance that only addresses the symptoms, not the root cause.

[0005] Secondly, there is the challenge of translating diagnostic results into physical guidance. Even after diagnosing the fault, maintenance personnel still need to spend time visually searching complex equipment. When the fault is located in a blind spot, simple visual or auditory cues are ineffective. Existing head-mounted devices focus only on information "display," neglecting to utilize their physical structures, such as headbands, to provide a more intuitive, audiovisual-based physical guidance method to solve the "last mile" problem of moving from abstract diagnosis to precise physical location. Therefore, an innovative solution that can address these challenges is urgently needed in this field.

[0006] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of the present invention, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0007] In view of this, the present invention provides an intelligent fault detection method and system for head-mounted devices, aiming to solve the problems of existing technologies in packaging equipment fault diagnosis, such as reliance on human experience, incomplete data collection, and a disconnect between diagnostic results and physical guidance. The present invention accurately locates the root cause of faults by fusing and analyzing multimodal physical data, and innovatively utilizes the headband structure of the head-mounted device to generate tactile guidance signals, transforming abstract diagnostic conclusions into intuitive physical orientation guidance. Finally, it combines augmented reality to generate standardized maintenance information, thereby improving the accuracy of diagnosis and location, shortening maintenance time, and reducing reliance on the experience and judgment of maintenance personnel.

[0008] This invention provides an intelligent fault detection method for head-mounted devices, applied to fault diagnosis of high-speed periodic packaging equipment with multiple coordinated motion stations, including:

[0009] Multimodal physical data of the packaging equipment during operation is collected using a head-mounted device. This multimodal physical data includes vibration signals, images of the equipment surface, and current fluctuation data.

[0010] The surface images of the equipment are processed to extract wear features of mechanical parts, and combined with vibration signals and current fluctuation data to generate a fault performance vector characterizing the physical state of the packaging equipment.

[0011] The fault performance vector is compared with preset physical parameter thresholds to determine the initial fault type of the packaging equipment;

[0012] Based on the initial fault type, a clustering algorithm is used to group the current fluctuation timing features and command execution deviation patterns in the fault performance vector to obtain a classified fault subset.

[0013] The categorized subset of faults is matched with data in the historical operation record database. When the matching degree exceeds a preset value, the root cause of the fault in the packaging equipment is determined. Based on the physical location of the root cause of the fault, one or more of the multiple tactile feedback actuators configured on the headband of the head-mounted device are driven to generate a tactile guidance signal pointing to the root cause of the fault.

[0014] In some optional embodiments, the method further includes, prior to acquiring multimodal physical data:

[0015] The system acquires the periodic synchronization signal of the packaging equipment and uses the periodic synchronization signal to trigger the acquisition of vibration signals, equipment surface images, and current fluctuation data to generate data frames synchronized with the equipment's motion cycle.

[0016] In some optional embodiments, generating a fault performance vector characterizing the physical state of the packaging equipment includes:

[0017] The data frames are input into a pre-defined spatiotemporal neural network based on the kinematic model of each station of the packaging equipment, so as to decouple and extract station-level fault features associated with the operating state of each independent station under a specific motion phase.

[0018] In some alternative embodiments, determining the root cause of packaging equipment failure includes:

[0019] Multiple workstation-level fault characteristics are constructed into a time-series cause-effect graph according to the material flow and control flow sequence of the packaging equipment. The initial fault root cause workstation that causes concurrent faults is identified in the time-series cause-effect graph through a time-series cause-effect analysis algorithm.

[0020] In some alternative embodiments, processing the device surface image to extract wear features of the mechanical components includes:

[0021] Semantic segmentation is performed on the equipment surface image to quantify the geometric size of the crack and the area of ​​the leakage region in the image;

[0022] The surface image of the device is coordinate-registered with the thermal image acquired by the infrared sensor to extract the temperature gradient and thermal field non-uniformity features of the abnormal heat source.

[0023] In some optional embodiments, generating the fault performance vector includes:

[0024] Wavelet packet transform is performed on vibration signals and current fluctuation data to extract the energy distribution characteristics of vibration signals and current fluctuation data in multiple frequency bands;

[0025] The kurtosis, skewness, and information entropy values ​​of vibration signals and current fluctuation data are calculated as time-domain statistical features.

[0026] In some optional embodiments, generating the fault performance vector further includes:

[0027] Obtain the current operating parameters of the packaging equipment;

[0028] The attention mechanism model dynamically weights energy distribution characteristics and time-domain statistical characteristics based on current operating parameters to generate a fault performance vector.

[0029] In some optional embodiments, determining the initial fault type includes:

[0030] The fault performance vector is input into an unsupervised anomaly detection model pre-trained based on normal operation data of packaging equipment to calculate anomaly scores that characterize the degree of deviation from the normal state.

[0031] When the anomaly score exceeds the preset anomaly threshold, it is determined that there is an anomaly in the packaging equipment, and subsequent fault type determination is triggered.

[0032] In some optional embodiments, after matching the categorized subset of faults with data in a historical operation record database, the method further includes:

[0033] When the matching degree between the fault subset and all data in the historical operation record database is lower than the preset value, the fault subset is marked as an unknown fault, and the human-machine collaborative diagnosis process is triggered.

[0034] In some optional embodiments, the human-machine collaborative diagnostic process includes:

[0035] The multimodal physical data and preliminary analysis results associated with the unknown fault were sent to a remote expert terminal via the network.

[0036] It receives manual diagnostic conclusions returned by remote expert terminals, associates these conclusions with unknown faults, and stores them in the historical operation record database for subsequent iterative training of the model.

[0037] In some optional embodiments, generating standardized maintenance guidance information includes:

[0038] Based on the identified root cause of the fault, extract the corresponding structured maintenance process from the maintenance knowledge base;

[0039] On the display screen of the head-mounted device, the 3D model, operation animation and key parameters of the structured maintenance process are overlaid on the physical parts of the packaging equipment to be operated in an augmented reality manner.

[0040] In some optional embodiments, after generating standardized maintenance guidance information, the method further includes:

[0041] After the maintenance steps are completed, the current image of the physical component to be operated is captured by the camera of the head-mounted device;

[0042] The current image is compared with a preset standard repair result image. Only when the consistency between the two reaches the preset standard is the guidance information for the next repair step allowed to be displayed.

[0043] In some optional embodiments, the method is also used for maintenance skills assessment, including:

[0044] Load a digital twin maintenance model containing standard operating sequences and tool usage trajectories;

[0045] Real-time tracking of the three-dimensional movement trajectory of tools used by maintenance personnel during maintenance operations;

[0046] The real-time tracked 3D motion trajectory of the tool is dynamically time-warped and compared with the standard tool usage trajectory in the digital twin maintenance model to calculate a quantitative evaluation score that represents the standardization of operation.

[0047] In some optional embodiments, the method further includes, before grouping the current fluctuation timing features and command execution deviation patterns in the fault performance vector using a clustering algorithm:

[0048] Principal component analysis (PCA) is used to reduce the dimensionality of the fault performance vector, and the principal component features with the highest contribution rate are extracted for subsequent clustering calculations.

[0049] In some alternative embodiments, after determining the root cause of the packaging equipment failure, the method further includes:

[0050] Based on the root cause of the fault, a preset simulation model is invoked to predict the impact of the fault on the remaining service life of the packaging equipment without maintenance.

[0051] This invention provides an intelligent fault detection system for head-mounted devices, comprising:

[0052] A headband is mounted around a head-mounted device, and multiple haptic feedback actuators are arranged circumferentially along the upper edge of the headband.

[0053] The data acquisition unit, configured on the head-mounted device, is used to collect multimodal physical data during the operation of the packaging equipment. The multimodal physical data includes vibration signals, images of the equipment surface, and current fluctuation data.

[0054] The feature generation unit processes the equipment surface image to extract wear features of mechanical parts, and combines vibration signals and current fluctuation data to generate a fault performance vector characterizing the physical state of the packaging equipment.

[0055] The preliminary diagnostic unit is used to compare the fault manifestation vector with preset physical parameter thresholds to determine the preliminary fault type of the packaging equipment;

[0056] The fault classification unit is used to group the current fluctuation timing features and command execution deviation patterns in the fault performance vector based on the preliminary fault type using a clustering algorithm, so as to obtain a classified fault subset.

[0057] The root cause determination unit is used to match the classified fault subset with the data in the historical operation record database. When the matching degree exceeds the preset value, the root cause of the packaging equipment fault and its physical location are determined.

[0058] A haptic feedback control unit is used to drive one or more of a plurality of haptic feedback actuators based on the physical location of the fault source to generate a haptic guidance signal pointing to the fault source.

[0059] In some optional embodiments, the data acquisition unit is further configured to:

[0060] The system acquires the periodic synchronization signal of the packaging equipment and uses the periodic synchronization signal to trigger the acquisition of multimodal physical data in order to generate a data frame that is synchronized with the equipment's motion cycle.

[0061] In some optional embodiments, the feature generation unit is further configured to:

[0062] The data frames are input into a spatiotemporal neural network based on the kinematic model of each station of the packaging equipment to decouple and extract station-level fault features associated with the operating state of each independent station under a specific motion phase.

[0063] The root cause determination unit is also used for:

[0064] Multiple workstation-level fault characteristics are constructed into a time-series cause-effect graph according to the material flow and control flow sequence of the packaging equipment. The initial fault root cause workstation that causes concurrent faults is identified in the time-series cause-effect graph through a time-series cause-effect analysis algorithm.

[0065] In some optional embodiments, the root cause determination unit is further configured to:

[0066] When the matching degree between the fault subset and all data in the historical operation record database is lower than the preset value, the fault subset is marked as an unknown fault, and the human-machine collaborative diagnosis process is triggered. The human-machine collaborative diagnosis process includes sending the multimodal physical data associated with the unknown fault to the remote expert terminal, and receiving the manual diagnosis conclusion returned by the remote expert terminal to update the historical operation record database.

[0067] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention.

[0068] The present invention provides a method and system for intelligent fault detection in head-mounted devices, which has the following beneficial effects:

[0069] By integrating multimodal data and addressing root causes, diagnostic accuracy has been improved. The innovative use of a headband to generate tactile guidance transforms diagnostic results into physical location guidance, significantly shortening fault location time. Combined with AR visualization guidance, reliance on maintenance personnel skills has been reduced. Through a self-learning mechanism, the system's diagnostic capabilities continuously evolve, laying the foundation for predictive maintenance. Attached Figure Description

[0070] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings.

[0071] Figure 1 This is a flowchart of an embodiment of the intelligent fault detection method for head-mounted devices according to the present invention;

[0072] Figure 2 This is a schematic diagram of the structure of an intelligent fault detection system for head-mounted devices according to an embodiment of the present invention. Detailed Implementation

[0073] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the invention will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0074] Furthermore, the accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0075] The flowchart shown in the attached diagram is merely an illustrative example and does not necessarily include all steps. For example, some steps may be broken down, while others may be combined or partially combined. Therefore, the actual execution order may change depending on the specific circumstances.

[0076] This invention relates to intelligent fault detection in packaging equipment using a head-mounted device, which relies on the deep integration of multimodal data fusion, pattern recognition, physical interaction guidance, and augmented reality technology. Multimodal data fusion integrates data from different sensors (such as cameras, vibration sensors, and current sensors) to provide a comprehensive description of the equipment's condition. Pattern recognition algorithms, particularly clustering analysis, are used to extract potential fault patterns from complex datasets. This invention combines digital diagnostic results with physical orientation guidance: the system converts the identified physical location of the fault's root cause into directional guidance signals emitted by haptic feedback actuators on the head-mounted device's headband. This haptic feedback technology surpasses traditional visual and auditory cues, providing maintenance personnel with a more intuitive and instinctive directional guidance, solving the challenge of moving from abstract diagnosis to precise physical location. Based on this physical guidance, augmented reality technology overlays detailed digital maintenance information onto the real world, providing maintenance personnel with intuitive operational guidance. By integrating these technologies, the system can diagnose faults more accurately and seamlessly translate diagnostic results into a closed-loop maintenance instruction process from "location finding" to "specific operation," thereby improving maintenance efficiency.

[0077] like Figure 1 As shown, this embodiment of the invention provides an intelligent fault detection method for head-mounted devices. This method is applied to fault diagnosis of high-speed, periodic packaging equipment with multiple coordinating motion stations. The method is characterized by the following steps:

[0078] S100 collects multimodal physical data of the packaging equipment during operation via a head-mounted device. This multimodal physical data includes vibration signals, images of the equipment surface, and current fluctuation data. In one specific implementation, the head-mounted device is augmented reality glasses equipped with an inertial measurement unit (IMU). The IMU collects triaxial acceleration data of key components of the equipment as vibration signals at a frequency of, for example, 200Hz. Simultaneously, a camera integrated into the augmented reality glasses captures color images of the equipment surface at a rate of, for example, 30 frames per second. Current fluctuation data is measured by a clamp meter and transmitted to the head-mounted device via wireless communication methods such as Bluetooth. In other optional implementations, sound signals can be collected via a microphone, or temperature distribution data can be acquired via an infrared camera to increase the dimensionality of the multimodal physical data and improve the comprehensiveness of the diagnosis.

[0079] S200 processes images of the equipment surface to extract wear features of mechanical components and combines them with vibration signals and current fluctuation data to generate a fault performance vector characterizing the physical state of the packaging equipment. For the processing of the equipment surface images, a pre-trained convolutional neural network (e.g., Mask R-CNN) can be used for semantic segmentation to identify and quantify surface cracks, oil stains, or corrosion areas as wear features. For the vibration signals and current fluctuation data, time-domain and frequency-domain analyses can be performed: for example, calculating the root mean square value, kurtosis, skewness, and other time-domain statistical features of the signal, and extracting frequency-domain features such as energy distribution in different frequency bands through Fast Fourier Transform (FFT) or wavelet packet transform. Finally, all the extracted quantized wear features, time-domain features, and frequency-domain features are combined into a high-dimensional mathematical vector, i.e., the fault performance vector.

[0080] S300 compares the fault manifestation vector with preset physical parameter thresholds to determine the preliminary fault type of the packaging equipment. These preset physical parameter thresholds are pre-set based on the equipment's design specifications, historical operating data, and expert experience. For example, a vibration energy threshold can be set; when the dimension value corresponding to high-frequency vibration energy in the fault manifestation vector exceeds this threshold, the preliminary fault type is determined to be "abnormal mechanical wear." Similarly, when the dimension value corresponding to the harmonic content of the motor current exceeds another preset threshold, the preliminary determination is "abnormal electrical drive." When processing multiple parameters, fuzzy logic or decision tree models can also be used to comprehensively evaluate multiple dimension values ​​to improve the accuracy of the preliminary judgment.

[0081] Based on the initial fault type, the S400 uses a clustering algorithm to group the current fluctuation timing features and command execution deviation patterns in the fault manifestation vectors to obtain a classified fault subset. The command execution deviation pattern can be obtained in real time from the equipment's PLC (Programmable Logic Controller), reflecting the difference between the actual motion trajectory of actuators such as servo motors and the command trajectory. Clustering algorithms such as DBSCAN or K-Means are used to analyze the continuously generated fault manifestation vector sequence over a period of time (e.g., the past 10 seconds), grouping vectors with similar change patterns into one category to form a fault subset. For example, a fault subset might exhibit "periodic current spikes occurring synchronously with command execution deviations at the folding station," providing a more refined classification basis for subsequent root cause localization.

[0082] S500 matches the categorized subset of faults with data in the historical operation record database. When the matching degree exceeds a preset value, the root cause of the packaging equipment fault is determined. Based on the physical location of the root cause, one or more of the multiple tactile feedback actuators configured on the headband of the head-mounted device are driven to generate a tactile guidance signal pointing to the root cause of the fault. This step specifically includes:

[0083] Matching and Root Cause Determination: Algorithms such as cosine similarity are used to calculate the similarity between the feature patterns of the current fault subset and the various fault patterns labeled in the historical database. When the highest similarity exceeds a preset value (e.g., 0.85), the root cause of the historical fault (e.g., "cam wear at station 3") is determined as the root cause of the current fault.

[0084] Physical positioning and tactile guidance: The system retrieves the physical spatial coordinates of the "cam at workstation #3" from a pre-set 3D digital model of the equipment. Simultaneously, the system uses the spatial positioning module built into the head-mounted device to obtain the real-time position and orientation of the maintenance personnel's head. The system then calculates the direction vector from the maintenance personnel's head to the physical spatial coordinates and drives one or more miniature vibration motors on the headband at the corresponding location to generate pulse vibrations of a specific frequency. The maintenance personnel will intuitively feel a clear directional guide from the physical world and subconsciously turn in that direction until the vibration shifts to directly in front, thus quickly and accurately focusing their vision on the physical area where the fault originates, greatly reducing search and judgment time.

[0085] Through the above-described scheme, this embodiment can utilize a head-mounted device to collect multimodal physical data during the operation of packaging equipment. Combined with advanced signal processing and machine learning technologies, it can achieve precise root cause tracing of concurrent faults in complex packaging equipment. Compared with existing technologies, this embodiment not only enables comprehensive perception and intelligent analysis of equipment status, but more importantly, it innovatively transforms diagnostic results into physical tactile guidance signals generated by the headband. This solves the problem of moving from abstract diagnosis to precise physical location, revolutionarily shortening the time maintenance personnel spend locating fault points in complex environments, thereby significantly reducing the dual dependence on maintenance personnel's advanced diagnostic experience and familiarity with equipment structure.

[0086] In one specific implementation, prior to acquiring multimodal physical data, the method further includes:

[0087] First, periodic pulse signals are acquired from the spindle encoder of the packaging equipment. Specifically, the encoder is configured to generate a 5V square wave pulse each time the spindle rotates one revolution. This pulse signal is then input to the GPIO interface of the intelligent sensing terminal as an external synchronization trigger signal for data acquisition. The intelligent sensing terminal includes a programmable system-on-chip (SoC) programmed to initiate data acquisition from the camera, IMU, and current sensor upon receiving each rising edge trigger signal. Next, under the control of the SoC, the camera, IMU, and current sensor synchronously acquire a frame of image data, a segment of vibration data (e.g., 1000 sampling points), and a segment of current data (e.g., 1000 sampling points). These synchronously acquired data frames are timestamped and labeled with corresponding phase tags (0-359 degrees). Finally, these data frames are stored chronologically in the terminal's local cache, ready to be uploaded to the edge computing node. In some alternative implementations, the periodic synchronization signal can come from encoder signals from other periodically moving components, such as camshafts or conveyor belts; periodic features can also be extracted by analyzing the voltage or current waveforms of a motor without an external encoder.

[0088] Through the above solution, this embodiment can ensure that the data collected from multiple sensors is precisely aligned with a specific phase of the packaging equipment cycle, providing a reliable timing reference for subsequent fault diagnosis and root cause tracing.

[0089] In one specific implementation, generating a fault performance vector characterizing the physical state of the packaging equipment includes:

[0090] First, the edge computing node receives phase sequence data frames, corresponding to each movement cycle of the packaging equipment, synchronously acquired and transmitted by the head-mounted device. Specifically, the spatiotemporal neural network is pre-trained, with its input layer adapted to the format of the phase sequence data frames and its output layer corresponding to the fault feature vector of each independent workstation.

[0091] Then, the spatiotemporal neural network integrates prior knowledge encoded in the device's kinematic model, which defines the motion state of each workstation within a specific phase interval. This kinematic model can consist of a set of mathematical equations describing the motion trajectory and velocity of each workstation, or it can consist of manually labeled lookup tables indicating the activation phase of each workstation.

[0092] Next, when processing the data frames, the attention mechanism in the neural network dynamically adjusts the attention weights for different sensor data streams based on the phase information provided by the kinematic model. For example, when processing data in the phase interval corresponding to the "box-retrieving station," the network increases the weights of the vibration and vision sensors associated with that station, while decreasing the weights of sensors associated with other stations. In this way, signals from different stations that are coupled together are effectively decoupled.

[0093] This spatiotemporal neural network extracts fault features that characterize the health status of each workstation under a specific phase from decoupled data through a combination of multiple convolutional layers, recurrent layers, and attention mechanism layers. These features can be time-domain statistical features (e.g., mean, variance), frequency-domain statistical features (e.g., peak frequency, energy distribution), or high-dimensional abstract features learned by the neural network.

[0094] Each workstation ultimately corresponds to a fault feature vector, which contains multiple dimensions, each representing the operational status of the workstation in different aspects. For example, a fault feature vector describing a "folding workstation" might include the following dimensions: folding angle deviation, paper jam frequency, motor current fluctuation amplitude, etc.

[0095] In other alternative implementations, the spatiotemporal neural network can be replaced with other types of sequence models, such as Long Short-Term Memory (LSTM) networks or Gated Recurrent Unit (GRU) networks. Furthermore, the kinematic model can also be encoded in other forms, such as expert rules or fuzzy logic. Simultaneously, the neural network can be trained using different strategies, such as supervised learning, unsupervised learning, or semi-supervised learning.

[0096] Through the above scheme, this embodiment can utilize a spatiotemporal neural network that integrates prior knowledge of equipment kinematics to intelligently decouple periodically synchronized multimodal data and extract fault features that can independently characterize the health status of each specific workstation under a specific action phase, thereby providing an accurate data foundation for subsequent fault diagnosis and root cause tracing.

[0097] In one specific implementation, determining the root cause of a packaging equipment malfunction includes:

[0098] First, complete material flow and control flow information for the packaging equipment needs to be obtained. Material flow information includes the sequence of workstations through which the material passes and the material transfer relationships between each workstation (e.g., the output of workstation A is the input of workstation B). Control flow information describes the linkage control logic between the various actuators of the equipment (e.g., a signal from sensor X triggers the action of actuator Y). Material flow and control flow information can be obtained from the equipment's design drawings, PLC programs, or the experience of domain experts. Then, based on the above information, each workstation is abstracted as a node, and the material transfer or control dependencies between workstations are abstracted as directed edges, thereby constructing a temporal cause-effect graph. For example, if the material output of workstation A is the material input of workstation B, then there exists a directed edge in the graph pointing from node A to node B. The temporal relationships between nodes can be quantified based on the material flow transfer time.

[0099] Specifically, the time-series cause-effect graph can be stored using a directed acyclic graph (DAG) data structure. Each node in the graph contains the time series of fault characteristics for that workstation. For example, the time series of characteristics for workstation A is... Edges represent the direction and strength of causal relationships. Next, a suitable temporal causal analysis algorithm needs to be selected. One option is the Granger causality test. For each directed edge in the graph (e.g., from node A to node B), a Granger causality test is performed to verify the characteristic time series of node A. Is the feature time series of node B... It has significant predictive power. If Capable of significantly predicting If the result is positive, it indicates that the operating status of workstation A has a causal influence on workstation B. Granger causality tests can be performed using statistical software or programming languages ​​(such as Python's statsmodels library).

[0100] Next, based on the Granger causality test results, the initial root cause station is located in the time-series causality graph. The initial root cause station is defined as one where there are no directed edges pointing to it, or where the Granger causality test results for all directed edges pointing to it are insignificant. In other words, the fault at this station is not caused by faults at other stations, but by its own internal factors. After determining the initial root cause station, corresponding maintenance guidance information can be extracted from the knowledge base based on the fault type of that station. In other optional implementations, besides the Granger causality test, other causal inference algorithms can be used, such as Do-calculus and structural causal models (SCM).

[0101] Through the above scheme, this embodiment can utilize the material flow and control flow information of the packaging equipment to construct a time-series cause-effect graph, and through the time-series cause-effect analysis algorithm, accurately identify the initial fault root cause workstation from complex concurrent fault phenomena, thereby avoiding blind repairs that only address symptoms and significantly improving repair efficiency.

[0102] In one specific implementation, processing the surface image of the device to extract wear features of the mechanical components includes:

[0103] First, image processing algorithms are used to process the surface images of the packaging equipment acquired by the head-mounted device. Specifically, deep learning semantic segmentation models such as Mask R-CNN are used to segment target objects in the image at the pixel level, thereby identifying defect areas such as cracks, oil stains, and corrosion. The model output includes the mask, bounding box, and confidence score of the defect area. For cracks, their pixel-level length is calculated; for oil stains or corrosion areas, their pixel area is calculated. This geometric dimension information is quantified into specific numerical values, for example, the crack length is 5.2 mm, and the oil stain area is 12.5 square centimeters. Then, the visible light image acquired by the head-mounted device is precisely registered with the thermal image acquired by the infrared thermal imager. The registration process uses feature point-based image registration algorithms, such as SIFT or ORB, to extract salient feature points from the two images, and the homography matrix between the images is estimated using the RANSAC algorithm to achieve pixel-level alignment. After registration, the segmented defect areas are overlaid with the thermal image to analyze the thermal distribution of the defect areas. For example, the average temperature, maximum temperature, and temperature difference (temperature gradient) of the cracked region compared to the surrounding normal region are calculated. Furthermore, the non-uniformity of the thermal field is calculated, for example, by calculating the standard deviation or variance of the temperature in the defective region. Finally, the quantified geometric dimensional information (crack length, oil contamination area) and thermal characteristic information (temperature gradient, thermal non-uniformity) are integrated into the fault performance vector, providing a basis for subsequent fault diagnosis. In other alternative implementations, the semantic segmentation model can be replaced with other types of image segmentation algorithms, such as FCN or DeepLab; the image registration algorithm can also be replaced with a mutual information-based registration method; and the extraction of thermal characteristic information can also employ other statistical measures, such as calculating the median or percentile temperature of the defective region.

[0104] By employing the above approach, this embodiment can fuse multi-source information from visible light images and infrared thermal images, thereby extracting defect features from the equipment surface more comprehensively and accurately, providing a more reliable data foundation for subsequent fault diagnosis. Compared to methods relying solely on a single image modality, this embodiment effectively improves the accuracy and robustness of fault diagnosis.

[0105] In one specific implementation, generating the fault manifestation vector includes:

[0106] First, the raw time-domain signals acquired from the vibration and current sensors are preprocessed. For the vibration signal, moving average filtering is used to remove noise, followed by normalization to scale the amplitude to the [-1, 1] interval. Similarly, baseline correction is performed on the current signal to eliminate the DC component, and outlier removal is performed. Specifically, continuous wavelet transform is used to extract the instantaneous frequency and time-frequency energy distribution of the signal, with Morlet wavelet selected as the wavelet basis function. Then, a suitable wavelet scale range is selected, and wavelet packet decomposition is performed on the preprocessed vibration and current signals to obtain decomposition coefficients for multiple frequency bands. The energy of the decomposition coefficients for each frequency band is calculated as the energy distribution characteristic of that frequency band. Specifically, the energy calculation formula is:

[0107]

[0108] in, This represents the total energy of the i-th frequency band. This represents the wavelet packet decomposition coefficients in the i-th frequency band. The total number of coefficients contained in the i-th frequency band.

[0109] Then, statistical characteristics are directly calculated in the time domain for the preprocessed vibration and current signals. Kurtosis is calculated to characterize the steepness of the signal amplitude distribution; skewness is calculated to characterize the symmetry of the signal amplitude distribution; and information entropy is calculated to characterize the complexity of the signal. Specifically, the kurtosis calculation formula is:

[0110]

[0111] The formula for calculating skewness is:

[0112]

[0113] Where kurt is the kurtosis value, skewness value, and x is the signal time series. The mean of the signal time series is denoted as . The standard deviation of the signal time series. This represents the mathematical expectation operator.

[0114] The calculation of information entropy first requires discretizing the signal amplitude into bins, and then statistically analyzing the frequency of sample points falling within each bin interval to obtain the probability distribution. The calculation formula is as follows:

[0115]

[0116] Where H is the information entropy value, and M is the total number of bins after discretization. Let be the probability that the signal amplitude falls within the i-th bin interval.

[0117] In some other alternative implementations, other time-frequency analysis methods, such as short-time Fourier transform (STFT) or Wigner-Ville distribution, can be used to extract the energy distribution characteristics of the signal at different times and frequencies.

[0118] Through the above approach, this embodiment can extract more comprehensive and distinctive features from vibration and current signals, providing a more reliable data foundation for subsequent fault diagnosis, thereby improving the accuracy and robustness of the diagnosis.

[0119] In one specific implementation, generating the fault manifestation vector further includes:

[0120] First, the system obtains the current operating condition parameters of the packaging equipment from the PLC via the CAN bus. These parameters include, but are not limited to: production speed (e.g., number of boxes produced per minute), motor load rate, material type of the current batch, ambient temperature, and total operating time. Then, the attention mechanism model receives the operating condition parameters along with the energy distribution features and temporal statistical features extracted from the wavelet packet transform. Specifically, the attention mechanism model can be a neural network containing a multilayer perceptron. Its input layer receives the normalized operating condition parameters, the intermediate layers perform feature transformation through a nonlinear activation function, and the output layer generates a corresponding weight coefficient for each energy distribution feature and temporal statistical feature. For example, when the motor load rate is high, the model increases the weight of the current harmonic component features related to motor overload and decreases the weight of the high-frequency vibration component features related to slight bearing wear, because current features better reflect the motor's health under high load. Next, the system multiplies each energy distribution feature and temporal statistical feature by its corresponding weight coefficient to obtain a weighted feature vector, which is then used as the final fault performance vector. In some other alternative implementations, the attention mechanism model can also adopt the Transformer architecture, which uses the self-attention mechanism to learn the dependencies between features under different operating parameters, thereby dynamically adjusting the feature weights.

[0121] Through the above scheme, this embodiment can adaptively adjust the weights of different fault characteristics according to the current operating status of the equipment, thereby improving the fault diagnosis accuracy and robustness under various operating conditions.

[0122] In one specific implementation, determining the initial fault type includes: First, the system deploys an unsupervised anomaly detection model pre-trained based on historical normal operation data. Specifically, this model employs the Isolation Forest algorithm, trained using multimodal data such as vibration, current, and temperature collected from the equipment under fault-free conditions over the past three months. During training, the Isolation Forest model constructs a series of random split trees to assess the ease with which new data samples are isolated. Then, in the real-time monitoring phase, the generated fault manifestation vector is input into the Isolation Forest model, which outputs an anomaly score between 0 and 1. A higher score indicates a greater deviation of the data sample from the normal state. Next, the anomaly score is compared with a preset anomaly threshold. This threshold is set based on the statistical distribution of historical data; for example, it is set to 0.75, meaning that an anomaly is determined only when the fault manifestation vector significantly deviates from the normal range. If the anomaly score exceeds 0.75, subsequent fault type determination processes are triggered, such as initiating a K-Means clustering algorithm to classify the fault subset. In other alternative implementations, other unsupervised anomaly detection models, such as One-Class SVM and autoencoders, can be used instead of the Isolation Forest model.

[0123] Through the above solution, this embodiment can achieve real-time monitoring of equipment operating status, detect potential abnormal operating conditions in advance, avoid the occurrence of sudden failures, and reduce unplanned downtime.

[0124] In one specific implementation, after matching the categorized subset of faults with data in the historical operation record database, the method further includes: when the categorized subset of faults matches the data in the historical operation record database, the system evaluates the highest matching score. Specifically, if the highest matching score is lower than a preset similarity threshold (e.g., 0.7), the system classifies the subset of faults as an "unknown fault." Then, the system automatically triggers a human-machine collaborative diagnostic process. This process begins by the system sending a notification to maintenance personnel or remote experts, requesting manual intervention for analysis. The notification includes equipment operating parameters at the time of the fault, sensor data snapshots, and other relevant diagnostic information to assist in manual judgment. In other optional implementations, the notification can be sent through various communication channels, such as email, instant messaging, or through a dedicated interface of the maintenance management system.

[0125] Through the above solution, this embodiment can handle unknown faults that occur during equipment operation and cannot be automatically diagnosed through existing knowledge bases, thereby expanding the scope of application of the intelligent diagnostic system and improving its robustness under complex working conditions.

[0126] In one specific implementation, the human-machine collaborative diagnostic process includes: the system first sends a data packet containing multimodal physical data (such as vibration spectrum, image texture features, and current harmonic components) and preliminary analysis results (e.g., anomaly scores given by the isolated forest model, and preliminary fault type labels) via encrypted HTTPS protocol to a pre-registered remote expert terminal, which runs a customized maintenance app. This maintenance app provides an expert interactive interface that integrates a digital twin model of the equipment, allowing experts to view real-time status data from multiple perspectives, replay video clips before and after the fault, and perform in-depth analysis of the raw signals using specialized tools (such as a virtual oscilloscope). Specifically, after analysis, the expert selects the cause of the fault through a drop-down menu on the app interface, such as "feed mechanism cam wear." Simultaneously, the expert can input structured maintenance suggestions in the app, such as "replace the cam and check the lubrication system."

[0127] The app then packages the expert's diagnostic conclusions and repair suggestions, along with the expert's digital signature, and sends them back to the cloud-based intelligent platform via a secure channel. Next, the cloud-based intelligent platform reviews the diagnostic conclusions (e.g., verifying the digital signature and checking the reasonableness of the conclusions). Once approved, it associates the expert's cause of the fault, "cam wear in the feeding mechanism," with the corresponding multimodal physical data package and stores it in the historical operation record database. The database also records metadata such as the expert's qualification level and historical diagnostic accuracy rate, used for subsequent AI model training. The system automatically prioritizes diagnostic conclusions from highly qualified experts. Finally, the system uses the newly added diagnostic data to incrementally learn the pre-trained AI model, for example, adjusting the weights of the neural network to enable it to identify similar fault patterns faster and more accurately. In other optional implementations, the remote expert terminal can use WebRTC-based audio and video communication to achieve real-time voice and video communication with on-site maintenance personnel for a clearer understanding of the fault situation.

[0128] Through the above solution, this embodiment can transform the expert's manual diagnostic experience into reusable knowledge and continuously improve the diagnostic capabilities of the AI ​​model, thereby solving the problem of handling unknown faults and enabling the system to have the ability to continuously learn and evolve.

[0129] In one specific implementation, generating standardized maintenance guidance information includes:

[0130] First, based on the root cause of the fault identified through source tracing analysis, a corresponding structured maintenance process document is retrieved from the cloud-based maintenance knowledge base. Specifically, this structured document uses common data exchange formats such as XML or JSON and includes information such as maintenance steps, required tools, safety precautions, and estimated operation time. The document also contains resource links that visualize the operation steps, such as CAD model files, 3D animation files, and image files with highlighted annotations.

[0131] Then, the edge computing nodes parse the structured maintenance process document to generate an augmented reality (AR) scene description file that can be displayed on the head-mounted device. Specifically, this scene description file defines the spatial positioning relationship of the 3D model on the real device, the playback order and triggering conditions of the operation animation, and the display method of parameters (such as torque values ​​and gap dimensions). For example, the scene description file may instruct the head-mounted device to automatically overlay the 3D model of the motor onto the real motor after detecting the appearance of the motor to be repaired in the user's field of vision, and highlight the bolts that need to be removed.

[0132] Next, based on the AR scene description file, the head-mounted device overlays information such as 3D models, operation animations, and parameters onto the physical components of the packaging equipment in an augmented reality manner. For example, in the step of replacing a bearing, the head-mounted device not only displays virtual disassembly tools and operation arrows around the bearing, but also displays the measured value of the bearing clearance in real time and prompts the user to adjust it to the appropriate range.

[0133] In some alternative implementations, the structured maintenance process documents in the cloud-based maintenance knowledge base can also be described using a dedicated domain-specific language (DSL). Edge computing nodes then convert the maintenance process into an AR scene description file using a DSL interpreter. In other implementations, visualization resources such as 3D models and operation animations can be stored on local storage media to reduce reliance on network bandwidth. In still other implementations, the spatial positioning relationships of the 3D model can be achieved using pre-labeled QR codes or image feature points, without relying on complex SLAM algorithms.

[0134] Through the above solution, this embodiment can transform complex maintenance tasks into intuitive and visual AR guidance, reducing reliance on the experience of maintenance personnel and reducing secondary failures caused by operational errors.

[0135] In one specific implementation, after generating AR maintenance guidance information, when the user completes the operation corresponding to the current step, the camera on the head-mounted device activates and captures an image of the physical component the user is operating, such as the installation status of a recently replaced timing belt. The system calls a pre-trained image quality assessment model to check whether the image sharpness, lighting conditions, and shooting angle meet the requirements. If the image quality is substandard, the system will provide a voice prompt to the user to retake the image until the conditions are met.

[0136] The system then inputs the captured images into a convolutional neural network-based image comparison module. This module has been pre-trained using a large number of qualified and unqualified synchronous belt installation images. Standard maintenance result images are pre-stored in a database, including multiple dimensions such as the correct range of synchronous belt tension, pulley alignment, and the integrity of the protective cover installation. The image comparison module extracts the feature vector of the current image and performs similarity calculations with the feature vectors of the standard images, for example, using a normalized cross-correlation algorithm.

[0137] Specifically, the system sets multiple similarity thresholds, each corresponding to a different quality level. For example, a similarity score higher than 0.95 indicates "excellent" installation quality, a score higher than 0.9 but lower than 0.95 indicates "qualified", and a score lower than 0.9 indicates "unqualified".

[0138] If the similarity score is below 0.9, the AR interface will highlight the problematic area and provide specific improvement suggestions, such as "Insufficient timing belt tension; please readjust using a tension gauge." Only when the similarity score reaches the "qualified" or "excellent" standard will the system allow the display of AR guidance information for the next maintenance step, guiding the user through subsequent operations.

[0139] In other alternative implementations, deep learning methods can be used for image comparison. For example, a Siamese network can be used to directly learn a similarity metric between two images, or a metric learning method can be used to embed images into a low-dimensional space, making similar images appear closer in space. Furthermore, the standard image for comparison can not be a single image, but rather a collection of images from different angles and under different lighting conditions, from which the system selects the image most similar to the current image for comparison.

[0140] Through the above solution, this embodiment can effectively ensure the quality of maintenance operations, avoid secondary failures caused by improper operation, and significantly improve the reliability of equipment maintenance.

[0141] In one specific implementation, the method is also used for maintenance skills assessment. First, a digital twin maintenance model containing standard operating sequences and tool usage trajectories is loaded. For example, for the operation of replacing a packaging machine conveyor belt, the digital twin maintenance model predefines the sequence of operating steps: 1) loosen the tensioning mechanism; 2) remove the old conveyor belt; 3) install the new conveyor belt; 4) adjust the tension; 5) fix the tensioning mechanism. For each step, the model also includes the three-dimensional motion trajectory (including position, posture, speed, etc.) of tools such as screwdrivers and wrenches during expert operation, as well as data such as the contact force between the tools and the target components. Then, the system tracks the three-dimensional motion trajectory of the tools held by the maintenance personnel in real time at a frequency of 60Hz using an inertial measurement unit (IMU) and spatial positioning module integrated into the head-mounted device. Specifically, the IMU data is processed by Kalman filtering to reduce noise. Next, the real-time tracked three-dimensional motion trajectory of the tools is compared with the standard tool usage trajectory in the digital twin maintenance model using a dynamic time warping (DTW) algorithm. The DTW algorithm allows for non-linear bending along the time axis to align differences in operating speed among different operators. Finally, based on the DTW alignment results, a quantitative evaluation score characterizing operational standardization is calculated. For example, the following three evaluation metrics can be defined: 1) Trajectory smoothness: assessing operational smoothness by calculating the root mean square (RMS) value of the distance deviation between the actual trajectory and the standard trajectory; 2) Tool posture accuracy: assessing the accuracy of tool usage angles by calculating the Euler angle difference between the actual tool posture and the standard posture; 3) Step completion time: recording the actual completion time of each step and comparing it with a standard time range. Each metric is assigned a different weight (e.g., trajectory smoothness 40%, tool posture accuracy 30%, step completion time 30%), and the weighted sum is used to obtain the final quantitative evaluation score. In other optional implementations, a Hidden Markov Model (HMM) can also be used to model the maintenance operation process, using a forward-backward algorithm to calculate the similarity between the maintenance personnel's operation sequence and the standard operation sequence as an evaluation score.

[0142] Through the above solution, this embodiment can objectively and quantitatively assess the operational standardization of maintenance personnel, identify non-standard behaviors during operation, and provide targeted guidance, which helps to improve maintenance quality and efficiency, and reduce the risk of secondary damage to equipment due to operational errors.

[0143] In one specific implementation, before grouping the current fluctuation timing features and command execution deviation patterns in the fault performance vector using a clustering algorithm, the method further includes: first, after the edge computing node receives the fault performance vector, it performs dimensionality reduction using Principal Component Analysis (PCA). Specifically, the PCA algorithm is configured to retain principal components that can explain 95% of the variance of the original data. For example, assuming the original fault performance vector is 20-dimensional, after PCA dimensionality reduction, only 8 principal components may be retained. Then, K-Means clustering is performed using the dimensionality-reduced principal component features to group the current fluctuation timing features and command execution deviation patterns. In other optional implementations, in addition to PCA, dimensionality reduction algorithms such as Linear Discriminant Analysis (LDA) or t-distributed neighborhood embedding (t-SNE) can be used to extract the principal component features with the highest contribution rate for subsequent clustering calculations.

[0144] Through the above scheme, this embodiment can reduce the computational load of the clustering algorithm, improve the clustering speed, and filter out noise features to improve the clustering effect, thereby making the subsequent determination of the root cause of the fault more accurate.

[0145] In one specific implementation, after the root cause identification unit completes the identification of the fault root cause, the method further includes: First, calling the equipment remaining useful life prediction model corresponding to the fault root cause from the cloud-based intelligent platform. This model is a Weibull proportional hazards model trained based on historical data, and its input parameters include the current equipment operating time, vibration intensity in the fault feature vector, temperature data, and cumulative output read from the PLC. Specifically, the model outputs a probability distribution describing the probability of equipment failure at different time points. Then, the system converts this probability distribution into an intuitive remaining useful life (RUL) prediction value, such as "average remaining useful life: 48 hours, confidence interval: 40-56 hours". Next, the system integrates this RUL prediction result into the AR maintenance guidance interface, prominently displaying it to maintenance personnel and managers through overlaid text, color coding, etc. For example, when the RUL is less than 24 hours, the RUL value on the maintenance guidance interface will be highlighted in red, accompanied by a voice prompt: "The equipment is about to reach a critical state, please arrange maintenance as soon as possible." In some alternative implementations, the remaining lifetime prediction model can employ different algorithms, such as support vector regression (SVR), neural networks, or lifetime models based on physical failure mechanisms. In other implementations, the display of RUL is not limited to numerical displays on an AR interface; it can also be achieved by generating equipment health status reports and periodically pushing them to equipment administrators.

[0146] Through the above solution, this embodiment can provide quantitative reference for maintenance decisions, helping enterprises to make a more reasonable balance between maintenance costs, production losses and equipment safety, and avoid losses caused by blind maintenance or overuse.

[0147] This invention provides an intelligent fault detection system for head-mounted devices, applied to fault diagnosis of high-speed, periodic packaging equipment with multiple coordinating motion stations. For example... Figure 2 As shown, the system includes:

[0148] The headband M100, with its ring-shaped physical structure, is fitted into a head-mounted device for user wearing and securing. Multiple miniature vibration motors, serving as tactile feedback actuators, are evenly distributed circumferentially within the headband M100.

[0149] The data acquisition unit M200 is configured on the head-mounted device. This unit is used to collect multimodal physical data during the operation of the packaging equipment, including vibration signals, images of the equipment surface, and current fluctuation data. Specifically, the data acquisition unit M200 includes a triaxial accelerometer to collect vibration signals, a visible light camera to collect image information, and connects to an external clamp-on ammeter via a Bluetooth module to obtain current fluctuation data.

[0150] The feature generation unit M300 processes images of the equipment surface to extract wear features of mechanical components and combines them with vibration signals and current fluctuation data to generate a fault performance vector characterizing the physical state of the packaging equipment. In one example, the feature generation unit M300 includes an image processing module and a signal processing module, which use edge detection algorithms and fast Fourier transform techniques, respectively, to transform the raw data into quantified crack length, vibration frequency features, and root mean square current values, ultimately combining them into a multidimensional fault performance vector.

[0151] The preliminary diagnostic unit M400 compares the fault manifestation vector with preset physical parameter thresholds to determine the initial fault type of the packaging equipment. For example, its internal comparator module compares the vibration amplitude with a preset threshold (such as 5g), and if it exceeds the threshold, it is determined to be "abnormal mechanical vibration".

[0152] The fault classification unit M500 is used to group the current fluctuation timing features and command execution deviation patterns in the fault performance vector based on the initial fault type using a clustering algorithm, thereby obtaining a classified fault subset. For example, under the initial type of "motor overload", the fault is further subdivided into two subsets, "periodic overload" and "random overload", using the K-means clustering algorithm.

[0153] The root cause determination unit M600 matches the categorized subset of faults with data in the historical operation record database. When the matching degree exceeds a preset value, it determines the root cause of the packaging equipment fault and its physical location in a preset equipment coordinate system. For example, when the matching degree calculated by cosine similarity exceeds 0.9, unit M500 extracts the root cause information from the historical matching history, such as "insufficient lubrication of cam at station 3," and queries the physical coordinates of the cam from the equipment's digital model.

[0154] The haptic feedback control unit M700, based on the physical location of the fault root cause determined by the root cause determination unit M600, drives one or more of the multiple haptic feedback actuators on the headband M100 to generate a haptic guidance signal pointing towards the fault root cause. Specifically, the unit first acquires the real-time spatial orientation of the head-mounted device, then calculates the azimuth angle from the maintenance personnel pointing to the fault root cause, and finally drives the vibration motor on the headband M100 at the corresponding azimuth to generate pulse vibration, providing the user with intuitive physical orientation guidance.

[0155] The aforementioned units work collaboratively to form a complete intelligent detection closed loop: the data acquisition unit M200 provides a comprehensive information foundation for subsequent analysis; the feature generation unit M300 extracts key fault information from multimodal data; the preliminary diagnosis unit M400 and the fault classification unit M500 refine the fault types step by step, improving diagnostic accuracy; and the root cause determination unit M600 identifies the root cause of the problem by matching it with historical data. The innovative combination of the newly added tactile feedback control unit M700 and the headband M100 transforms diagnostic conclusions into physical guidance, solving the challenge of translating abstract information into precise physical location. Through this solution, this embodiment not only enhances the level of diagnostic intelligence but also improves the efficiency and accuracy of fault location for maintenance personnel in complex environments through deep integration of hardware and software.

[0156] In one specific implementation, the data acquisition unit is equipped with an encoder signal reading module. This module is connected to the programmable logic controller (PLC) of the packaging equipment via an industrial Ethernet interface (e.g., Profinet, EtherCAT) to acquire the rotary encoder signal of the equipment's spindle in real time. This encoder signal is used as a periodic synchronization signal, with each pulse or a preset number of pulses representing a specific phase of the equipment's operation. The data acquisition unit includes a phase synchronization controller that triggers data acquisition operations from a high-resolution RGB camera, an inertial measurement unit (IMU), and an external current sensor based on the received encoder signal pulses. For example, the phase synchronization controller can be configured to send acquisition commands to the camera, IMU, and current sensor at the rising edge of each encoder pulse, ensuring that the acquired image frames, vibration data, and current data are precisely aligned with the corresponding equipment phase. The acquired multimodal data is timestamped and phase-tagged, and stored as phase sequence data frames for subsequent analysis and processing. In other alternative implementations, the periodic synchronization signal may also come from proximity sensors or photoelectric switches mounted on moving parts of the equipment, rather than necessarily relying on the encoder signal. Phase synchronization controllers can also use digital circuits or software algorithms based on phase-locked loops (PLLs) to achieve phase synchronization.

[0157] Through the above solution, this embodiment can ensure the temporal consistency of multimodal data acquisition, avoid analysis errors caused by data asynchrony, and thus improve the accuracy of fault diagnosis.

[0158] In one specific implementation, the feature generation unit of the edge computing node includes a pre-trained spatiotemporal neural network based on the Transformer architecture. The network takes periodically synchronized multimodal data frames as input and outputs decoupled fault feature vectors for each workstation. This spatiotemporal neural network includes:

[0159] The embedding layer is used to convert the triaxial acceleration signals from the IMU, image frames captured by the RGB camera, and current values ​​collected by the current sensor into high-dimensional vector representations. For image frames, a pre-trained ResNet-50 model can be used to extract image features.

[0160] The temporal attention layer utilizes a multi-head attention mechanism to learn the correlations between multimodal features at different time steps (i.e., different phases), focusing on features related to the current workstation action. The calculation of the query, key, and value in this attention layer is guided by the kinematic model read from the PLC. For example, when the network processes data for the phase interval corresponding to the "box-picking station," the kinematic model provides a mask, making the attention mechanism focus more on sensor signals related to that station, such as the pressure sensor reading of the vacuum suction cup.

[0161] The spatial attention layer uses a convolutional neural network to learn the correlations between features at different spatial locations (e.g., different parts of a device).

[0162] The fusion layer combines the outputs of the temporal attention and spatial attention layers to obtain the final workstation-level fault feature vector.

[0163] The root cause determination unit of the edge computing node constructs a time-series causal graph from the fault feature vectors of each workstation, according to the material flow of the cartoning machine (e.g., the carton sequentially passes through the picking station, flap folding station, filling station, and sealing station) and control flow logic (e.g., the synchronous operation of all stations driven by the main motor). This time-series causal graph is a directed acyclic graph, where nodes represent workstations and edges represent causal relationships between workstations. The root cause determination unit uses the Granger causality test algorithm to analyze the time series data of each node in the graph. The Granger causality test determines whether a causal relationship exists between two time series by analyzing whether the past values ​​of one time series can significantly predict the future values ​​of another time series. In some alternative implementations, other causal inference algorithms can also be used, such as LiNGAM (Linear Non-Gaussian Acyclic Model) or PCMCI (Peter and Clark Momentary Conditional Independence) algorithms. For example, if the test results show that the feature vector of the "box picking station" can significantly predict the feature vector of the "tongue folding station", but the converse is not true, then it means that the "box picking station" is the initial root cause of the failure of the "tongue folding station".

[0164] Through the above scheme, this embodiment can use deep learning methods to automatically extract the fault characteristics of each workstation, and use causal inference methods to accurately identify the initial root cause of concurrent faults, thereby avoiding misjudgment by traditional methods in complex fault scenarios.

[0165] In one specific implementation, the root cause determination unit includes an unknown fault identification module, a data encapsulation module, a remote communication module, and a knowledge update module. The unknown fault identification module is configured to trigger an unknown fault processing flow and mark the fault subset as "awaiting expert diagnosis" when the matching degree between the fault subset output by the fault classification unit and all data entries in the historical operation record database is lower than a preset threshold (e.g., cosine similarity lower than 0.6). Upon receiving the fault subset in the "awaiting expert diagnosis" state, the data encapsulation module automatically collects multimodal physical data related to the fault occurrence time window (e.g., 30 seconds before and after), including but not limited to: high-resolution RGB image sequences from intelligent sensing terminals, IMU vibration data, infrared thermal imaging data, and current waveform data and PLC operation logs from external sensors. The data encapsulation module uniformly encapsulates these heterogeneous data into a compressed package and generates a metadata file containing data descriptions, fault phenomenon descriptions, and preliminary diagnostic results. The remote communication module is configured to securely transmit encapsulated data packets to a pre-registered remote expert terminal via an encrypted channel (e.g., HTTPS based on the TLS protocol). This remote expert terminal can be an application running on a PC, tablet, or dedicated mobile device. The remote communication module is also responsible for receiving diagnostic conclusions from the expert terminal, including a description of the root cause of the fault, recommended maintenance measures, and new rules or data for updating the knowledge base. Upon receiving the diagnostic conclusions from the expert terminal, the knowledge update module first verifies the completeness and consistency of the conclusions (e.g., checking whether the maintenance measures match the cause of the fault), and then adds new fault case information (including multimodal data, expert diagnostic conclusions, and maintenance measures) to the historical operation record database of the cloud-based intelligent platform. The knowledge update module is also configured to trigger an incremental training process for the AI ​​model, for example, fine-tuning the spatiotemporal neural network using new fault case data to improve its ability to identify unknown faults. In other optional implementations, the remote communication module can employ different communication protocols, such as gRPC based on the QUIC protocol, or asynchronous communication mechanisms based on message queues (e.g., RabbitMQ or Kafka). In addition, the data encapsulation module can support different data compression formats (e.g., LZ4 or Zstandard) to optimize transmission efficiency.

[0166] Through the above solution, this embodiment can efficiently integrate expert diagnostic knowledge into the intelligent maintenance system, effectively solve the diagnostic bottleneck of the system when facing unknown or rare faults, and significantly improve the system's self-learning ability and adaptability.

[0167] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A fault intelligent detection method for head-mounted devices, applied to fault diagnosis of high-speed periodic packaging equipment with multiple coordinated motion stations, characterized in that, include: Multimodal physical data of the packaging equipment during operation is collected using a head-mounted device. The multimodal physical data includes vibration signals, images of the equipment surface, and current fluctuation data. The surface image of the equipment is processed to extract wear features of mechanical components, and combined with the vibration signal and the current fluctuation data to generate a fault performance vector characterizing the physical state of the packaging equipment. The fault performance vector is compared with a preset physical parameter threshold to determine the initial fault type of the packaging equipment; Based on the preliminary fault type, the current fluctuation timing features and command execution deviation patterns in the fault performance vectors are grouped using a clustering algorithm to obtain a classified fault subset. Specifically, DBSCAN or K-Means clustering algorithms are used to analyze the fault performance vector sequence generated continuously within a preset time period, and vectors with similar change patterns are grouped into one category to obtain a classified fault subset. The command execution deviation pattern is obtained in real time from the programmable logic controller of the device and is used to reflect the difference between the actual motion trajectory of the actuator and the command trajectory. The categorized subset of faults is matched with data in the historical operation record database. When the matching degree exceeds a preset value, the root cause of the fault in the packaging equipment is determined. Based on the physical location of the root cause of the fault, one or more of the multiple tactile feedback actuators configured on the headband of the head-mounted device are driven to generate a tactile guidance signal pointing to the root cause of the fault.

2. The method according to claim 1, characterized in that, Before acquiring the multimodal physical data, the method further includes: The periodic synchronization signal of the packaging equipment is acquired, and the acquisition of the vibration signal, the surface image of the equipment, and the current fluctuation data is phase-triggered based on the periodic synchronization signal to generate a data frame synchronized with the movement cycle of the equipment.

3. The method according to claim 2, characterized in that, The generated fault performance vector characterizing the physical state of the packaging equipment includes: The data frames are input into a pre-defined spatiotemporal neural network based on the kinematic model of each station of the packaging equipment, so as to decouple and extract station-level fault features associated with the operating state of each independent station under a specific motion phase.

4. The method according to claim 3, characterized in that, Determining the root cause of the packaging equipment failure includes: Multiple workstation-level fault features are constructed into a time-series cause-effect graph according to the material flow and control flow sequence of the packaging equipment. The initial fault root cause workstation that causes concurrent faults is identified in the time-series cause-effect graph through a time-series cause-effect analysis algorithm.

5. The method according to claim 1, characterized in that, The process of processing the surface image of the device to extract wear features of the mechanical component includes: Semantic segmentation processing is performed on the surface image of the device to quantify the geometric size of the crack and the area of ​​the leakage region in the image; The surface image of the device is coordinate-registered with the thermal image acquired by the infrared sensor to extract the temperature gradient and thermal field non-uniformity features of the abnormal heat source.

6. The method according to claim 1, characterized in that, The generation of the fault manifestation vector includes: Wavelet packet transform is performed on the vibration signal and the current fluctuation data to extract the energy distribution characteristics of the vibration signal and the current fluctuation data in multiple frequency bands; The kurtosis, skewness, and information entropy of the vibration signal and the current fluctuation data are calculated as time-domain statistical features.

7. The method according to claim 6, characterized in that, The generation of the fault performance vector also includes: Obtain the current operating parameters of the packaging equipment; The energy distribution characteristics and the time-domain statistical characteristics are dynamically weighted according to the current operating condition parameters using an attention mechanism model to generate the fault performance vector.

8. The method according to claim 1, characterized in that, Before grouping the current fluctuation timing features and the instruction execution deviation patterns in the fault performance vector using the clustering algorithm, the method further includes: Principal component analysis (PCA) is used to reduce the dimensionality of the fault performance vector to extract the principal component features with the highest contribution rate for subsequent clustering calculations.

9. The method according to claim 1, characterized in that, After determining the root cause of the packaging equipment malfunction, the method further includes: Based on the root cause of the fault, a preset simulation model is invoked to predict the impact of the fault on the remaining service life of the packaging equipment without maintenance.

10. A fault intelligent detection system for head-mounted devices, characterized in that, include: A headband is arranged around the head-mounted device, and a plurality of tactile feedback actuators are arranged circumferentially on the headband. A data acquisition unit, configured on a head-mounted device, is used to acquire multimodal physical data during the operation of the packaging equipment. The multimodal physical data includes vibration signals, images of the equipment surface, and current fluctuation data. The feature generation unit is used to process the surface image of the equipment to extract wear features of mechanical parts, and combine the vibration signal with the current fluctuation data to generate a fault performance vector characterizing the physical state of the packaging equipment. A preliminary diagnostic unit is used to compare the fault manifestation vector with a preset physical parameter threshold to determine the preliminary fault type of the packaging equipment; The fault classification unit is used to group the current fluctuation timing features and command execution deviation patterns in the fault performance vectors based on the preliminary fault type using a clustering algorithm to obtain a classified fault subset. Specifically, the DBSCAN or K-Means clustering algorithm is used to analyze the fault performance vector sequence generated continuously within a preset time, and vectors with similar change patterns are grouped into one category to obtain a classified fault subset. The command execution deviation pattern is obtained in real time from the programmable logic controller of the device and is used to reflect the difference between the actual motion trajectory of the actuator and the command trajectory. The root cause determination unit is used to match the classified fault subset with the data in the historical operation record database. When the matching degree exceeds a preset value, the root cause of the fault of the packaging equipment and its physical location are determined. A haptic feedback control unit is configured to drive one or more of the plurality of haptic feedback actuators based on the physical location of the fault source to generate a haptic guidance signal directed to the fault source.

Images