Industrial equipment intelligent maintenance method and equipment based on multi-mode perception and medium

Abstract

The invention relates to the technical field of industrial automation, in particular to an industrial equipment intelligent maintenance method and equipment based on multi-modal sensing and a medium. The method comprises the following steps: acquiring multi-source sensing data of target industrial equipment through a multi-source sensor assembly; establishing an equipment virtual model based on the multi-source sensing data; performing dynamic time warping and three-dimensional space mapping on the data, and inputting a fault diagnosis algorithm to obtain a fault diagnosis type and confidence; and any maintenance mode is automatically, collaboratively and manually confirmed according to the confidence coefficient. During collaborative maintenance, mechanical arm tools are switched according to fault types, a touch glove sensor of an operator is calibrated, and fault points are highlighted in the virtual model to guide mechanical arm positioning; the mechanical arm is driven to execute preliminary operation, glove touch data are collected, the clamping force and the vibration frequency spectrum are fused to judge the bolt state, and then mechanical arm operation parameters are adjusted according to glove feedback data. According to the invention, precision, high efficiency and safety of industrial maintenance can be ensured.

Description

Technical Field

[0001] This invention relates to the field of industrial automation technology, and in particular to intelligent maintenance methods, equipment and media for industrial equipment based on multimodal perception. Background Technology

[0002] The stable operation of industrial equipment directly determines production efficiency and quality. However, the current maintenance model, which is mainly based on manual inspection or single sensor detection, has systemic bottlenecks: First, the perception dimension is limited, and over-reliance on vision makes it impossible to effectively capture internal hidden faults and key mechanical and acoustic features, resulting in a high rate of missed and misdiagnosed cases under complex working conditions. Second, multi-source data (such as visual, acoustic, and mechanical data) are difficult to integrate effectively due to inconsistent sampling frequencies and spatiotemporal coordinates, leading to information distortion and the inability to form a reliable joint diagnostic basis. Third, the decision-making mechanism is rigid and cannot be stratified according to the confidence level of fault diagnosis. This wastes manpower on simple faults and blindly automates complex faults, which can easily lead to safety risks.

[0003] These issues collectively hinder the intelligent upgrade of industrial maintenance towards precision, efficiency, and safety. Summary of the Invention

[0004] This invention provides an intelligent maintenance method, equipment, and medium for industrial equipment based on multimodal perception, in order to solve the problems of low accuracy, low efficiency, and low safety in existing industrial maintenance technologies.

[0005] This invention discloses an intelligent maintenance method for industrial equipment based on multimodal perception, applied to a multimodal perception device. The multimodal perception device includes a multi-source sensor assembly, a robotic arm, and a tactile glove. The multi-source sensor assembly includes: a vision module comprising a 3D-ToF camera; a tactile module comprising a pressure-sensitive sensing network disposed at the end of the robotic arm; and an acoustic module comprising a microphone array. The intelligent maintenance method for industrial equipment based on multimodal perception includes: acquiring multi-source sensing data of the target industrial equipment through the multi-source sensor assembly; the multi-source sensing data including at least one of the following: 3D point cloud and texture image data of the equipment surface, clamping force and vibration spectrum data, and frequency anomaly feature data; establishing a virtual model of the target industrial equipment based on the multi-source sensing data, ensuring that the virtual coordinate system of the virtual model is aligned with the physical coordinate system of the target industrial equipment; and inputting the multi-source sensing data after dynamic time warping and 3D spatial mapping into a fault diagnosis algorithm to obtain the fault diagnosis type and diagnosis confidence. The fault diagnosis algorithm is a convolutional neural network and... A hybrid deep network of gated loop units; selecting a matching maintenance mode based on the diagnostic confidence level, the maintenance mode including any one of automatic maintenance, collaborative maintenance, and manual confirmation; when the maintenance mode is collaborative maintenance, switching the operating tool used by the robotic arm according to the fault diagnosis type; when the operator wears the tactile gloves, starting the calibration program to complete the zeroing and accuracy verification of the sensor parameters on each tactile glove; highlighting the fault point in the virtual model, the fault point corresponding to the working position corresponding to the fault diagnosis type, so that the operator can operate the robotic arm to the working position based on the fault point; driving the robotic arm to perform the preliminary working action corresponding to the fault diagnosis type, and collecting multi-dimensional tactile data from the tactile gloves; determining and displaying the bolt status in real time based on the data fusion model and the real-time collected clamping force and vibration spectrum data; collecting multi-dimensional feedback data from the tactile gloves, adjusting the robotic arm operating parameters according to the multi-dimensional feedback data, the multi-dimensional data being input by the operator based on the bolt status and the multi-dimensional tactile data.

[0006] Optionally, the step of adjusting the robotic arm's operating parameters based on the multi-dimensional feedback data includes:

[0007] When the resistance of all five proximal phalanges bending sensors is greater than 80kΩ, the pressure detected by the fingertip pressure gauge is greater than 3N, and the data of the strain gauge on the back of the hand shows no fluctuation, the robotic arm immediately stops the current action and maintains the working posture. When the strain gauge on the back of the hand detects a periodic strain change of 1-2Hz, the resistance of the knuckle bending sensor is <30kΩ, and the fingertip pressure is <0.5N, the robotic arm returns to the initial standby position. When the resistance of the five proximal phalanges bending sensors is less than 28kΩ, the strain gauge data of the palm and back of the hand show no fluctuation, and the pressure on the fingertips and thumb side is less than 0.3N, the robotic arm resumes the working state before the pause.

[0008] Optionally, the method further includes: When the fingertip pressure is <1N, the palm strain is <100με, the thumb side pressure is <0.5N, and the torque is <5Nm, and the bolt is in a state that requires tightening, the mechanical arm is controlled to increase the torque. When the fingertip pressure is 2-5N, the palm strain is 200-400με, the thumb side pressure is 1-3N, and the torque is 5-20Nm, and the bolt is in a normal state, maintain the current operating parameters; When the fingertip pressure suddenly increases to >5N and the palm strain suddenly jumps to >500με, the tactile glove triggers vibration feedback and an LED alarm, and automatically sends a pause signal to the robotic arm.

[0009] Optionally, after the step of establishing a virtual model of the target industrial equipment based on the multi-source sensor data, the following steps are included: The virtual model maps perceived data in real time to simulate fault evolution; The actual maintenance data is uploaded to the cloud, and the fault feature library and diagnostic model are automatically updated to achieve closed-loop optimization.

[0010] Optionally, the convolutional neural network includes three convolutional layers and two max pooling layers, with convolutional kernel sizes of 3×3, 5×5, and 3×3 respectively, a stride of 1, and a pooling kernel size of 2×2. The gated loop unit includes two hidden layers, each with 256 units and a dropout ratio of 0.2.

[0011] Optionally, the step of selecting a matching maintenance mode based on the diagnostic confidence level includes: When the diagnostic confidence level is greater than 95%, the system enters automatic repair mode. When the diagnostic confidence level is between 60% and 95%, enter collaborative repair mode; When the diagnostic confidence level is less than 60%, the system enters manual confirmation mode.

[0012] The present invention also discloses a multimodal sensing device, which includes a multi-source sensor assembly, a robotic arm and a tactile glove, and a control system connected to the multi-source sensor assembly, the robotic arm and the tactile glove, the multimodal sensing device being used to implement the method described above.

[0013] The present invention also discloses a storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method described above.

[0014] The present invention also discloses an intelligent repair device, including a memory and a processor. The memory stores a computer program, which, when executed by the processor, causes the processor to perform the steps of the method described above.

[0015] The beneficial effects of the intelligent maintenance method, equipment, and medium for industrial equipment based on multimodal perception provided in this invention are as follows: The equipment status is characterized by joint analysis of multi-source data, including 3D point cloud / texture, clamping force and vibration spectrum, and frequency anomaly features. After dynamic time warping and 3D spatial mapping, the data is input into a CNN+GRU hybrid deep network for diagnosis. This reduces the risk of misjudgment and missed diagnosis caused by single sensor occlusion, noise, or operating condition fluctuations, thereby improving the stability and reliability of fault type identification. The system switches between automatic maintenance, collaborative maintenance, and manual confirmation based on diagnostic confidence levels. This improves automation efficiency in high-confidence scenarios and reduces the risk of misoperation by introducing manual confirmation in low-confidence scenarios, achieving a dynamic balance between efficiency and safety. When collaborative maintenance mode is selected, the robotic arm operating tools are automatically switched according to the fault diagnosis type, and the corresponding work location is highlighted in the virtual model. This reduces the time cost for operators to find fault points, select tools, and plan paths, and minimizes operational deviations caused by differences in experience. Attached Figure Description

[0016] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a flowchart illustrating an embodiment of the intelligent maintenance method for industrial equipment based on multimodal perception provided by the present invention. Figure 2 This is a schematic diagram of the structure of an embodiment of the multimodal sensing device provided by the present invention; Figure 3 This is a schematic diagram of the internal structure of an intelligent maintenance device in one embodiment of the present invention.

[0017] The labels for the attached figures are as follows: 10. Multimodal sensing device; 11. Multi-source sensor assembly; 111. Vision module; 112. Tactile module; 113. Acoustic module; 12. Robotic arm; 13. Tactile glove. Detailed Implementation

[0018] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0019] Please refer to the following: Figure 1 and Figure 2 , Figure 1 This is a flowchart illustrating an embodiment of the intelligent maintenance method for industrial equipment based on multimodal perception provided by the present invention. Figure 2 This is a schematic diagram of an embodiment of the multimodal sensing device provided by the present invention. The multimodal sensing device 10 includes a multi-source sensor assembly 11, a robotic arm 12, and a tactile glove 13. The multi-source sensor assembly 11 covers three core sensing modules: vision, touch, and acoustic. The vision module 111 uses a 3D-ToF camera with a resolution of 1920×1080@60fps, which can capture geometric defects such as cracks and deformations on the surface of the device at the 0.1mm level. The touch module 112 is a flexible pressure-sensitive sensor network deployed at the end of the robotic arm 12, which can collect clamping force in the range of 0.1-50N and vibration spectrum in the frequency band of 0-10kHz in real time to determine the tightness of bolts and the wear state of components. The acoustic module 113 is composed of an 8-channel microphone array, which collects the sound patterns of the device operation at a sampling rate of 44.1kHz and extracts characteristic frequency signals of faults such as bearing wear and gearbox noise. The robotic arm 12 is not only the motion carrier for performing maintenance operations, but also the mounting platform for the tactile sensor 112. Its end effector supports quick 10-second tool changes such as electric screwdrivers and torque wrenches, and can complete precise operations under the guidance of sensory data. The tactile glove 13, as a key terminal for two-way human-machine interaction, has built-in fingertip capacitive pressure pads, proximal knuckle flexion sensors, metal foil strain gauges on the palm and back of the hand, and a large-size pressure pad on the thumb side. It can collect pressure, strain, and posture data of the operator's hand in real time to achieve synchronous linkage between the robotic arm and human body movements. It can also receive tactile signals fed back by the system to assist the operator in perceiving the operating status of the equipment.

[0020] In one embodiment, the haptic glove is scientifically equipped with seven types of sensors based on the functional areas of the hand, fully covering the two core requirements of force perception and posture recognition. Each sensor has a clear division of labor and parameter matching, accurately capturing multi-dimensional data of hand operations.

[0021] Capacitive pressure pads are installed at the tips of all five fingers from the thumb to the little finger. This sensor uses a flexible electrode structure, changing the capacitance value by varying the electrode spacing d. The calculation formula is C = ε₀ε₀. A / d, whose capacitance change rate ΔC / C0 is approximately equal to the electrode spacing change rate Δd / d0, has a resolution of up to 0.1N. Its core function is to sense pressure changes when the fingertip contacts a tool or bolt.

[0022] Five proximal knuckles (near the metacarpophalangeal joint) from the thumb to the little finger are equipped with bending sensors. These sensors are based on the principle of conductive ink variable resistance. The resistance is 25kΩ in the flat state and increases to 100kΩ when bent to 90°. A voltage divider circuit composed of a 47kΩ pull-down resistor outputs a 0-5V analog signal, which is mainly responsible for capturing the degree of knuckle bending and thus identifying hand posture.

[0023] A metal foil strain gauge is installed in the middle of the palm. It works based on the resistance strain effect. The resistance change rate ΔR / R0 is equal to the product of the sensitivity coefficient K (K≈2) and the strain value ε. The range is ±500με and the accuracy is 0.1με. Its core function is to sense the strain change generated when the palm comes into contact with the tool.

[0024] Metal foil strain gauges are also installed on the back of the hand, with technical parameters exactly the same as those on the middle of the palm. They are attached to the tendons on the back of the hand and are specifically designed to capture strain fluctuations during hand swinging.

[0025] An additional capacitive pressure pad is installed on the thumb side (the area between the thumb and index finger), with the same technical parameters as the capacitive pressure pad at the fingertip. At the same time, the sensing area is expanded to 1cm×2cm to meet the force sensing needs when using the tool to push with the thumb and index finger, and to accurately capture changes in the pressure of the thumb.

[0026] Based on the collaborative acquisition and data fusion of the above-mentioned multi-sensor, when abnormal conditions such as bolt overtightening or component jamming are detected (such as fingertip pressure suddenly exceeding 5N or palm strain value suddenly exceeding 500με), the glove will immediately trigger a dual prompt of vibration feedback and LED alarm, and at the same time automatically send a "pause" control signal to the robotic arm to avoid damage to components due to operation exceeding limits.

[0027] The intelligent maintenance method for industrial equipment based on multimodal perception provided by this invention specifically includes the following steps: S101: Acquire multi-source sensing data of the target industrial equipment through a multi-source sensor assembly. The multi-source sensing data includes at least one of the following: three-dimensional point cloud and texture image data of the equipment surface, clamping force and vibration spectrum data, and frequency anomaly feature data.

[0028] In a specific implementation scenario, relying on the aforementioned multi-source sensor components, synchronous data acquisition is performed on the equipment's operating status and operational interaction process, obtaining multi-source sensor data covering geometric morphology, contact mechanics, and acoustic characteristics. Specifically, a 3D-ToF camera uses structured light technology to acquire 3D point cloud and texture image data of the equipment surface. The 3D point cloud data accurately records the 3D coordinate information of the equipment surface, with a measurement accuracy of up to 0.1mm, clearly capturing the location, size, and morphological characteristics of geometric defects such as micro-cracks and local deformations. The synchronously acquired texture image data supplements the appearance information of the equipment surface, such as color and material texture, and helps identify surface material anomalies caused by wear and corrosion. Together, they construct a complete visual state data of the equipment surface.

[0029] A flexible pressure-sensitive sensor network covering the end of the robotic arm collects clamping force and vibration spectrum data. It captures clamping force signals in real time during operation, with a range of 0.1-50N and a measurement accuracy of ±0.05N, accurately reflecting contact mechanical states such as bolt tightness and component clamping stability. Simultaneously, it collects vibration spectrum data in the 0-10kHz frequency band, capturing vibration anomalies caused by component wear, loose connections, and other issues, providing mechanical evidence for assessing the health status of moving components such as bearings and gears.

[0030] An 8-channel microphone array (sampling rate 44.1kHz) collects full-frequency acoustic signature signals during equipment operation. Preprocessing techniques such as wavelet transform are used to extract frequency anomaly data in characteristic frequency bands. The system focuses on the 3-5kHz characteristic frequency band corresponding to bearing wear and the 1-2kHz abrupt change signal corresponding to gearbox noise. It also captures sudden, irregular frequency fluctuations during equipment operation, enabling acoustic monitoring of hidden internal faults and overcoming the limitations of relying solely on visual or mechanical perception.

[0031] During actual data acquisition, the multi-source sensor components achieve synchronous data acquisition under the collaborative control of edge computing nodes, with the time synchronization error controlled within 10ms, ensuring the consistency of various data in the spatiotemporal dimension and laying the foundation for the accurate calculation of subsequent cross-modal feature fusion and intelligent diagnostic models.

[0032] S102: Establish a virtual model of the target industrial equipment based on multi-source sensor data, and ensure that the virtual coordinate system of the virtual model is aligned with the physical coordinate system of the target industrial equipment.

[0033] In a specific implementation scenario, a high-fidelity digital twin virtual model of the target industrial equipment is constructed based on the visual, tactile, and acoustic multi-source sensor data collected during the multimodal perception stage. During the modeling process, the virtual coordinate system of the virtual model is strictly aligned with the physical coordinate system of the target industrial equipment, laying a spatial benchmark for subsequent intelligent operation of the entire process.

[0034] Based on the 3D point cloud and texture image data of the equipment surface collected by the 3D-ToF camera, the overall structure, surface details and geometric dimensions of the equipment are accurately reproduced, including the position and shape of key structures such as cavities, bolt holes and transmission components, ensuring that the geometric accuracy is consistent with the physical equipment (error ≤ 0.1mm, matching the measurement accuracy of the vision sensor). Combined with the deployment position data of the tactile sensor network at the end of the robotic arm and the installation coordinate data of the microphone array, the physical positions of various sensor nodes are synchronously mapped to the virtual model, forming an integrated virtual carrier with sensor annotations. At the same time, the structural parameters and material properties in the equipment design drawings are incorporated to optimize the physical simulation characteristics of the model, so that the virtual model can synchronously reflect the mechanical response, vibration transmission and other characteristics of the equipment.

[0035] Coordinate system alignment is a core prerequisite for ensuring the consistency of interaction between the virtual model and the physical device, and requires a standardized calibration process to achieve precise bidirectional alignment. First, a global physical coordinate system is established using the geometric reference points of the physical device (such as the center of the device base or the endpoints of key axes) as a reference, serving as the benchmark for physical spatial positioning. Then, the origin of the virtual coordinate system of the virtual model is aligned with the origin of the physical coordinate system. Using the visual point cloud coordinate system as an intermediate bridge, physical spatial elements such as tactile sensor nodes, microphone arrays, and robotic arm motion trajectories are mapped one by one into the virtual coordinate system through coordinate transformation matrices (R rotation matrix, T translation matrix). Simultaneously, calibration tools (such as laser trackers) are used to collect the coordinate data of key feature points of the physical device and corresponding feature points of the virtual model. Alignment deviations are corrected through iterative optimization algorithms to ensure that the spatial position error between the two coordinate systems is <2mm, and that all coordinate axes are completely consistent.

[0036] The virtual model constructed and aligned in this way can not only accurately reproduce the spatial shape and sensor layout of physical equipment, but also serve as a unified carrier for multimodal data spatiotemporal alignment, fault simulation evolution, and robotic arm path planning. In the subsequent maintenance stage, the robotic arm operation path can be planned based on the aligned coordinate system, and the maintenance action effect can be simulated, ensuring the precise synchronization between virtual planning and physical operation, and maximizing the supporting role of digital twin technology in intelligent maintenance.

[0037] S103: After performing dynamic time warping and three-dimensional spatial mapping on multi-source sensor data, the data is input into the fault diagnosis algorithm to obtain the fault diagnosis type and diagnosis confidence. The fault diagnosis algorithm is a hybrid deep network of convolutional neural network and gated recurrent unit.

[0038] In a specific implementation scenario, after data acquisition, dynamic time warping (DTW) and 3D spatial mapping of multi-source sensor data are required to ensure that subsequent CNN and GRU models can accurately process device state phenomena corresponding to the same time and location. For time alignment, considering the different sampling frequencies of visual (60FPS, approximately 16.67ms per frame), tactile (up to 10kHz, sampled once every 0.1ms), and acoustic (44.1kHz, sampled once every 0.023ms), nanosecond-level timestamps are first provided using the PTP precise time protocol as a unified benchmark. Then, DTW (Dynamic Time Warping algorithm) is used to refine and align signal sequences with different time resolutions, ensuring that key event points such as peaks and abrupt changes in each modality correspond one-to-one, avoiding fusion misalignment. Spatial alignment uses the device's 3D digital twin model as a unified coordinate system and the visual point cloud coordinates as a benchmark. Through coordinate transformation matrices (R, T), the positions of the tactile sensor nodes (the end face of the robotic arm) and microphone array are mapped to the visual point cloud coordinate system, achieving spatial uniformity. The final output is the aligned structured multimodal dataset aligned_feat.

[0039] The preprocessed multi-source feature data is input into the fault diagnosis algorithm for deep learning to obtain the fault diagnosis type and diagnostic confidence level. The fault diagnosis algorithm employs a hybrid deep network constructed from convolutional neural networks (CNNs) and gated recurrent units (GRUs) to fully adapt to the characteristics of multimodal data: utilizing the convolution and pooling structures of CNNs, spatial features are accurately extracted from visual 3D point clouds, tactile force distribution, and other data, capturing spatial correlation information such as the geometric location and force distribution of the fault on the equipment surface; leveraging the gating mechanism and temporal modeling capabilities of GRUs, the temporal variation patterns of acoustic frequency anomalies and tactile vibration spectra are deeply mined to capture the dynamic features in the fault evolution process. After weighted fusion of the two types of features through an attention mechanism, the specific fault diagnosis type (supporting multi-fault coupled identification, such as gearbox seal aging + bearing wear) and diagnostic confidence level in the 0-100% range are output by a classifier, providing accurate and reliable diagnostic basis for subsequent hierarchical decision-making.

[0040] Specifically, the preprocessed multi-source feature data is input into the fault diagnosis algorithm for deep learning to obtain the fault diagnosis type and diagnostic confidence. The fault diagnosis algorithm uses a hybrid deep network constructed from convolutional neural networks (CNNs) and gated recurrent units (GRUs) to fully adapt to the characteristics of multimodal data. The core of the algorithm is to achieve accurate diagnosis through "modular feature extraction + attention fusion".

[0041] The spatial feature extraction using CNN aims to uncover spatial pattern features of equipment faults, encompassing information such as geometric shape, surface texture, pressure distribution, and vibration hotspots. Input data consists of visual data (3D point cloud and texture image data of the equipment surface) and tactile data (clamping force and vibration spectrum data) from the aligned_feat dataset. The processing employs a customized CNN structure: containing three convolutional layers (kernel sizes of 3×3, 5×5, and 3×3 respectively, with a stride of 1), used to extract local spatial patterns layer by layer; a 2×2 max-pooling layer is used between every two convolutional layers to achieve feature dimensionality reduction and computational cost while preserving significant fault-related features, and the ReLU activation function enhances the model's nonlinear fitting capability. The extracted features are specifically divided into two categories: visual features focus on crack length, orientation, coordinates of surface deformation areas, and color / material texture anomalies; tactile features focus on the spatial distribution of pressure values ​​on the contact surface and the spatial location of high-energy vibration points. The final output is a spatial feature vector (dimension can be set to 1×N), which encodes the spatial map of the equipment's geometric and mechanical anomalies.

[0042] Temporal feature extraction is performed based on GRU to capture the temporal evolution pattern of faults, i.e., the changes in internal audio and vibration over time. The input data is the acoustic data (frequency anomaly feature data) from `aligned_feat`, specifically the time-domain waveform and frequency-domain features of the voiceprint signal. A two-layer GRU structure is used, with 256 hidden units in each layer, and a 0.2 Dropout mechanism is introduced to suppress model overfitting. Through the gating characteristics of the GRU model, the long-term and short-term dependencies of the voiceprint signal can be effectively captured. It can not only identify instantaneous signal abrupt changes but also accurately capture dynamic features such as continuous frequency shifts and amplitude changes. For example, the gradual increase in energy in the 3–5kHz frequency band corresponding to bearing wear, and the critical transient peak in the 1–2kHz frequency band corresponding to gearbox noise. The final output is the `temporal_feat` temporal feature vector.

[0043] Spatial and temporal features are fused into a unified diagnostic basis, and the fault type and reliability are output simultaneously. The fusion process is implemented using an attention mechanism with weighting. First, the basic weights of the importance of each modality for fault diagnosis are preset (visual 0.4, tactile 0.3, acoustic 0.3). Then, the weight distribution is dynamically adjusted by the attention layer based on the abnormal correlation of the current data. For example, when the abnormal features of acoustic data are significant, the weight of the acoustic modality is temporarily increased to ensure that the fusion result fits the actual fault scenario. Finally, the fused feature vector fusion_feat is output.

[0044] Subsequently, the fused vector is processed by a softmax multi-class logistic regression classifier to output two core results: one is the fault type (supporting multi-label and multi-fault coupled identification, such as gearbox seal aging + bearing wear); the other is the diagnostic confidence in the range of 0~100%, which serves as the core basis for subsequent hierarchical decision-making.

[0045] S104: Select a matching maintenance mode based on diagnostic confidence level. The maintenance mode includes any one of automatic maintenance, collaborative maintenance, and manual verification.

[0046] In a specific implementation scenario, after obtaining the diagnosed fault type and its corresponding confidence level (0~100%), the appropriate maintenance mode is dynamically selected based on the diagnostic confidence level. Maintenance modes include three categories: automatic maintenance, collaborative maintenance, and manual confirmation. Each maintenance mode corresponds to a clearly defined confidence threshold range and differentiated execution logic.

[0047] When the diagnostic confidence level is greater than 95%, the system automatically enters automatic maintenance mode. A confidence level exceeding 95% indicates a clear fault characteristic and extremely high diagnostic reliability, belonging to a known and simple fault with a single scenario (such as a loose bolt, grinding minor surface cracks, or replacing standardized parts). Maintenance accuracy and safety can be guaranteed without manual intervention. In this mode, the robotic arm automatically matches the appropriate tool based on the fault type (e.g., a torque wrench for bolt tightening, a laser cleaning head for surface cleaning), completing tool switching via a 10-second quick-change interface. Simultaneously, it calls upon a cloud-based digital twin model to plan the optimal work path, combining force and vibration data from tactile sensors to automatically execute maintenance actions according to preset parameters (e.g., bolt tightening torque 5-20 Nm). During the process, operating parameters are monitored in real time. If abnormalities such as overtightening or jamming occur (e.g., torque exceeding 20 Nm), the robotic arm immediately triggers a pause mechanism to prevent component damage. The core value of this mode lies in maximizing maintenance efficiency and reducing manual intervention costs by 80%, making it suitable for routine fault handling of batch standardized equipment.

[0048] When the diagnostic confidence level is between 60% and 95%, the system automatically enters collaborative maintenance mode. This confidence level indicates that the fault can be preliminarily identified, but there may be multiple fault couplings (such as aging of the gearbox seal ring + bearing wear), unclear fault boundaries, or complex operation adaptations (such as precision component alignment, multi-step disassembly). Purely automatic maintenance poses operational risks, while completely manual operation is inefficient. Therefore, a human-machine collaborative and complementary execution logic is adopted. Once in this mode, the AR guidance system uses SLAM technology to overlay virtual maintenance steps, highlighted fault points, and key operating parameters (such as torque range and disassembly sequence) onto the real-world equipment view, providing operators with visual guidance. Operators wear tactile gloves integrated with multiple sensors, which collect hand posture and contact force data in real time. They control the robotic arm's movements through gesture commands (clenching a fist to pause, waving to reset, etc.) and simultaneously receive vibration and pressure signals from the gloves to help determine bolt tightness and component wear. The robotic arm performs repetitive, high-precision actions (such as constant torque tightening and precise positioning), allowing operators to focus on complex decision-making processes such as fault diagnosis and anomaly identification, achieving a balance between efficiency and accuracy.

[0049] When the diagnostic confidence level is less than 60%, the system automatically enters manual verification mode. A confidence level below 60% typically corresponds to two scenarios: first, the fault characteristics are ambiguous, indicating a rare, unknown, or conflicting multimodal data (e.g., visual detection of surface defects but abnormal acoustic frequencies); second, the fault involves core precision components of the equipment (e.g., core circuit boards, internal structures of sealed cavities), where the tolerance for repair actions is extremely low, requiring further verification based on human experience. Upon entering this mode, the core data from the multimodal fusion diagnostic phase (including aligned raw sensor data, spatial / temporal feature maps, and preliminary fault type inferences) and the fault simulation results from the digital twin model are simultaneously provided to maintenance personnel through a visual interface, offering data support for manual judgment. Simultaneously, the robotic arm is in standby mode, requiring operator authorization via a dedicated command to activate. During maintenance, the operator can autonomously control the robotic arm for assisted operations or perform entirely manual operation. The core purpose is to compensate for the algorithm's insufficient adaptation to unknown faults and complex scenarios through human experience, avoiding secondary damage to the equipment due to misoperation and ensuring maintenance reliability.

[0050] The switching of the three maintenance modes is automatically triggered based on the diagnostic confidence level. At the same time, it supports operators to manually switch modes in emergency scenarios, forming a flexible decision-making mechanism with automatic judgment as the main approach and manual intervention as a supplement. This not only adapts to the needs of different fault scenarios, but also provides diverse scenario samples for subsequent maintenance data feedback and model optimization.

[0051] S105: When the maintenance mode is collaborative maintenance, switch the operating tool used by the robotic arm according to the fault diagnosis type; when the operator wears tactile gloves, start the calibration program to complete the zeroing and accuracy verification of the sensor parameters on each tactile glove; highlight the fault point in the virtual model, and the fault point corresponds to the working position corresponding to the fault diagnosis type, so that the operator can operate the robotic arm to the working position based on the fault point.

[0052] In a specific implementation scenario, based on the fault diagnosis type output by multimodal fusion diagnostics, an automatic tool switching mechanism for the robotic arm is triggered. The robotic arm controller invokes tool adaptation logic to match the modular tool corresponding to the fault type. For example, for bolt loosening / tightening faults, it switches to a torque wrench; for component surface cleaning needs, it switches to a laser cleaning head; and for precision disassembly operations, it switches to an electric screwdriver. Relying on the quick-change interface at the robotic arm's end effector, tool replacement can be completed within 10 seconds. After switching, the tool center point calibration is automatically performed, and the tool coordinates are synchronously updated to the digital twin virtual model, ensuring that the tool's operating range accurately corresponds to the virtual coordinate system.

[0053] Upon detecting that the operator is wearing a multi-sensor integrated tactile glove, the glove calibration program is automatically initiated to zero out the parameters and verify the accuracy of each sensor, ensuring the reliability of human-machine interaction data. The calibration process is performed on the four types of sensors built into the glove: the fingertip capacitive pressure gauge is calibrated to a resolution of 0.1N to ensure accurate contact pressure detection; the resistance range of the proximal knuckle flexion sensor is calibrated (25kΩ-100kΩ corresponding to a flexion angle of 0°-90°), and the 0-5V analog signal output by the voltage divider circuit is calibrated simultaneously; the metal foil strain gauges on the palm and back of the hand are calibrated to a standard state with a range of ±500με and an accuracy of 0.1με to eliminate initial strain deviation; the thumb-side pressure gauge (1cm×2cm sensing area) is simultaneously aligned with the calibration parameters of the fingertip pressure gauge.

[0054] After tool switching and glove calibration are synchronized, the corresponding fault point is highlighted based on the aligned virtual model and the fault diagnosis type. Specifically, the work location corresponding to the fault diagnosis type (such as the gearbox bearing mounting position, bolt tightening area) is marked with a highlighted icon (red dynamic flashing effect) in the virtual model. At the same time, the virtual highlighted information is superimposed onto the equipment's real-world image (AR guidance) using SLAM technology. The marked content simultaneously includes the work range and key operating parameter prompts (such as bolt tightening torque 5-20Nm). The operator can intuitively view the fault location through the AR interface, and manually control the robotic arm movement based on the spatial coordinate guidance of the virtual model, or use haptic glove gesture commands (such as opening the hand to guide the robotic arm movement) to accurately position the robotic arm's end effector to the work location. The positioning error is ≤2mm, ensuring the targeted nature of the operation.

[0055] S106: Drive the robotic arm to perform preliminary operation actions corresponding to the fault diagnosis type, and collect multi-dimensional tactile data of the tactile glove; determine and display the bolt status in real time based on the data fusion model and the real-time collected clamping force and vibration spectrum data; collect multi-dimensional feedback data of the tactile glove, and adjust the robotic arm operation parameters according to the multi-dimensional feedback data. The multi-dimensional data is input by the operator based on the bolt status and multi-dimensional tactile data.

[0056] In a specific implementation scenario, after the robotic arm reaches the work position, it is driven to perform basic actions matching the fault diagnosis type. For example, for bearing wear faults, a component disassembly pre-processing action is performed; for bolt-related faults, a pre-tightening / pre-loosening action is performed. During the operation, two types of core data are collected simultaneously: first, the clamping force (range 0.1-50N, accuracy ±0.05N) and vibration spectrum data (0-10kHz frequency band) of the robotic arm end effector; second, the multi-dimensional tactile data of the tactile glove, including fingertip contact pressure, knuckle flexion posture, palm and back of hand strain values, and thumb lateral pushing pressure. All data is transmitted in real time to edge computing nodes and intelligent decision-making centers to provide a basis for status judgment and parameter adjustment.

[0057] Based on a preset data fusion model (pressure-strain-torque correlation mapping model), the system performs fusion calculations on the real-time collected clamping force, vibration spectrum data, and glove tactile data to determine the bolt status in real time and provide feedback to the operator via an AR interface. Specifically, a loose bolt status (torque < 5 Nm) corresponds to fingertip pressure < 1 N and palm strain < 100 με, and the AR indicates that the bolt status needs to be tightened; a normal status (torque 5-20 Nm) corresponds to fingertip pressure 2-5 N and palm strain 200-400 με, and the AR indicates that the bolt status is normal; an overtight / jammed status (torque > 20 Nm) corresponds to a sudden increase in fingertip pressure exceeding 5 N and a sudden jump in palm strain > 500 με, immediately triggering glove vibration feedback and an AR red alarm, while simultaneously pausing the robotic arm's movement.

[0058] The operator combines the bolt status displayed on the AR interface with multi-dimensional tactile data (such as vibration feedback and pressure sensing) transmitted through the haptic glove, inputting multi-dimensional feedback data through hand gestures. For example, if the operator senses that the bolt is too tight, they can trigger the robotic arm to pause by clenching their fist, adjust the hand pressure, and then resume operation by opening their hand. If abnormal vibration is detected, they can guide the robotic arm to fine-tune its position by waving their hand. The glove feedback data is analyzed in real time, and the robotic arm's operating parameters, including clamping force threshold, movement speed, and torque output range, are dynamically adjusted. This approach relies on the robotic arm to ensure operational accuracy while using human experience to compensate for algorithmic limitations in complex scenarios, ensuring maintenance safety and reliability.

[0059] The logic for parsing operator gesture commands and adjusting robotic arm parameters is as follows: When the resistance values ​​of all five proximal knuckle bending sensors are greater than 80kΩ, combined with the previously mentioned calibration parameters (25kΩ-100kΩ corresponds to bending angles of 0°-90°), it can be determined that the knuckle bending angles all exceed 70°, consistent with the bending state of the knuckles when making a fist. Simultaneously, the contact pressure detected by the fingertip capacitive pressure gauge is greater than 3N, corresponding to the pressure generated when the hand grips a tool tightly or presses naturally when making a fist. Furthermore, the data from the metal foil strain gauge on the back of the hand shows no significant fluctuations (strain value < 50με), eliminating interference from hand movements. Once these three conditions are met, it is determined that the current operator has input a pause command. With a response delay controlled within 200ms, the robotic arm immediately stops its current operation while maintaining its existing working posture (including tool position and clamping force). Furthermore, the AR interface can simultaneously display a pause notification, and the haptic glove stops its vibration feedback function to prevent damage to components or operational deviations caused by robotic arm displacement.

[0060] When the strain gauge on the back of the hand detects a periodic strain change of 1-2Hz, and the strain fluctuation amplitude is ±100με, it can be determined that the operator is waving their hand left or right. Simultaneously, the resistance values ​​of the five proximal knuckle flexion sensors are all less than 30kΩ, corresponding to a knuckle flexion angle of less than 20°, indicating that the knuckles are in an extended, unbent state. Furthermore, the pressure detected by the fingertip capacitive pressure gauge is less than 0.5N, ruling out interference from hand contact with the tool or pressing. When all three conditions are met, it is determined that the operator has input a reset command. The robotic arm will return to its initial standby position along a preset path, automatically avoiding obstacles during the process, and the end effector will return to its original position synchronously. Further, the AR guidance interface is reset, and the highlighted virtual fault point remains unchanged, facilitating subsequent repositioning operations. After the action is completed, a reset signal is fed back, and the robotic arm enters standby mode to await the next command.

[0061] When the resistance values ​​of all five proximal knuckle flexion sensors are less than 28kΩ, it can be determined that the knuckle flexion angle is less than 10°, indicating a fully extended, open hand. Simultaneously, the data from the metal foil strain gauges on the palm and back of the hand remain stable (strain value < 50με), with no strain fluctuations caused by pressing or swaying. Furthermore, the pressure detected by the capacitive pressure gauges at the fingertips and the thumb side pressure gauge is less than 0.3N, indicating that the hand is not in contact with any tool or applying any additional force, eliminating false triggering interference. Once these conditions are met, it is determined that the operator has input a recovery command, and the robotic arm immediately returns to its pre-pause operating state, continuing to perform the operation according to the pre-pause operating parameters (such as torque range and movement speed). Further, the AR interface indicates that the operation has resumed, and the glove simultaneously restores its vibration feedback function, ensuring a smooth maintenance process.

[0062] By combining and verifying data from multiple sensors in the tactile glove, false triggering caused by a single sensor failure can be effectively avoided. At the same time, it can accurately match the core operational requirements in human-machine collaborative work, realize flexible control of the robotic arm's movements, and balance maintenance safety and operational continuity.

[0063] In other implementation scenarios, the virtual model maintains real-time linkage with the target industrial equipment throughout the collaborative maintenance process, dynamically mapping multi-source sensing data and simulating fault evolution to provide predictive support for maintenance decisions. Based on the aligned virtual and physical coordinate systems described above, the model receives real-time data on the clamping force and vibration spectrum of the robotic arm end effector, as well as pressure and strain signals collected by the tactile glove. It synchronously updates the stress state, vibration distribution, and operating trajectory of the virtual equipment, achieving millisecond-level synchronous mapping between the physical scene and the virtual space. Simultaneously, combining fault diagnosis types and real-time sensing data, the model simulates fault evolution paths using simulation algorithms. For example, for bearing wear faults, based on the energy change trend in the acoustic 3-5kHz frequency band, it predicts the rate of wear aggravation over time and its impact on surrounding transmission components; for bolt loosening faults, it combines pressure-strain data to simulate the abnormal vibration diffusion that may be caused by the expansion of the loosening range, providing data support for operators to proactively avoid potential risks and optimize maintenance sequences.

[0064] After a maintenance task is completed, the entire process of actual maintenance data is automatically uploaded to a cloud database. Based on a preset algorithm, the fault feature library and diagnostic model are iteratively updated, constructing a closed-loop system of data acquisition, diagnosis and maintenance, and model optimization. The uploaded data covers multi-dimensional core information: First, basic fault data, including the finally confirmed fault type, diagnostic confidence, actual fault location and evolution characteristics, supplementing samples of rare faults and multi-fault coupled scenarios; second, maintenance process data, including robotic arm operation parameters (tool type, torque range, motion path), tactile glove feedback data, operator intervention records, and maintenance time; third, maintenance result data, including the equipment status after re-inspection, fault elimination effect, and component replacement information. Effective maintenance data is added to the fault feature library, enriching the visual, tactile, and acoustic multimodal feature dimensions and improving the feature maps of different fault scenarios. Subsequently, the diagnostic model iteration program is automatically triggered, fine-tuning the CNN+GRU hybrid model parameters based on the newly added data, optimizing the convolution kernel size, the number of GRU hidden layer units, and the attention weight allocation ratio, correcting the classifier threshold, and improving the diagnostic accuracy and generalization ability of the fault diagnosis algorithm for complex and rare faults.

[0065] Through this closed-loop optimization mechanism, actual operation data can be continuously accumulated, the error range of diagnostic confidence can be gradually reduced, the trigger ratio of automatic maintenance mode can be increased, and the cost of manual intervention can be reduced. At the same time, the model can be continuously adapted to the changes in fault characteristics caused by equipment aging and changes in operating conditions, ensuring that the performance of the entire intelligent maintenance system continues to improve with the usage scenarios, and realizing the advancement from passive maintenance to predictive maintenance.

[0066] As described above, this embodiment uses multi-source data, including 3D point cloud / texture, clamping force and vibration spectrum, and frequency anomaly features, to jointly characterize the equipment status. After dynamic time warping and 3D spatial mapping, the data is input into a CNN+GRU hybrid deep network for diagnosis. This reduces the risk of misjudgment and missed judgment caused by single sensor obstruction, noise, or operating condition fluctuations, thereby improving the stability and reliability of fault type identification. The system switches between automatic maintenance, collaborative maintenance, and manual confirmation based on diagnostic confidence levels. This improves automation efficiency in high-confidence scenarios and introduces manual confirmation to reduce the risk of misoperation in low-confidence scenarios, achieving a dynamic balance between efficiency and safety. When collaborative maintenance mode is selected, the robotic arm operating tools are automatically switched according to the fault diagnosis type, and the corresponding work location is highlighted in the virtual model. This reduces the time cost for operators to find fault points, select tools, and plan paths, and reduces operational deviations caused by experience differences.

[0067] Figure 3 A schematic diagram of the internal structure of an intelligent maintenance device in one embodiment is shown. Figure 3 As shown, the intelligent maintenance device includes a processor, a memory, and a network interface connected via a system bus. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and may also store a computer program. When executed by the processor, this computer program enables the processor to implement an intelligent maintenance method for industrial equipment based on multimodal perception. The internal memory may also store a computer program, which, when executed by the processor, enables the processor to execute an application of the intelligent maintenance method for industrial equipment based on multimodal perception. Those skilled in the art will understand that... Figure 2 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0068] In one embodiment, an intelligent repair device is provided, including a memory and a processor. The memory stores a computer program, which, when executed by the processor, causes the processor to perform the steps described above.

[0069] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, causes the processor to perform the steps described above.

[0070] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0071] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0072] It should be understood that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Those skilled in the art can modify the technical solutions described in the above embodiments, or make equivalent substitutions for some of the technical features; and all such modifications and substitutions should fall within the protection scope of the appended claims of the present invention.

Claims

1. A method for intelligent maintenance of industrial equipment based on multimodal perception, characterized in that, This is applied to a multimodal sensing device, which includes a multi-source sensor assembly, a robotic arm, and a tactile glove. The multi-source sensor assembly includes: The vision module includes a 3D-ToF camera; A tactile module, the tactile module including a pressure-sensitive sensing network disposed at the end of a robotic arm; An acoustic module, the acoustic module including a microphone array; The intelligent maintenance method for industrial equipment based on multimodal perception includes: The multi-source sensor assembly acquires multi-source sensing data of the target industrial equipment, including at least one of the following: three-dimensional point cloud and texture image data of the equipment surface, clamping force and vibration spectrum data, and frequency anomaly feature data. A virtual model of the target industrial equipment is established based on the multi-source sensor data, ensuring that the virtual coordinate system of the virtual model is aligned with the physical coordinate system of the target industrial equipment; After performing dynamic time warping and three-dimensional spatial mapping on the multi-source sensor data, the data is input into a fault diagnosis algorithm to obtain the fault diagnosis type and diagnosis confidence. The fault diagnosis algorithm is a hybrid deep network of convolutional neural network and gated recurrent unit. Based on the diagnostic confidence level, a matching maintenance mode is selected, which includes any one of automatic maintenance, collaborative maintenance, and manual verification. When the maintenance mode is collaborative maintenance, the operating tool used by the robotic arm is switched according to the fault diagnosis type. When the operator puts on the tactile gloves, the calibration procedure is initiated to zero out the sensor parameters and verify the accuracy of each tactile glove. The fault point is highlighted in the virtual model. The fault point corresponds to the work position corresponding to the fault diagnosis type, so that the operator can operate the robotic arm to the work position based on the fault point. The robotic arm is driven to perform preliminary operational actions corresponding to the fault diagnosis type, and multi-dimensional tactile data of the tactile glove is collected. The bolt status is determined and displayed in real time based on the data fusion model and the real-time collected clamping force and vibration spectrum data; multi-dimensional feedback data of the tactile glove is collected, and the operating parameters of the robotic arm are adjusted according to the multi-dimensional feedback data. The multi-dimensional data is input by the operator based on the bolt status and the multi-dimensional tactile data.

2. The intelligent maintenance method for industrial equipment based on multimodal perception according to claim 1, characterized in that, The step of adjusting the robotic arm's operating parameters based on the multi-dimensional feedback data includes: When the resistance of all five proximal phalanges bending sensors is greater than 80kΩ, the pressure detected by the fingertip pressure gauge is greater than 3N, and the data of the strain gauge on the back of the hand shows no fluctuation, the robotic arm immediately stops the current action and maintains the working posture. When the strain gauge on the back of the hand detects a periodic strain change of 1-2Hz, the resistance of the knuckle bending sensor is <30kΩ, and the fingertip pressure is <0.5N, the robotic arm returns to the initial standby position. When the resistance of the five proximal phalanges bending sensors is less than 28kΩ, the strain gauge data of the palm and back of the hand show no fluctuation, and the pressure on the fingertips and thumb side is less than 0.3N, the robotic arm resumes the working state before the pause.

3. The intelligent maintenance method for industrial equipment based on multimodal perception according to claim 2, characterized in that, The method further includes: When the fingertip pressure is <1N, the palm strain is <100με, the thumb side pressure is <0.5N, and the torque is <5Nm, and the bolt is in a state that requires tightening, the mechanical arm is controlled to increase the torque. When the fingertip pressure is 2-5N, the palm strain is 200-400με, the thumb side pressure is 1-3N, and the torque is 5-20Nm, and the bolt is in a normal state, maintain the current operating parameters; When the fingertip pressure suddenly increases to >5N and the palm strain suddenly jumps to >500με, the tactile glove triggers vibration feedback and an LED alarm, and automatically sends a pause signal to the robotic arm.

4. The intelligent maintenance method for industrial equipment based on multimodal perception according to claim 1, characterized in that, Following the step of establishing a virtual model of the target industrial equipment based on the multi-source sensor data, the following steps are included: The virtual model maps perceived data in real time to simulate fault evolution; The actual maintenance data is uploaded to the cloud, and the fault feature library and diagnostic model are automatically updated to achieve closed-loop optimization.

5. The intelligent maintenance method for industrial equipment based on multimodal perception according to claim 4, characterized in that, The method further includes: Based on the fusion of operating parameters using a long short-term memory network, the remaining lifespan of the target industrial equipment components is predicted, and a maintenance warning is generated 7 days in advance.

6. The intelligent maintenance method for industrial equipment based on multimodal perception according to claim 1, characterized in that, The convolutional neural network includes three convolutional layers and two max pooling layers. The convolutional kernel sizes are 3×3, 5×5, and 3×3, respectively, with a stride of 1, and the pooling kernel size is 2×2. The gated loop unit includes two hidden layers, each with 256 units and a dropout ratio of 0.

2.

7. The intelligent maintenance method for industrial equipment based on multimodal perception according to claim 1, characterized in that, The step of selecting a matching maintenance mode based on the diagnostic confidence level includes: When the diagnostic confidence level is greater than 95%, the system enters automatic repair mode. When the diagnostic confidence level is between 60% and 95%, enter collaborative repair mode; When the diagnostic confidence level is less than 60%, the system enters manual confirmation mode.

8. A multimodal sensing device, characterized in that, The multimodal sensing device includes a multi-source sensor assembly, a robotic arm, and a tactile glove, as well as a control system connected to the multi-source sensor assembly, the robotic arm, and the tactile glove. The multimodal sensing device is used to implement the method described in any one of claims 1-7.

9. A storage medium, characterized in that, The device stores a computer program that, when executed by a processor, causes the processor to perform the steps of the method as described in any one of claims 1 to 7.

10. An intelligent maintenance device, characterized in that, It includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the method as described in any one of claims 1 to 7.

Images