Inspection method based on multi-modal fusion, six-degree-of-freedom mechanical arm, terminal and storage medium

Abstract

The invention belongs to the technical field of equipment detection, and particularly relates to a multi-modal fusion-based inspection method, a six-degree-of-freedom mechanical arm, a terminal and a storage medium, and the method comprises the steps: constructing a multi-modal sensing tail end module integrated with a visual sensor, an infrared thermal imaging sensor and a touch sensor on the six-degree-of-freedom mechanical arm; performing space-time alignment and fusion processing on the acquired image data, heat map data and tactile data; judging the operation state of the equipment based on the fused comprehensive feature data; the visual sensor provides equipment surface appearance and identification information, the infrared thermal imaging sensor captures temperature distribution and thermal gradient change, the touch sensor detects mechanical vibration, surface texture and contact pressure, a three-dimensional sensing system is formed, and the problem of one-sidedness of single-mode data is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of equipment detection, and particularly relates to an inspection method based on multi-modal fusion, a six-degree-of-freedom robotic arm, a terminal, and a storage medium. Background Art

[0002] In the field of modern industrial inspection, especially for the status monitoring and anomaly diagnosis of key facilities such as data centers, power cabinets, communication base stations, and rail transit control equipment, traditional inspection methods face many challenges. Traditional inspection methods often rely on single sensor technologies, such as fixed cameras, thermal imagers, or manual inspections, and these methods have obvious limitations.

[0003] In the prior art, industrial equipment inspection often uses fixed cameras, thermal imagers, or mobile inspection robots for monitoring, but there are the following deficiencies: single visual information is easily affected by light and occlusion, and it is difficult to judge the detailed state of the equipment; thermal imaging can be used to detect temperature rise, but the resolution is low, and it is difficult to accurately locate local heat sources; high-reliability detection means such as light touch testing of mechanical components, detection of structural looseness, and physical contact feedback cannot be realized; there is a lack of a cross-modal collaborative diagnosis mechanism, the accuracy of the judgment result is low, and there are misjudgments or missed judgments. Summary of the Invention

[0004] In view of the above deficiencies of the prior art, the present invention provides an inspection method based on multi-modal fusion, a six-degree-of-freedom robotic arm, a terminal, and a storage medium to solve the above technical problems.

[0005] In a first aspect, the present invention provides an inspection method based on multi-modal fusion, including the following steps: Construct a multi-modal perception end module integrating a visual sensor, an infrared thermal imaging sensor, and a tactile sensor on a six-degree-of-freedom robotic arm; Control the six-degree-of-freedom robotic arm to move to the equipment inspection point, and sequentially activate each modal perception sensor to collect image data, heat map data, and tactile data of the equipment at the inspection point; Perform spatio-temporal alignment and fusion processing on the collected image data, heat map data, and tactile data; Judge the operating state of the equipment based on the fused comprehensive feature data, and when it is determined that the equipment is in an abnormal state, generate an abnormal label. At the same time, generate an inspection report and upload it to the remote terminal.

[0006] A further improvement of this technical solution is that the method of controlling the six-degree-of-freedom robotic arm to move to the equipment inspection point includes: Collect all equipment information to be inspected, including equipment type, inspection content, last inspection time, and equipment location; Assign a priority to each inspection task according to the priority judgment factors; the priority judgment factors include the importance of the equipment, the failure history, and the maintenance cycle; Sort the inspection tasks according to the priority, and use the pre-stored D-H parameter path planning algorithm to formulate an inspection path in the order of the priority of the inspection tasks; Assign the formulated inspection path to the six-degree-of-freedom robotic arm to control the six-degree-of-freedom robotic arm to move to the equipment inspection point.

[0007] A further improvement of this technical solution is that the method for spatially aligning the collected image data, thermal map data, and tactile data includes: Define a three-dimensional space coordinate system and determine the positions of each equipment to be inspected in the three-dimensional space; Preprocess the collected image data and extract image features from the preprocessed image data; Normalize the thermal map data and extract thermal map features from the normalized thermal map data; Filter the tactile data and extract tactile features from the filtered tactile data; Use wavelet transform to perform multi-scale decomposition on the extracted image features, thermal map features, and tactile features to generate feature data of each modality at different scales; Use the Dirac function to align the feature data of each modality at different scales to the same spatial coordinate.

[0008] A further improvement of this technical solution is that the Dirac function is used to align the feature data of each modality at different scales to the same spatial coordinate, and its spatial alignment formula is: ; Among them, represents the multi-modal spatial fusion feature at the spatial coordinate (x, y, z); is the eigenvalue of the image data at the spatial position ([[]] i , j , k ) at scale s; is the eigenvalue of the thermal map data at the spatial position ([[]] i , j , k ) at scale s; is the eigenvalue of the tactile data at the spatial position ([[]] i , j , k ) at scale s; S is the total number of scales; n, m, and l are the dimensions of the image data, thermal map data, and tactile data at scale s, corresponding to the resolution or sampling points of the sensor data; is a Dirac function, which is used to align the feature data of different modalities to the same spatial coordinates (x, y, z); 、 and are the weights of the image data, heat map data, and tactile data at the spatial position ( i , j , k ) at scale s, respectively.

[0009] A further improvement of this technical solution is that the method for fusing the spatially and temporally aligned feature data includes: Fuse the spatially and temporally aligned feature data according to the feature fusion formula to obtain a multi-modal comprehensive feature ; The feature fusion formula is: ; Among them, 、 and are the feature values of the image data, heat map data, and tactile data at scale s, spatial position (x, y, z), and time t, respectively; 、 and are the weights of the image data, heat map data, and tactile data at time t at the spatial position (x, y, z) at scale s, respectively.

[0010] A further improvement of this technical solution is that based on the fused comprehensive feature data, the method for judging the operating state of the device includes: Input the multi-modal comprehensive feature obtained by fusing the spatially and temporally aligned feature data into a pre-constructed fault identification model, and output a fault diagnosis result; the fault diagnosis result is represented by the most likely fault category C, and the calculation formula for the most likely fault category C is: ; Among them, is the conditional probability of the fault category c corresponding to the multi-modal comprehensive feature .

[0011] A further improvement of this technical solution also includes: When the output result is a fault type, the six-degree-of-freedom robotic arm automatically adjusts its position on this detection device to confirm the fault location.

[0012] In a second aspect, the present invention provides a six-degree-of-freedom robotic arm that executes the multi-modal fusion-based inspection method described in any one of the above, including a six-degree-of-freedom robotic arm body and a multi-modal perception end module installed at the top of the six-degree-of-freedom robotic arm body, and a driving module is installed at the bottom of the six-degree-of-freedom robotic arm body; The multi-modal perception end module includes an end housing installed at the top of the six-degree-of-freedom robotic arm body, and a visual sensor and an infrared thermal imaging sensor installed inside the end housing. A detection window is provided on the end housing, a transparent protective cover is installed in the detection window, and a tactile sensor is installed on the end housing around the detection window; The drive module includes a drive housing installed at the bottom of the six-degree-of-freedom robotic arm body. A main control board and a power supply unit for powering the entire six-degree-of-freedom robotic arm are provided inside the drive housing. A road condition acquisition camera, several single-point laser detectors and several lidars are provided on the outer wall of the drive housing; A mobile chassis is provided at the bottom of the drive housing, and an anti-collision strip is provided on the periphery of the mobile chassis.

[0013] In a third aspect, the present invention provides a terminal, including: A processor and a memory, wherein, The memory is used to store a computer program, The processor is used to call and run the computer program from the memory, so that the terminal executes the method of the above-mentioned terminal.

[0014] In a fourth aspect, the present invention provides a computer storage medium, and instructions are stored in the computer-readable storage medium. When it runs on a computer, the computer is made to execute the methods described in the above aspects.

[0015] The beneficial effects of the present invention are as follows: Enhanced spatial dimension complementarity: The visual sensor provides information on the surface topography and markings of the device (such as indicator light status, physical deformation), the infrared thermal imaging sensor captures the temperature distribution and thermal gradient changes, and the tactile sensor detects mechanical vibrations, surface textures and contact pressures. The three form a three-dimensional perception system of "appearance-thermal characteristics-mechanical characteristics", solving the problem of one-sidedness of single-modal data.

[0016] Improved anti-interference ability: Under harsh working conditions such as strong light and occlusion, the tactile sensor can supplement the data in the visual blind area; the infrared thermal imaging is not affected by light and can penetrate the transparent protective cover to detect the temperature rise of internal components, realizing all-weather and full-scene coverage.

[0017] The fault location accuracy has increased significantly: By lightly touching and scanning the device surface with the tactile sensor and combining with the hot spot location of the infrared thermal imaging, the local overheating points and mechanical looseness positions can be accurately identified, reducing the fault location error.

[0018] In addition, the design principle of the present invention is reliable, the structure is simple, and it has a very wide application prospect. Description of the Drawings

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0020] Figure 1 It is a schematic flowchart of the method for an embodiment of the present invention.

[0021] Figure 2 It is a schematic structural diagram of a six-degree-of-freedom robotic arm for an embodiment of the present invention.

[0022] Figure 3 It is a schematic structural diagram of a terminal provided by an embodiment of the present invention.

[0023] 210 is the six-degree-of-freedom robotic arm body, 221 is the vision sensor, 222 is the infrared thermal imaging sensor, 223 is the tactile sensor, 231 is the driving housing, and 232 is the mobile chassis. Detailed implementation manners

[0024] To make the objectives, features, and advantages of the present invention more obvious and understandable, the following will clearly and completely describe the technical solutions in the present invention with reference to the accompanying drawings in the specific embodiments of the present invention. Obviously, the embodiments described below are only a part of the embodiments of the present invention, rather than all of them. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of this application.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs. The terms used in the description of the present invention in this specification are only for the purpose of describing specific embodiments, and are not intended to limit the present invention.

[0026] Figure 1 It is a schematic flowchart of a patrol inspection method based on multi-modal fusion provided by the present invention. Among them, Figure 1 The execution subject can be a six-degree-of-freedom robotic arm. According to different requirements, the order of the steps in this flowchart can be changed, and some can be omitted.

[0027] Such as Figure 1 shown, this method includes: Step 110, constructing a multi-modal perception end module integrating a vision sensor, an infrared thermal imaging sensor, and a tactile sensor on the six-degree-of-freedom robotic arm; Step 120: Control the six-degree-of-freedom robotic arm to move to the equipment inspection point, and sequentially activate each modal perception sensor to collect image data, thermal map data, and tactile data of the equipment at the inspection point. Step 130: Perform spatio-temporal alignment and fusion processing on the collected image data, thermal map data, and tactile data. Step 140: Judge the operating state of the equipment based on the fused comprehensive feature data. When it is determined that the equipment is in an abnormal state, generate an abnormal label. At the same time, generate an inspection report and upload it to the remote terminal.

[0028] For the convenience of understanding the present invention, the principle of the inspection method based on multi-modal fusion of the present invention will be further described below in combination with the process of inspecting equipment based on multi-modal in the embodiments.

[0029] S110: Construct a multi-modal perception end module integrating a vision sensor, an infrared thermal imaging sensor, and a tactile sensor on the six-degree-of-freedom robotic arm.

[0030] Install the integrated multi-modal perception end module on the flange (ISO 9409-1-50-40-M6 standard interface) at the end of the six-degree-of-freedom robotic arm. Specifically, the vision sensor uses an industrial camera of model Basler acA2440-75μm (global shutter CMOS, resolution 5472×3648 pixels, frame rate 75fps) to collect high-resolution visible light images of the equipment surface topography, indicator light status, label text (such as equipment number, operating parameters), etc.

[0031] The infrared thermal imaging sensor uses an infrared camera of model FLIR A6750sc LWIR (uncooled vanadium oxide detector, resolution 640×512 pixels, temperature measurement range -40°C to 1500°C, frame rate 30Hz) to detect the surface temperature distribution of the equipment and identify local overheating points (such as the temperature rising to 85°C due to poor contact of the power cabinet connector) and abnormal heat dissipation (such as the hot spot area of the communication base station power module exceeding the threshold).

[0032] The set tactile sensor array includes several piezoresistive thin film sensors of model Tekscan FlexiForce A201 (range 0~110N, resolution 0.1N, response time <1ms) to detect the surface vibration of the equipment (frequency range 5Hz~2kHz), contact pressure (such as detecting the bolt tightening force of the rail transit control cabinet), and surface texture (identifying mechanical damages such as rust and wear).

[0033] For example, during the inspection of servers in a data center, an industrial camera detects oil stains on the surface of a fan (visual feature), and an infrared camera simultaneously discovers abnormal temperature rise in the bearing area (82°C vs. normal value of 65°C). A joint diagnosis determines it as an early fault caused by insufficient lubrication. Or, during the inspection of a rail transit control cabinet, a tactile sensor detects abnormal vibration in the bolt tightening area (main frequency 120 Hz, RMS value 0.8 N), an industrial camera confirms the misalignment of the bolt locking marks, and the infrared camera shows local temperature rise (78°C). A comprehensive judgment determines that the bolt is loose.

[0034] Through cross-verification of multi-modal features, the present invention can eliminate misjudgments of a single modality (such as light reflection being mistaken for equipment oil leakage in visual inspection, and false hot spots caused by environmental thermal radiation interference in infrared detection), reducing the false alarm rate of faults.

[0035] Specifically, controlling a six-degree-of-freedom robotic arm to move to the equipment inspection point, the method includes: Collecting all equipment information to be inspected, including equipment type, inspection content, last inspection time, and equipment location; Assigning a priority to each inspection task according to priority judgment factors; the priority judgment factors include the importance of the equipment, fault history, and maintenance cycle; Sorting the inspection tasks according to the priority and formulating an inspection path in the order of the priority of the inspection tasks using a pre-stored D-H parameter path planning algorithm; Assigning the formulated inspection path to the six-degree-of-freedom robotic arm to control the six-degree-of-freedom robotic arm to move to the equipment inspection point.

[0036] Specifically, the system generates a task list according to a preset inspection task, including information such as equipment type, inspection area, priority, etc.; defining the position coordinates of each target equipment in the task list ; and loading the generated task list into the task scheduling platform of the six-degree-of-freedom robotic arm, and the platform plans the execution path of the robotic arm according to the task requirements.

[0037] Using the kinematic model of the pre-stored D-H parameter path planning algorithm for path planning, the kinematic equation is as follows: ; Wherein, is the rotation angle of the i-th joint of the six-degree-of-freedom robotic arm; is the twist angle of the i-th link of the six-degree-of-freedom robotic arm; is the length of the i-th link of the six-degree-of-freedom robotic arm; is the offset of the i-th link of the six-degree-of-freedom robotic arm.

[0038] Obtaining the current position of the end effector of the six-degree-of-freedom robotic arm through a position sensor installed on the six-degree-of-freedom robotic arm ; and calculate the error between the target position and the current position: ; ; .

[0039] Use a PID controller to adjust the movement of each joint, and the control formula is as follows: ; Where, is the target angle of the i-th joint of the six-degree-of-freedom robotic arm at time t; is the error between the target angle and the current angle; , and are the proportional, integral, and differential gains respectively. According to the actual requirements of the task, the motion trajectory of the robotic arm is adjusted in real time. For example, if it is detected that the position of the target device has changed, update the target position coordinates , and recalculate the path planning.

[0040] Through priority sorting and path optimization, the present invention ensures that critical devices are inspected in a timely manner, while reducing the moving distance and time of the robotic arm, and improving the overall inspection efficiency. Patrol resources are allocated according to the importance of the equipment and the risk of failure, ensuring that high-risk equipment receives the necessary attention, while not over-consuming resources on low-risk equipment. By analyzing the historical failure data and maintenance cycle of the equipment, the system can predict potential failures and intervene in advance, thereby reducing unexpected downtime and maintenance costs.

[0041] In addition, the method for spatially aligning the collected image data, thermal map data, and tactile data includes: Define a three-dimensional space coordinate system and determine the positions of each device to be inspected in the three-dimensional space; Preprocess the collected image data and extract image features from the preprocessed image data; Normalize the thermal map data and extract thermal map features from the normalized thermal map data; Filter the tactile data and extract tactile features from the filtered tactile data; Use wavelet transform to perform multi-scale decomposition on the extracted image features, thermal map features, and tactile features to generate feature data of each modality at different scales; Use the Dirac function to align the feature data of each modality at different scales to the same spatial coordinate.

[0042] Among them, the Dirac function is used to align the feature data of each modality at different scales to the same spatial coordinates, and its spatial alignment formula is: ; Among them, represents the multi-modal spatial fusion feature at the spatial coordinates (x, y, z); is the eigenvalue of the image data at the spatial position ( i , j , k ) at scale s; is the eigenvalue of the heat map data at the spatial position ( i , j , k ) at scale s; is the eigenvalue of the tactile data at the spatial position ( i , j , k ) at scale s; S is the total number of scales; n, m, and l are the dimensions of the image data, heat map data, and tactile data at scale s, corresponding to the resolution or sampling points of the sensor data; is the Dirac function, which is used to align the feature data of different modalities to the same spatial coordinates (x, y, z); , and are the weights of the image data, heat map data, and tactile data at the spatial position ( i , j , k ) at scale s, respectively.

[0043] In multi-modal data fusion, the data collected by different sensors (such as image sensors, thermal imaging sensors, and tactile sensors) usually have different spatial resolutions and coordinate systems. The purpose of spatial alignment is to align these data into a unified spatial coordinate system for subsequent processing. Through spatial alignment, data of different modalities can be compared and fused at the same three-dimensional spatial coordinates (x, y, z). The aligned multi-modal spatial fusion feature provides a unified data format, making the data of different modalities consistent in space. This provides a basis for subsequent temporal alignment and feature fusion. Through spatial alignment, the spatial deviation between data of different modalities can be reduced, ensuring that in subsequent processing, data of different modalities can accurately correspond to the same physical position.

[0044] Although the multi-modal spatial fusion feature It does not directly participate in the calculation in the subsequent time series alignment and feature fusion steps, but it is an important intermediate result. It records the state of different modality data after spatial alignment and provides a reference for subsequent processing. In practical applications, the spatially aligned data can be used to verify the accuracy of alignment or for further spatial analysis when needed. In applications such as fault diagnosis, the multi-modal spatial fusion features after spatial alignment can be used for comprehensive analysis. For example, by visualizing the multi-modal spatial fusion features , the distribution and relationship of different modality data in space can be intuitively observed. This comprehensive analysis helps to understand the mutual relationship between different modality data, thereby improving the accuracy and reliability of fault diagnosis.

[0045] In the feature fusion step, although the eigenvalue of each modality data is directly used , and , these eigenvalues are extracted on the basis of spatial alignment. The multi-modal spatial fusion features after spatial alignment ensure the spatial consistency of these eigenvalues, enabling the feature fusion to proceed smoothly.

[0046] In the present invention, by using the Dirac function for spatial alignment, the feature data of different modalities can be accurately mapped to the same spatial coordinates. This method eliminates the data deviation caused by differences in sensor positions or installation angles, ensuring the consistency and accuracy of the fused data. Accurate data fusion is the basis of fault diagnosis. Only when the data is accurate and error-free can the diagnostic result be trustworthy. In the process of fault diagnosis, reliability is crucial. By precisely spatially aligning image, heat map, and tactile data, it can be ensured that the information obtained from different sensors does not conflict with each other during analysis, thereby enhancing the reliability of fault diagnosis. For example, if the image data shows an abnormality on the surface of a certain component, and the heat map data and tactile data also support this finding, then this diagnostic result is more reliable.

[0047] The spatially aligned data can be more easily further analyzed and processed. Since all the data are in the same coordinate system, they can be directly compared and combined without additional transformation or adjustment. This not only saves time but also reduces the errors that may occur during the data processing. Using wavelet transform for multi-scale decomposition can capture the feature information at different scales. This multi-scale analysis helps to more comprehensively understand the state of the device because the features at different scales may correspond to different types of faults or anomalies. For example, the large-scale features may be related to the overall structural problems, while the small-scale features may indicate local defects. By assigning weights to the feature data at different scales, this method allows the system to adjust the contribution degrees of each modal data according to the actual application requirements. This adaptability enables the system to flexibly cope with different types of inspection tasks and environmental conditions, improving the practicality and flexibility of the system.

[0048] Furthermore, time alignment is performed on the collected image data, heat map data, and tactile data. Specifically, the Kalman filter algorithm is used to estimate and correct the temporal variations of each modal data to achieve time alignment. The specific steps include: Initialize the state variables , representing the state estimate of the system at the initial moment, which can be based on the initial sensor measurements or preset initial states; Initialize the covariance matrix of the state variables , representing the uncertainty of the initial state estimate; usually, the initial value of the covariance matrix can be set according to the accuracy of the sensor and the initial conditions of the system; Use the state transition matrix to predict the state variables at the previous moment to obtain the prior estimate of the state variables at the current moment : ; where is the prior estimate of the state variables at the previous moment; the purpose of this step is to predict the state at the current moment; Update the covariance matrix of the state variables , considering the influence of the process noise: ; where is the covariance matrix of the state variables at the previous moment; is the covariance matrix of the process noise, representing the uncertainty in the system dynamic model; Calculate the Kalman gain , used to adjust the influence of the observed value on the state estimate: ; where is the observation matrix, mapping the state variables to the observation space; is the covariance matrix of the observation noise, representing the uncertainty of the observed value; Use the observed value Correct the prior estimate of the state variable to obtain the posterior estimate : ; Adjust the estimate of the state variable by comparing the difference between the observed value and the predicted value; Update the covariance matrix of the state variable , considering the influence of the observed value: , reducing the uncertainty of the state estimate and improving the accuracy of the estimate.

[0049] In the present invention, the timestamp deviation (up to 16.7 ms) caused by the difference in sampling mechanisms of different sensors (such as the frame rate of an industrial camera being 75 fps, an infrared camera being 30 Hz, and a tactile sensor being 1 kHz) is compressed from ±5 ms to within ±0.3 ms through the prediction-correction mechanism of Kalman filtering, meeting the synchronization requirements of high-speed dynamic processes (such as sudden changes in equipment temperature rise and transient responses of mechanical vibrations) in industrial inspection. For example, when detecting a transient overload of the power supply module of a communication base station, image data (recording the flashing of indicator lights), thermal map data (capturing the temperature rise curve), and tactile data (detecting vibration shocks) can be precisely correlated with the moment of fault occurrence (such as the time difference between the temperature rise from 60°C to 85°C, the color change of the indicator light, and the vibration peak value <1 ms) after time alignment, avoiding misjudgments caused by time misalignment (such as misjudging the temperature rise lag as a heat dissipation fault).

[0050] Regarding the noise of an industrial camera in low-light environments (such as dark current noise and readout noise), the non-uniformity noise (NUC error) of an infrared camera, and the electromagnetic interference noise of a tactile sensor, Kalman filtering adaptively adjusts the observation weight through the observation noise covariance matrix R, increasing the image signal-to-noise ratio (SNR) by 12 dB, optimizing the temperature measurement accuracy of the thermal map from ±2°C to ±0.5°C, and reducing the repeatability error of tactile pressure measurement from ±0.5 N to ±0.05 N. By modeling dynamic uncertainties such as the movement jitter of the robotic arm (±0.1 mm positioning error) and the temperature drift of the sensor (0.02°C / h) through the process noise covariance matrix Q, in the inspection of power cabinets, the image blurring (the smear length is reduced from 5 mm to 0.8 mm) and tactile data distortion (the peak shift of the vibration spectrum is reduced from ±15 Hz to ±2 Hz) caused by the acceleration and deceleration of the robotic arm are successfully eliminated.

[0051] When the data of a certain sensor is lost or abnormal (such as the lens of the infrared camera being blocked), Kalman filtering uses the combined action of the state transition matrix F and the observation matrix H to perform state estimation compensation using other modal data (such as inferring the temperature distribution from visual data and inferring the thermal expansion effect from tactile data).

[0052] In addition, the methods for fusing the feature data after spatial alignment and time alignment include: Fuse the spatially and temporally aligned feature data according to the feature fusion formula to obtain multi-modal comprehensive features ; The feature fusion formula is: ; Among them, , and are the feature values of the image data, heat map data, and tactile data at scale s, spatial position (x, y, z), and time t respectively; , and are the weights of the image data, heat map data, and tactile data at spatial position (x, y, z) at time t under scale s respectively.

[0053] Through comprehensive processing of different modal data, the fusion process of the present invention can provide more comprehensive device status information, thereby significantly improving the accuracy and reliability of fault diagnosis. This multi-modal fusion method can reveal fault features that may not be detected by single-modal data, thereby reducing the situations of missed diagnosis and misdiagnosis. The fused multi-modal comprehensive features can more effectively represent the actual state of the device because it synthesizes various types of data such as images, heat maps, and tactile sensations. This rich data representation ability makes subsequent data analysis and pattern recognition more accurate. The multi-modal data fusion process enhances the robustness of the system to environmental changes and sensor noise by integrating information from different sensors. Even if the data of a certain modality is interfered or has errors, the fused data can still maintain high accuracy, thereby improving the adaptability and robustness of the system.

[0054] After that, based on the fused comprehensive feature data, judge the operating state of the device. The method includes: Fuse the spatially and temporally aligned feature data to obtain multi-modal comprehensive features Input it into a pre-constructed fault recognition model, and output a fault diagnosis result; the fault diagnosis result is represented by the most likely fault category C. The calculation formula for the most likely fault category C is: ; Among them, is the conditional probability of the fault category c corresponding to the multi-modal comprehensive feature .

[0055] By fusing the feature data after spatial alignment and temporal alignment into multi-modal comprehensive features and inputting them into the fault identification model, the present invention can significantly improve the accuracy of fault diagnosis. This is because the fused features contain more comprehensive information, can capture the subtle changes in the device state, and thus can more accurately identify faults. Using the fused comprehensive features for fault diagnosis can not only identify the current fault state, but also predict potential fault risks based on historical data and trend analysis to achieve predictive maintenance.

[0056] This inspection method further includes that when the output result is the fault type, the six-degree-of-freedom robotic arm automatically adjusts its position on this detection device to confirm the fault location, specifically: After the fault identification model outputs the most likely fault category C, analyze whether this fault type requires further position confirmation; If the fault location needs to be confirmed, corresponding robotic arm position adjustment instructions will be generated, and these instructions are based on the fault category C and the specific layout of the device; After receiving the instructions, the six-degree-of-freedom robotic arm automatically moves to the indicated position and uses the sensors on its multi-modal perception end module to further detect the fault area; The six-degree-of-freedom robotic arm collects data at the new position and sends these data back to the fault identification model for further analysis to confirm the exact fault location; Once the fault location is confirmed, the result is fed back to the remote terminal, and an inspection report containing the fault location information is automatically generated.

[0057] As Figure 2 shown, the present invention provides a six-degree-of-freedom robotic arm that executes the inspection method based on multi-modal fusion described in any one of the above, including a six-degree-of-freedom robotic arm body 210 and a multi-modal perception end module installed at the top of the six-degree-of-freedom robotic arm body 210. A drive module is installed at the bottom of the six-degree-of-freedom robotic arm body 210; the multi-modal perception end module includes an end housing installed at the top of the six-degree-of-freedom robotic arm body 210, and a visual sensor 221 and an infrared thermal imaging sensor 222 installed inside the end housing. A detection window is provided on the end housing, a transparent protective cover is installed inside the detection window, and a tactile sensor 223 is installed on the end housing around the detection window; the drive module includes a drive housing 231 installed at the bottom of the six-degree-of-freedom robotic arm body 210. A main control board and a power supply unit for powering the entire six-degree-of-freedom robotic arm are provided inside the drive housing 231. A road condition acquisition camera, several single-point laser detectors, and several lidars are provided on the outer wall of the drive housing 231; a mobile chassis 232 is provided at the bottom of the drive housing 231, and an anti-collision strip is provided on the periphery of the mobile chassis 232.

[0058] Among them, the road condition acquisition camera is used to capture the visual information of the environment where the robotic arm is located in real time, including the device position, obstacles, and changes in the surrounding environment. The lidar emits laser beams and receives the signals reflected from the environment to establish a three-dimensional model of the surrounding environment; it monitors the changes in the environment around the robotic arm in real time, such as moving objects or suddenly emerging obstacles, to achieve dynamic obstacle avoidance; it provides the environmental information required for path planning for the robotic arm so that it can select the optimal path to reach the inspection point. The single-point laser detector emits a laser beam and receives the reflected laser, which can accurately measure the distance between the robotic arm and the object; it detects obstacles on the traveling path of the robotic arm to prevent collisions and damage; it assists the robotic arm in self-positioning to ensure its position accuracy in space.

[0059] The end housing is fixed to the top of the six-degree-of-freedom robotic arm body through an adapter ring; the infrared thermal imaging sensor uses an infrared camera, and the vision sensor uses an industrial camera. The infrared camera and the industrial camera are arranged side by side in the end housing, and a common optical path design is achieved through an optical beam splitter to reduce space occupation. A 3×3 array tactile unit is embedded around the detection window of the end housing, and the surface is covered with a silicone buffer layer (Shore hardness 40A). The end housing is made of 6061-T6 aluminum alloy (surface anodized treatment), and the transparent protective cover is made of transparent germanium glass (transmittance > 92% for the 8-14μm infrared band). For the electrical interfaces inside the six-degree-of-freedom robotic arm, M12 aviation plugs are used, integrating gigabit Ethernet (industrial camera), CoaXPress (infrared camera), and I2C (tactile sensor) data buses.

[0060] The six-degree-of-freedom robotic arm in the present invention provides flexible operation capabilities, can quickly and accurately reach the designated inspection point, and significantly improves the inspection efficiency. The multi-modal perception end module integrates a vision sensor, an infrared thermal imaging sensor, and a tactile sensor, and can capture device status information from multiple dimensions, enhancing the inspection accuracy. The design of the robotic arm enables it to reach areas that are difficult to access by traditional methods, such as high places or narrow spaces, thus expanding the inspection scope.

[0061] Figure 3 FIG. 300 is a schematic structural diagram of a terminal 300 provided by an embodiment of the present invention, and the terminal 300 can be used to execute the inspection method based on multi-modal fusion provided by the embodiment of the present invention.

[0062] Among them, the terminal 300 may include: a processor 310, a memory 320, and a communication module 330. These components communicate through one or more buses. Those skilled in the art can understand that the structure of the server shown in the figure does not constitute a limitation to the present invention. It can be a bus structure, a star structure, and may also include more or fewer components than shown in the figure, or combine some components, or have different component arrangements.

[0063] Among them, the memory 320 can be used to store the execution instructions of the processor 310. The memory 320 can be implemented by any type of volatile or non-volatile storage terminal or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disc. When the execution instructions in the memory 320 are executed by the processor 310, the terminal 300 can execute some or all of the steps in the above method embodiments.

[0064] The processor 310 is the control center of the storage terminal, connecting various parts of the entire electronic terminal through various interfaces and lines. By running or executing the software programs and / or modules stored in the memory 320, and by calling the data stored in the memory, it executes various functions of the electronic terminal and / or processes data. The processor can be composed of an integrated circuit (IC). For example, it can be composed of a single packaged IC, or can be composed of multiple packaged ICs with the same or different functions connected together. For example, the processor 310 can include only a central processing unit (CPU). In the embodiment of the present invention, the CPU can be a single arithmetic core or can include multiple arithmetic cores.

[0065] The communication module 330 is used to establish a communication channel so that the storage terminal can communicate with other terminals. It receives user data sent by other terminals or sends user data to other terminals.

[0066] The present invention also provides a computer storage medium. Among them, the computer storage medium can store a program, and when the program is executed, it can include some or all of the steps in the various embodiments provided by the present invention. The storage medium can be a magnetic disk, an optical disc, a read-only memory (ROM), a random access memory (RAM), etc.

[0067] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solutions in the embodiments of the present invention, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk, or an optical disc, etc., various media that can store program codes, including several instructions to enable a computer terminal (which can be a personal computer, a server, or a second terminal, a network terminal, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0068] For the same and similar parts among the various embodiments in this specification, reference can be made to each other. In particular, for the terminal embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and for the relevant parts, reference can be made to the descriptions in the method embodiments.

[0069] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division. In actual implementation, there can be other division methods. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the systems or modules can be in electrical, mechanical, or other forms.

[0070] The modules described as separate components may or may not be physically separated. The components shown as modules may or may not be physical modules, that is, they can be located in one place, or they can be distributed to multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0071] In addition, in each embodiment of the present invention, the various functional modules can be integrated in a processing module, or each module can exist physically alone, or two or more modules can be integrated in one module.

[0072] Although the present invention has been described in detail by reference to the accompanying drawings and in conjunction with the preferred embodiments, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, those of ordinary skill in the art can make various equivalent modifications or substitutions to the embodiments of the present invention, and these modifications or substitutions should all be within the scope of the present invention / Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered by the protection scope of the present invention.

Claims

1. A patrol inspection method based on multimodal fusion, characterized in that It includes the following steps: Construct a multi-modal perception end module integrating a vision sensor, an infrared thermal imaging sensor, and a tactile sensor on a six-degree-of-freedom robotic arm; Control the six-degree-of-freedom robotic arm to move to the equipment inspection point, and sequentially activate each modal perception sensor to collect image data, thermal map data, and tactile data of the equipment at the inspection point; Perform spatio-temporal alignment and fusion processing on the collected image data, thermal map data, and tactile data; Judge the operating state of the equipment based on the fused comprehensive feature data, and when it is determined that the equipment is in an abnormal state, generate an abnormal label. At the same time, generate an inspection report and upload it to the remote terminal.

2. The inspection method based on multimodal fusion according to claim 1, wherein Control the six-degree-of-freedom robotic arm to move to the equipment inspection point, and its method includes: Collect all equipment information to be inspected, including equipment type, inspection content, last inspection time, and equipment location; Assign a priority to each inspection task according to the priority evaluation factors; the priority evaluation factors include the importance of the equipment, failure history, and maintenance cycle; Sort the inspection tasks according to the priority, and use the pre-stored D-H parameter path planning algorithm to formulate an inspection path according to the priority order of the inspection tasks; Assign the formulated inspection path to the six-degree-of-freedom robotic arm to control the six-degree-of-freedom robotic arm to move to the equipment inspection point.

3. The inspection method based on multi-modal fusion according to claim 1, characterized in that, The method for spatially aligning the collected image data, thermal map data, and tactile data includes: Define a three-dimensional space coordinate system and determine the positions of each equipment to be inspected in the three-dimensional space; Preprocess the collected image data and extract image features from the preprocessed image data; Normalize the thermal map data and extract thermal map features from the normalized thermal map data; Filter the tactile data and extract tactile features from the filtered tactile data; Use wavelet transform to perform multi-scale decomposition on the extracted image features, thermal map features, and tactile features to generate feature data of each modality at different scales; Use the Dirac function to align the feature data of each modality at different scales to the same spatial coordinate.

4. The inspection method based on multimodal fusion according to claim 3, wherein Use the Dirac function to align the feature data of each modality at different scales to the same spatial coordinate, and its spatial alignment formula is: ； Among them, represents the multi-modal spatial fusion feature at the spatial coordinates (x, y, z); is the eigenvalue at the spatial position ([ i , j , k ) of the image data at scale s; is the eigenvalue at the spatial position ([ i , j , k ) of the heat map data at scale s; is the eigenvalue at the spatial position ([ i , j , k ) of the tactile data at scale s; S is the total number of scales; n, m, and l are the dimensions of the image data, heat map data, and tactile data at scale s, corresponding to the resolution or sampling points of the sensor data; is the Dirac function, used to align the feature data of different modalities to the same spatial coordinates (x, y, z); , and are the weights at the spatial positions ([ i , j , k ) of the image data, heat map data, and tactile data at scale s, respectively.

5. The inspection method based on multimodal fusion according to claim 4, wherein The method for fusing the feature data after spatial alignment and temporal alignment includes: Fuse the spatially and temporally aligned feature data according to the feature fusion formula to obtain multi-modal comprehensive features ; The feature fusion formula is: ； Among them, , and are the eigenvalues of the image data, heat map data, and tactile data at scale s, spatial position (x, y, z), and time t, respectively; , and are the weights of the image data, heat map data, and tactile data at time t at spatial position (x, y, z) under scale s, respectively.

6. The inspection method based on multi-modal fusion according to claim 5, characterized in that Judge the operating state of the equipment based on the fused comprehensive feature data, and its method includes: The spatially and temporally aligned feature data will be fused to obtain multimodal comprehensive features Input it into a pre-constructed fault recognition model, and output the fault diagnosis result; the fault diagnosis result is represented by the most likely fault category C, and the calculation formula for the most likely fault category C is as follows: ； Among them, is the multi-modal comprehensive feature The conditional probability of the corresponding fault category c.

7. The inspection method based on multimodal fusion according to claim 2, wherein It also includes: When the output result is the fault type, the six-degree-of-freedom robotic arm automatically adjusts its position on this detection equipment to confirm the fault position.

8. A six-degree-of-freedom robotic arm, characterized in that, Implement the inspection method based on multi-modal fusion according to any one of claims 1-7, including a six-degree-of-freedom robotic arm body and a multi-modal perception end module installed at the top of the six-degree-of-freedom robotic arm body, and a drive module is installed at the bottom of the six-degree-of-freedom robotic arm body; The multi-modal perception end module includes an end housing installed at the top of the six-degree-of-freedom robotic arm body, a vision sensor and an infrared thermal imaging sensor installed inside the end housing, a detection window is provided on the end housing, a transparent protective cover is installed inside the detection window, and a tactile sensor is installed on the end housing around the detection window; The driving module includes a driving housing installed at the bottom end of the six-degree-of-freedom robotic arm body. A main control board and a power supply unit for powering the entire six-degree-of-freedom robotic arm are arranged inside the driving housing. A road condition acquisition camera, a plurality of single-point laser detectors and a plurality of lidars are arranged on the outer wall of the driving housing; a mobile chassis is arranged at the bottom end of the driving housing, and an anti-collision strip is arranged on the periphery of the mobile chassis.

9. A terminal, characterized in that, Comprising: A processor; A memory for storing the execution instructions of the processor; Wherein, the processor is configured to execute the method according to any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the program is executed by the processor, it implements the method according to any one of claims 1-7.

Images