Collaborative diagnosis method based on multi-modal perception

Abstract

The application discloses a collaborative diagnosis method based on multi-modal perception, which collects visible light images, infrared thermal image data and / or acoustic imaging data through a multi-modal detection device, identifies a measured target and an auxiliary test instrument in a detection scene, and obtains instrument identification information and spatial position information; according to the instrument identification information, a communication protocol and a handshake parameter are matched, a wireless connection with the auxiliary test instrument is established, and timing data of the auxiliary test instrument is obtained; the timing data is time-synchronized and spatially correlated with a two-dimensional thermal image matrix and / or a two-dimensional acoustic image matrix of a corresponding region of the measured target, and multi-modal data after alignment is obtained; fusion diagnosis is performed based on the multi-modal data, and a fault type, a fault root cause and / or a diagnosis confidence are obtained. The application can realize automatic access of the auxiliary test instrument and collaborative analysis of multi-source detection data, and improve fault diagnosis efficiency and accuracy.

Description

Technical Field

[0001] This application relates to the field of industrial fault identification technology, and in particular to a collaborative diagnostic method based on multimodal perception. Background Technology

[0002] With the development of industrial intelligent inspection and IoT technology, multimodal inspection equipment such as infrared thermal imagers and acoustic imagers have been widely used for condition monitoring and fault diagnosis in electrical equipment, electronic circuits, and mechanical equipment. Existing inspection methods typically rely on a single inspection device to collect thermal, acoustic, or other condition data, combined with human experience or preset rules to determine the fault type. However, in complex industrial environments, equipment faults often manifest simultaneously as abnormal temperatures, abnormal currents, voltage fluctuations, changes in mechanical vibration, or abnormal environmental parameters, making accurate fault diagnosis difficult based solely on single thermal or acoustic data. Furthermore, on-site inspection personnel often need to operate multiple instruments simultaneously, such as thermal imagers, multimeters, oscilloscopes, and clamp meters, manually recording and comparing the results from different instruments. This approach is not only inefficient but also lacks a unified time reference and spatial correspondence between data from different instruments, making collaborative analysis of multi-source inspection data difficult and prone to misjudgment or omission.

[0003] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Application content

[0004] In view of at least one of the above technical problems, this application provides a collaborative diagnostic method based on multimodal perception to solve the problem that in existing industrial testing, the testing instruments are independent of each other, and it is difficult to achieve temporal and spatial alignment and fusion analysis of multi-source testing data, resulting in low efficiency and accuracy of fault diagnosis.

[0005] This application provides a collaborative diagnostic method based on multimodal perception, the method comprising: Detection data is collected using a multimodal detection device, and the detection data includes visible light images, infrared thermal images, and / or acoustic imaging data. Based on the detection data, identify the target to be tested and at least one auxiliary testing instrument in the detection scene, and obtain the instrument identification information and spatial location information of the auxiliary testing instrument. The instrument identification information includes the instrument category and / or instrument model. Based on the instrument identification information, the communication protocol and handshake parameters corresponding to the auxiliary test instrument are obtained from the preset protocol library, and a wireless handshake is performed with the auxiliary test instrument according to the communication protocol and handshake parameters. Acquire the timing data collected by the auxiliary testing instrument; Based on the spatial location information, the time-series data is synchronized and spatially correlated with the data of the corresponding region of the target in the two-dimensional thermal image matrix and / or two-dimensional acoustic image matrix acquired by the multimodal detection device to obtain aligned multimodal data; Based on the aligned multimodal data, the tested target is fused for diagnosis to obtain the fault diagnosis result.

[0006] This collaborative diagnostic method automatically identifies the target under test and auxiliary testing instruments through multimodal detection equipment, and establishes a wireless connection by matching the communication protocol according to the instrument identification information. It synchronizes the time-series data collected by the auxiliary testing instruments with thermal imaging data and / or acoustic imaging data in time and spatially correlates them, thereby realizing collaborative fusion diagnosis of multi-source detection data. This reduces manual configuration and data comparison, and improves the efficiency and accuracy of fault diagnosis.

[0007] In some possible implementations, the acquisition of detection data via a multimodal detection device includes: Within the same detection period, multiple consecutive visible light images of the detection scene are acquired, and infrared thermal imaging data and / or acoustic imaging data corresponding to the multiple visible light images are acquired simultaneously; wherein, the visible light images, the infrared thermal imaging data and / or acoustic imaging data are time-stamped to obtain a group of associated data at the same detection time.

[0008] In some possible implementations, the step of identifying the target to be tested and at least one auxiliary testing instrument in the detection scene based on the detection data, and obtaining the instrument identification information and spatial location information of the auxiliary testing instrument, includes: The detection data is input into a preset target recognition model to perform spatial feature extraction and temporal feature extraction, thereby obtaining the target category and target location of the target being tested, as well as the instrument category, instrument model, spatial location coordinates, and recognition confidence level of the auxiliary testing instrument.

[0009] In some possible implementations, obtaining the identification confidence level of the auxiliary testing instrument includes: Based on the detection data and / or wireless scanning information, determine the environmental noise evaluation value of the detection scene; The handshake trigger threshold is determined based on the environmental noise evaluation value; The recognition confidence scores corresponding to multiple consecutive frames of the detection data are obtained, and a weighted integral is performed to obtain the cumulative recognition confidence score. When the cumulative identification confidence level reaches the handshake trigger threshold, a wireless handshake with the auxiliary testing instrument is triggered.

[0010] In some possible implementations, determining the environmental noise evaluation value of the detection scene based on the detection data and / or wireless scanning information includes: The visible light noise value is determined based on the image contrast and / or edge sharpness of the visible light image; The infrared thermal noise value is determined based on the inter-frame noise variance of the infrared thermal image data; The radio frequency interference value is determined based on the radio frequency electromagnetic interference intensity in the wireless scanning information; The environmental noise evaluation value is calculated based on the visible light noise value, the infrared thermal noise value, and / or the radio frequency interference value; wherein, the larger the environmental noise evaluation value, the higher the handshake trigger threshold.

[0011] In some possible implementations, obtaining the communication protocol and handshake parameters corresponding to the auxiliary testing instrument from a pre-set protocol library based on the instrument identification information includes: Using the instrument category and / or instrument model as an index, the preferred communication protocol, handshake parameters, and data format corresponding to the auxiliary test instrument can be retrieved from the preset protocol library; When the preferred communication protocol fails to perform a wireless handshake, the system switches to an alternative communication protocol based on the preset priority of the communication protocols, and then performs a wireless handshake again based on the alternative communication protocol.

[0012] In some possible implementations, when multiple auxiliary testing instruments are identified in the testing scenario, a wireless handshake is performed with the multiple auxiliary testing instruments to establish a wireless communication link, including: Assign a corresponding visual instance identifier to each identified auxiliary testing instrument; Send handshake requests to the plurality of auxiliary test instruments and obtain the radio frequency signal strength of the response signals returned by each auxiliary test instrument; The visual recognition objects of each auxiliary testing instrument are determined based on the visual instance identifier and the spatial location information, and each visual recognition object is matched and bound to the corresponding communication object based on the radio frequency signal strength obtained during the wireless handshake process.

[0013] In some possible implementations, the step of performing a wireless handshake with the auxiliary test instrument according to the communication protocol and handshake parameters includes: Send a handshake request to the auxiliary testing instrument and determine whether a handshake response is received from the auxiliary testing instrument within a preset handshake timeout period; If the handshake response is not received within the preset handshake timeout period, the backoff waiting time is determined by the binary exponential backoff algorithm, and the handshake request is resent after the backoff waiting time. If the handshake response is received, the auxiliary test instrument is authenticated, and the wireless communication link is established after the authentication is successful. If the wireless communication link is not established after the number of handshake retries reaches the preset maximum number of retries, the corresponding auxiliary test instrument will be marked as unconnectable, and subsequent fusion diagnosis will continue based on the auxiliary test instrument that has successfully established a wireless communication link.

[0014] In some possible implementations, the step of synchronizing and spatially associating the temporal data with the data of the corresponding region of the target in the two-dimensional thermal image matrix and / or two-dimensional acoustic image matrix acquired by the multimodal detection device based on the spatial location information includes: Using the clock of the multimodal detection device as a reference, timestamp synchronization information is sent to the auxiliary testing instrument; Based on the data timestamps fed back by the auxiliary testing instrument, the time offset of the time series data relative to the two-dimensional thermal imaging matrix and / or the two-dimensional acoustic imaging matrix is ​​determined; Based on the time offset, time compensation is performed on the time-series data; Based on the spatial position information of the auxiliary testing instrument, the compensated time-series data is associated with the area corresponding to the target in the two-dimensional thermal imaging matrix and / or the two-dimensional acoustic imaging matrix; Data packets that do not fall within the preset sliding time window are marked as lost, and the lost data is filled using nearest neighbor interpolation or linear interpolation; If the number of consecutive packet losses reaches a preset threshold, a new wireless handshake with the corresponding auxiliary testing instrument will be initiated.

[0015] In some possible implementations, the step of performing fusion diagnosis on the target under test based on the aligned multimodal data to obtain a fault diagnosis result includes: Spatiotemporally aligned multimodal data is input into a preset fusion diagnostic model, and the two-dimensional thermal image matrix and / or two-dimensional acoustic image matrix in the spatiotemporally aligned multimodal data are feature-encoded to obtain thermal feature vectors and / or acoustic feature vectors. Feature encoding is performed on the temporal data in the spatiotemporally aligned multimodal data to obtain auxiliary test feature vectors; The thermal feature vector and / or acoustic feature vector, and the auxiliary test feature vector are mapped to the isomorphic latent space and the features are integrated to obtain multimodal feature data; Based on the multimodal feature data, fault diagnosis is performed on the target under test to obtain the fault diagnosis result; the fault diagnosis result includes fault type, fault root cause and / or diagnosis confidence.

[0016] The present application will be further described below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of the collaborative diagnostic method provided in the embodiments of this application. Detailed Implementation

[0019] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the specific embodiments of this application are described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0020] This embodiment provides a collaborative diagnostic method based on multimodal perception. The method uses an infrared thermal imager, an acoustic imager, or a detection device with visible light imaging, infrared imaging, and acoustic imaging capabilities as the multimodal detection device. The multimodal detection device automatically identifies auxiliary testing instruments in the detection scene, establishes a wireless communication relationship with the auxiliary testing instruments, and correlates the electrical, mechanical, or environmental parameters collected by the auxiliary testing instruments with thermal and / or acoustic data to perform collaborative diagnosis of the target under test.

[0021] To enable those skilled in the art to better understand the present invention, some of the technical terms involved in the present invention will be explained below.

[0022] Multimodal inspection equipment refers to inspection equipment that has at least one of the following capabilities: visible light image acquisition capability, infrared thermal image data acquisition capability, and acoustic imaging data acquisition capability. Multimodal inspection equipment can be an infrared thermal imager with a visible light camera, an acoustic imager, a thermoacoustic composite inspection device, a portable industrial inspection terminal, or a head-mounted industrial inspection terminal.

[0023] The target under test refers to electrical equipment, electronic components, mechanical equipment or their local areas that require operational status monitoring or fault diagnosis, such as circuit boards, chips, switch contacts, terminals, distribution cabinets, bearings, compressors, fans and refrigeration systems, etc.

[0024] Auxiliary testing instruments refer to testing instruments capable of detecting the target under test or its operating environment and outputting test data wirelessly. Auxiliary testing instruments may include digital multimeters, digital oscilloscopes, clamp meters, vibration meters, thermometers, hygrometers, anemometers, or other instruments capable of acquiring electrical, mechanical, or environmental parameters. The timing data uploaded by auxiliary testing instruments can be voltage values, current values, voltage waveforms, current waveforms, mechanical vibration signals, temperature and humidity data, or wind speed data, etc.

[0025] Time series data refers to data sequences collected by auxiliary testing instruments in chronological order, such as voltage sequences, current sequences, voltage waveforms, current waveforms, vibration signals, temperature sequences, humidity sequences, or wind speed sequences.

[0026] Visual instance identifiers refer to the identifiers assigned by the target recognition model to each auxiliary testing instrument in the detection scene, used to distinguish different auxiliary testing instruments. Even if multiple auxiliary testing instruments are of the same instrument category or model, they will each have different visual instance identifiers.

[0027] A communication object refers to a wireless device detected by a multimodal detection device during wireless scanning or handshake that can return a wireless response signal. Visual recognition objects and communication objects belong to the visual physical space and the radio frequency communication space, respectively. This invention establishes a correspondence between the two through information such as instrument model, spatial location information, and radio frequency signal strength.

[0028] like Figure 1 As shown, this embodiment provides a collaborative diagnostic method based on multimodal perception, the method including steps S100 to S600.

[0029] S100, collect detection data through a multimodal detection device, the detection data including visible light images, infrared thermal images and / or acoustic imaging data.

[0030] In S100, visible light images are used to identify the appearance features of the target under test and auxiliary testing instruments; infrared thermal imaging data are used to characterize the two-dimensional temperature distribution of the target under test; and acoustic imaging data are used to characterize the spatial distribution of sound sources in or around the target under test.

[0031] Infrared thermal imaging data can be output as a two-dimensional temperature matrix from an infrared sensor array. Each element in the two-dimensional temperature matrix corresponds to the temperature value at a spatial location in the detected scene.

[0032] Acoustic imaging data can be acquired by a microphone array or an ultrasonic sensor array, and a two-dimensional acoustic image matrix is ​​formed based on the phase difference and amplitude difference between the signals acquired by each acoustic sensor. Each matrix element in the two-dimensional acoustic image matrix can correspond to the sound pressure level, sound intensity, or sound source probability at the corresponding spatial location.

[0033] In some embodiments, the acquisition of detection data via a multimodal detection device includes: Within the same detection period, multiple consecutive visible light images of the detection scene are acquired, and infrared thermal imaging data and / or acoustic imaging data corresponding to the multiple visible light images are acquired simultaneously; wherein, the visible light images, the infrared thermal imaging data and / or acoustic imaging data are time-stamped to obtain a group of associated data at the same detection time.

[0034] Understandably, multimodal detection equipment uses its own system clock as a unified time reference, adding a timestamp to each frame of visible light image, infrared thermal image data, or acoustic imaging data it acquires. Visible light images, infrared thermal image data, and / or acoustic imaging data acquired at the same detection time or within the same preset time range are grouped into a correlated data set.

[0035] For example, the infrared thermal imaging acquisition frequency of the multimodal detection equipment is 30 frames / second, the visible light image acquisition frequency is 30 frames / second, and the acoustic imaging frequency can be determined according to the operating frequency of the acoustic acquisition array. Data acquired at the same time or with a time difference less than a preset time difference are assigned the same association group identifier.

[0036] In this way, the appearance, temperature and / or acoustic state of the target under test at the same time can be determined in subsequent processing steps.

[0037] S200, based on the detection data, identify the target to be tested and at least one auxiliary testing instrument in the detection scene, and obtain the instrument identification information and spatial location information of the auxiliary testing instrument. The instrument identification information includes the instrument category and / or instrument model.

[0038] Specifically, the detection data is input into a preset target recognition model to perform spatial feature extraction and temporal feature extraction, thereby obtaining the target category and target location of the target being tested, as well as the instrument category, instrument model, spatial location coordinates, and recognition confidence level of the auxiliary testing instrument.

[0039] The target recognition model employs a multi-task target detection model capable of simultaneously recognizing both the target under test and the auxiliary testing instrument. This target recognition model includes a shared feature extraction network, a target under test recognition output branch, and an auxiliary testing instrument recognition output branch.

[0040] The shared feature extraction network performs spatial and temporal feature extraction on the input detection data.

[0041] Spatial features can include the contour, edges, texture, button layout, display area shape, interface location, and local structural features of the target or auxiliary testing instrument. Shared feature extraction networks can employ depthwise separable convolutions to reduce the computational burden of the target recognition model on multimodal detection devices.

[0042] Temporal features can include changes in the position of the same target across multiple frames of detection data, the degree of overlap of detection boxes, changes in class probability, and changes in local features. Temporal features can be used to suppress flickering of recognition results caused by occlusion, motion blur, or instantaneous changes in illumination.

[0043] The target recognition output branch outputs the target category and target location based on the extracted features. The target location can be represented by a target detection box, target contour, target center coordinates, or region of interest.

[0044] The auxiliary testing instrument identification output branch outputs the instrument category, instrument model, spatial location coordinates, and identification confidence level of the auxiliary testing instrument based on the extracted features.

[0045] In this embodiment, the instrument model can be identified based on features such as the shape of the auxiliary testing instrument's casing, the position of the display screen, the layout of knobs or buttons, the arrangement of interfaces, and external markings. Since different instrument models may use different communication protocols, handshake parameters, or data formats, the instrument model can serve as an index for subsequent protocol queries.

[0046] In some embodiments, the training data for the target recognition model may include a self-built dataset, a publicly available instrument image dataset, and industrial field acquisition data.

[0047] Training data can include sample images under normal lighting, strong backlighting, low light, dust occlusion, partial occlusion, motion blur, surface oil stains, and at different shooting distances and angles. In some embodiments, the number of training samples for each instrument category or model can reach more than 10,000 images. Data augmentation can also be performed through methods such as rotation, scaling, cropping, brightness adjustment, background replacement, and generative adversarial networks.

[0048] In some embodiments, the target recognition model outputs an auxiliary testing instrument's recognition confidence score based on the detection data at the current moment. The formula for calculating the recognition confidence score is: , In the formula, Let N be the recognition confidence level at the current time t, and N be the number of features involved in the spatial feature confidence calculation. The weight corresponding to the j-th spatial feature is... This represents the probability of the instrument category or instrument model output by the target recognition model at the current moment. To extract functionals from spatial features based on depthwise separable convolution, This is the intersection-union ratio (IoU) of the detection frames of the same auxiliary testing instrument in the current frame and the previous frame. This is a temporal Markov smoothing operator used to suppress inter-frame flicker. To assist in testing the features of the region of interest corresponding to the instrument, For cross-modal attention mapping, To evaluate the environmental noise level of the detection scene, For environmental noise penalties, It is a non-linear activation function.

[0049] In some embodiments, after obtaining the identification confidence level of the auxiliary testing instrument, the environmental noise evaluation value of the testing scene is determined based on the detection data and / or wireless scanning information.

[0050] Environmental noise assessment values ​​can be determined based on at least one of visible light noise values, infrared thermal noise values, and radio frequency interference values. Environmental noise assessment values ​​can be expressed as: In the formula, E v E represents the visible light noise value. ir E represents the infrared thermal noise value. rf This represents the radio frequency interference value. , , These are the weights for the corresponding noise values.

[0051] When using three noise values ​​simultaneously, the following can be satisfied: =1. When radio frequency scanning information is not obtained, it can be made... =0, and normalize the remaining weights.

[0052] The visible light noise value is determined based on the image contrast and / or edge sharpness of the visible light image. The visible light noise value can be expressed as: In the formula, The normalized image contrast. The normalized edge sharpness is represented by a1 and a2, which indicate the corresponding weights. It's understandable that lower image contrast and edge sharpness result in higher visible light noise values.

[0053] The infrared thermal noise value is determined based on the inter-frame noise variance of the infrared thermal image data. For example, for two consecutive frames of infrared thermal image data, the inter-frame interpolation is calculated: , Infrared noise value can be expressed as: In the formula, Ω represents the pixel region involved in the calculation. This represents the number of pixels in that pixel region. This represents the mean of the inter-frame differences.

[0054] The radio frequency interference (RF) value is determined based on the RF electromagnetic interference intensity in the wireless scanning information. The RF interference value can be determined based on the noise power, channel occupancy rate, or received signal noise floor obtained by the multimodal detection device scanning within a preset wireless frequency band. The stronger the RF electromagnetic interference, the greater the RF interference value.

[0055] In some embodiments, to avoid direct triggering of the wireless handshake by misidentification in a single frame, the dynamic identification confidence scores of multiple consecutive frames are weighted and integrated to obtain the cumulative identification confidence score. The cumulative identification confidence score can be expressed as: In the formula, K represents the cumulative identification confidence at the current moment, and K is the number of consecutive frames participating in the weighted integral. The historical frame attenuation coefficient satisfies 0. 1.

[0056] Since the weight of the current frame is 1, the weight of historical frames further away from the current frame is lower. Therefore, recent detection results have a greater impact on the cumulative recognition confidence.

[0057] In some embodiments, the handshake trigger threshold can be determined based on an environmental noise assessment value. The handshake trigger threshold can be: In the formula, The threshold for triggering a handshake. Based on the trigger threshold, This is the environmental noise adjustment factor. Therefore, the higher the environmental noise evaluation value, the higher the handshake trigger threshold.

[0058] When satisfied ≥ At this time, a wireless handshake is triggered between the multimodal detection equipment and the auxiliary testing instruments.

[0059] S300: Based on the instrument identification information, obtain the communication protocol and handshake parameters corresponding to the auxiliary test instrument from the preset protocol library, and perform a wireless handshake with the auxiliary test instrument according to the communication protocol and handshake parameters; After identifying the instrument category and / or instrument model of the auxiliary test instrument, the instrument category and / or instrument model are used as an index to query the preset protocol library to obtain the preferred communication protocol, handshake parameters and data format corresponding to the auxiliary test instrument.

[0060] The pre-built protocol library is stored using a hash table or a key-value map. The keys in the pre-built protocol library can be instrument model identifiers, and the values ​​can include the preferred communication protocol, alternative communication protocols, handshake parameters, and data format.

[0061] Handshake parameters may include communication frequency band, broadcast identifier, device address format, handshake request field, response field, session key negotiation parameters, and handshake timeout.

[0062] The data format may include the data header, sampling timestamp, data length, data type, check field, and unit information for time-series data.

[0063] For example, for smart clamp meters, the preferred communication protocol is Bluetooth Low Energy; for digital oscilloscopes, the preferred communication protocol is Wi-Fi.

[0064] Multimodal testing equipment prioritizes sending handshake requests to auxiliary testing instruments according to the preferred communication protocol.

[0065] When the preferred communication protocol fails to perform a wireless handshake, the system switches to an alternative communication protocol based on the preset priority of the communication protocols, and then performs a wireless handshake again based on the alternative communication protocol.

[0066] For example, the communication protocol priority can be: BLE, Wi-Fi, and Universal Broadcast Discovery Protocol. The specific priority of the communication protocol can be determined based on the instrument's power consumption, communication distance, data volume, and the on-site radio frequency environment.

[0067] In some embodiments, when multiple auxiliary testing instruments are identified in the detection scene, the target recognition model assigns a unique visual instance identifier to each auxiliary testing instrument.

[0068] The multimodal testing equipment sends handshake requests to multiple auxiliary test instruments and obtains the handshake response and the radio frequency signal strength of the response signal returned by each auxiliary test instrument.

[0069] First, based on the instrument model identifier in the handshake response, the visual recognition objects and communication objects are initially screened. For visual recognition objects and communication objects with different instrument models, a correspondence can be directly established based on the instrument model.

[0070] For multiple auxiliary test instruments of the same model, further matching is performed based on spatial location information and radio frequency signal strength.

[0071] Spatial location information for auxiliary testing instruments may include their center position in the visible light image, the area of ​​the detection frame, and an estimated visual distance relative to the multimodal detection device. Radio frequency signal strength can be represented by the Received Signal Strength Indication (RSSI).

[0072] Multiple visual recognition objects are sorted according to the visual distance between the auxiliary testing instrument and the multimodal detection device, and multiple communication objects are sorted according to the radio frequency signal strength from strong to weak. The visual recognition objects and communication objects corresponding to the sorted positions are then matched.

[0073] Once matching is complete, a binding record is established, including the visual instance identifier, instrument identification information, spatial location information, communication object identifier, and wireless communication link identifier. Therefore, the time-series data received through the corresponding wireless communication link can be identified as data collected by the auxiliary testing instrument represented by the corresponding visual instance.

[0074] In some embodiments, after the multimodal detection device sends a handshake request to the auxiliary testing instrument, it determines whether it receives a handshake response from the auxiliary testing instrument within a preset handshake timeout period.

[0075] For example, the preset handshake timeout can be 3 seconds, or it can be adjusted according to the communication protocol and the on-site environment.

[0076] If a handshake effect is not received within the preset handshake timeout period, the truncated binary exponential backoff algorithm is used to determine the backoff waiting time. The backoff waiting time can be expressed as: In the formula, To avoid waiting time, k is the current number of consecutive failed handshakes. This is the upper limit of the truncation range for the exponential backoff window. For the preset unit time slot, rand(0, m) is an integer randomly selected from 0 to m.

[0077] For example, when k=1, the backoff waiting time is determined between 0 and 1 unit time slots. When k=2, the backoff waiting time is determined between 0 and 3 unit time slots. When k= At this point, the random backoff window stops expanding. After the backoff waiting time ends, the multimodal detection device resends the handshake request.

[0078] In some embodiments, if the multimodal detection device receives a handshake response from the auxiliary test instrument, it performs authentication on the auxiliary test instrument. Authentication can be performed using pre-shared key authentication, digital certificate authentication, or asymmetric authentication based on elliptic curve cryptography.

[0079] For example, a multimodal detection device sends a random number to an auxiliary testing instrument; the auxiliary testing instrument signs the random number or its digest using its private key and returns the signature result; the multimodal detection device verifies the signature result using the auxiliary testing instrument's public key stored in a protocol library. If the verification passes, the auxiliary testing instrument's identity is confirmed as legitimate, and a wireless communication link is established. By performing identity authentication during the handshake process, the risk of unauthorized device access or replay attacks can be reduced.

[0080] In some embodiments, when the number of handshake retries reaches the preset maximum number of retries and a wireless communication link is still not established, the corresponding auxiliary test instrument is marked as unconnectable, and subsequent fusion diagnosis continues based on the auxiliary test instrument that has successfully established a wireless communication link.

[0081] The maximum number of retries can be set to 5.

[0082] Auxiliary test instruments marked as unconnectable will not participate in subsequent multimodal data fusion. The multimodal detection equipment will continue subsequent fusion diagnostics based on auxiliary test instruments that have successfully established wireless communication links.

[0083] For example, in a testing scenario, if a digital oscilloscope, digital multimeter, and clamp meter are identified, and the digital oscilloscope fails to connect, the data collected by the digital multimeter, clamp meter, and multimodal testing equipment itself can still be used for fusion diagnosis.

[0084] S400, acquire the timing data collected by the auxiliary testing instrument; After the wireless communication link is established, the multimodal detection device acquires the timing data collected by the auxiliary test instrument through the wireless communication link. The auxiliary test instrument can actively upload timing data according to a preset sampling frequency, or it can upload timing data after receiving a data subscription request from the multimodal detection device.

[0085] For example, for a digital oscilloscope, timing data may include multiple sampling time points and the corresponding voltage or current values ​​at each sampling time point. For a digital multimeter or clamp meter, timing data may include continuous measurements of voltage, current, resistance, or power. For a vibration meter, timing data may include vibration acceleration sequences, vibration velocity sequences, or vibration displacement sequences. For a thermometer / hygrometer or anemometer, timing data may include temperature sequences, humidity sequences, or wind speed sequences.

[0086] After receiving the time-series data, the data packets can be verified and parsed, and the raw data can be converted into standard data records containing data values, sampling timestamps, physical units, and visual instance identifiers according to the corresponding data format in the preset protocol library.

[0087] It can also perform outlier handling, missing value imputation, and normalization on time series data. Outlier handling can be achieved through range limiting, adjacent data difference limiting, or statistical outlier detection.

[0088] For time series data of different physical quantities, they can be normalized to the same numerical range to reduce the impact of different units on subsequent feature integration.

[0089] S500, based on the spatial location information, the time-series data is synchronized and spatially correlated with the data of the corresponding area of ​​the target under test in the two-dimensional thermal image matrix and / or two-dimensional acoustic image matrix acquired by the multimodal detection device to obtain aligned multimodal data; In some embodiments, timestamp synchronization information is sent to the auxiliary testing instrument based on the clock of the multimodal detection device. The time offset of the timing data relative to the two-dimensional thermal imaging matrix and / or the two-dimensional acoustic imaging matrix is ​​determined based on the data timestamps fed back by the auxiliary testing instrument. The multimodal detection device uses a simplified network time protocol for clock synchronization.

[0090] For example, let t1 be the time when the multimodal detection device sends the timestamp synchronization data packet, t2 be the local time when the auxiliary test instrument receives the data packet, t3 be the local time when the auxiliary test instrument sends the response data packet, and t4 be the time when the multimodal detection device receives the response data packet. The time offset of the auxiliary test instrument relative to the multimodal detection device can be expressed as: In the formula, This refers to the time offset of the auxiliary testing instrument relative to the clock of the multimodal testing equipment.

[0091] The time series data is time-compensated based on the time offset.

[0092] If the original data timestamp in the time series data is The compensated data timestamp is then: = , This is for converting data timestamps to the clock reference of the multimodal detection equipment.

[0093] In some embodiments, the compensated time-series data is associated with the region corresponding to the target in the two-dimensional thermal imaging matrix and / or the two-dimensional acoustic imaging matrix, based on the spatial location information of the auxiliary testing instrument.

[0094] The acquisition time of each frame of the two-dimensional thermal image matrix or two-dimensional acoustic image matrix Set a sliding time window centered on the current element: ,in, The width of half the time window.

[0095] For example, The time window is 50 milliseconds, meaning the sliding time window is 50 milliseconds before and after the acquisition time of the thermal or acoustic image frame. If the compensated time-series data timestamps satisfy: Then, the compensated time-series data will be temporally correlated with the two-dimensional thermal imaging matrix and / or the two-dimensional acoustic imaging matrix.

[0096] If the expected data packet does not fall within the corresponding sliding time window, the corresponding data packet is marked as lost. For single or a small number of consecutive lost packets, nearest neighbor interpolation or linear interpolation can be used to fill the gap. Nearest neighbor interpolation can be expressed as: ,in, The effective sampling time that is closest to the time t to be filled. These are valid sampled values.

[0097] Linear interpolation can be expressed as: ,in, , and These are the effective sampled values ​​before and after the time point to be filled.

[0098] When the number of consecutive packet losses reaches a preset threshold, the corresponding wireless communication link is determined to be abnormal, and the wireless handshake with the corresponding auxiliary test instrument is re-executed.

[0099] The preset threshold can be set to the point where no valid data is obtained for three consecutive sliding time windows.

[0100] In some embodiments, the region of interest corresponding to the target in the visible light image is determined based on the target location output by the target recognition model. Since the imaging coordinate systems of the visible light camera, infrared sensor array, and acoustic sensor array may be different, each sensor can be spatially calibrated in advance.

[0101] For example, a mapping matrix from visible light image coordinates to infrared thermal image coordinates can be used to map the target coordinates in the visible light image to a two-dimensional thermal image matrix.

[0102] For acoustic imaging data, the position of the target to be measured can be mapped into a two-dimensional acoustic-image matrix based on the installation position and attitude parameters between the visible light camera and the acoustic sensor array.

[0103] After mapping, the region of interest corresponding to the target under test is determined in the two-dimensional thermal image matrix and / or the two-dimensional acoustic image matrix.

[0104] In some embodiments, after the visual recognition object and the communication object are bound together, an association record is established between the auxiliary testing instrument, the time-series data, and the target under test. The association record includes a visual instance identifier, a communication object identifier, a target under test identifier, a region of interest for the target under test, and a data timestamp.

[0105] The target identification can be determined based on the current detection task and the spatial relationship between the auxiliary testing instrument and the target. For example, when the measurement probe, test line, or measurement end of the auxiliary testing instrument can be identified in the visible light image, the target corresponding to the auxiliary testing instrument can be determined based on the spatial connection between the measurement probe, test line, or measurement end and the region of interest of the target. When the measurement probe or test line is not identified, the target corresponding to the auxiliary testing instrument can be determined based on the spatial position of the auxiliary testing instrument, the spatial position of the target, and the preset measurement object in the current detection task.

[0106] The time-compensated temporal data is correlated to the corresponding region of interest (ROI) of the measured target in the 2D thermal image matrix and / or 2D acoustic image matrix to obtain aligned multimodal data. The aligned multimodal data can be represented as: In the formula, Identify the target being measured. The area corresponding to the target being measured. This is the two-dimensional thermal image matrix of the region corresponding to the target being measured at the f-th detection time. This is the two-dimensional acoustic-image matrix of the corresponding region. The time series data corresponds to the target being measured and falls within the corresponding sliding time window. This corresponds to the detection time.

[0107] Therefore, the aligned multimodal data not only has a unified time reference, but also clearly records the spatial correspondence between the time series data and the specific target area being tested, thus enabling it to be directly input into the subsequent fusion diagnostic model.

[0108] S600, based on the aligned multimodal data, perform fusion diagnosis on the target under test to obtain fault diagnosis results.

[0109] In S600, after the time-series data acquired by the auxiliary testing instrument is synchronized and spatially correlated with the data of the corresponding area of ​​the target under test in the two-dimensional thermal image matrix and / or two-dimensional acoustic image matrix, the aligned multimodal data is input into the preset fusion diagnostic model.

[0110] It should be noted that the aligned multimodal data in this step refers to data that has already been timestamped and associated with the target area under test, and it still retains the respective data formats of the two-dimensional thermal image matrix, the two-dimensional acoustic image matrix, and the time-series data. Subsequently, the fusion diagnostic model performs feature encoding on each of these data, and maps the encoded features to the isomorphic latent space.

[0111] In some embodiments, the step of performing fusion diagnosis on the target under test based on the aligned multimodal data to obtain a fault diagnosis result includes: Spatiotemporally aligned multimodal data is input into a preset fusion diagnostic model. The two-dimensional thermal image matrix and / or two-dimensional acoustic image matrix in the spatiotemporally aligned multimodal data are feature-encoded to obtain thermal feature vectors and / or acoustic feature vectors.

[0112] Optionally, if the aligned multimodal data includes a two-dimensional thermal image matrix, the two-dimensional thermal image matrix of the region corresponding to the target under test is input into a thermal encoder, and the two-dimensional thermal image matrix is ​​feature-encoded by the thermal encoder to obtain a thermal feature vector.

[0113] The thermal encoder uses a convolutional neural network, such as a lightweight ResNet or MobileNet network, to compress the two-dimensional thermal image matrix of the region corresponding to the measured target into thermal image coding features.

[0114] Based on the obtained thermal imaging coding features, temperature statistical features of the corresponding area of ​​the measured target can also be extracted. The temperature statistical features include at least one of the following: maximum temperature rise, temperature gradient direction, highest temperature, average temperature, and area of ​​abnormal hot spots within the area.

[0115] The thermal imaging coding features are concatenated with the temperature statistical features to obtain a thermal feature vector.

[0116] For example, when the target being tested is an air switch in a distribution cabinet, the thermal feature vector can include the convolutional coding features of the area corresponding to the air switch, the highest temperature, the maximum temperature rise relative to the ambient temperature, and the direction of the temperature gradient.

[0117] Optionally, if the aligned multimodal data includes a two-dimensional acoustic image matrix, the two-dimensional acoustic image matrix of the region corresponding to the target under test is input into the acoustic image encoder, and features such as the location of abnormal sound sources, sound intensity distribution, and abnormal acoustic regions are extracted by the acoustic image encoder to obtain an acoustic feature vector.

[0118] It is worth noting that when the aligned multimodal data includes both a two-dimensional thermal image matrix and a two-dimensional acoustic image matrix, thermal eigenvectors and acoustic eigenvectors are obtained respectively; when only one of them is included, the corresponding thermal eigenvector or acoustic eigenvector is obtained.

[0119] The temporal data in the spatiotemporally aligned multimodal data is feature-encoded to obtain auxiliary test feature vectors.

[0120] The time-series data from the aligned multimodal data is input into the instrument encoder, which performs feature encoding on the time-series data to obtain an auxiliary test feature vector. Depending on the type of auxiliary test instrument, the time-series data may include voltage waveforms, current waveforms, voltage measurements, current measurements, mechanical vibration signals, temperature and humidity data, or wind speed data.

[0121] The instrument encoder extracts time-domain statistical features, frequency-domain features, and time-frequency features from the time-series data. The time-domain statistical features may include the mean, peak-to-peak value, and root mean square value; the frequency-domain features may include the dominant frequency and harmonic distribution extracted through Fast Fourier Transform; and the time-frequency features may include anomalous transient features extracted through wavelet analysis.

[0122] The auxiliary test feature vector is obtained by concatenating at least one of the time-domain statistical features, frequency-domain features, and time-frequency features.

[0123] For example, when the auxiliary testing instrument is a digital oscilloscope, the average voltage, peak-to-peak value, root mean square value, dominant frequency, harmonic distribution, and abnormal transient characteristics corresponding to voltage spikes can be extracted from the voltage waveform acquired by the oscilloscope.

[0124] When the auxiliary testing instrument is a vibration meter, the root mean square value of vibration, the dominant frequency of vibration, the peak value of abnormal spectrum and transient impact characteristics can be extracted from the vibration time sequence signal.

[0125] When the auxiliary testing instrument is a temperature and humidity timer, since the temperature and humidity data change relatively slowly, the time-domain features such as the mean, maximum, minimum and trend of change can be mainly extracted.

[0126] The thermal feature vector and / or acoustic feature vector, and the auxiliary test feature vector are mapped to the isomorphic latent space and the features are integrated to obtain multimodal feature data.

[0127] Since the thermal feature vector, acoustic feature vector, and auxiliary test feature vector originate from different types of detection data and have different feature dimensions, sampling frequencies, and physical dimensions, the corresponding feature vectors are mapped to the isomorphic latent space in the regenerating kernel Hilbert space.

[0128] Alternatively, when mapping using thermal eigenvectors and auxiliary test eigenvectors, the global manifold alignment objective functional is: In the formula, The global manifold alignment objective functional, This is the thermal feature vector obtained by encoding the two-dimensional thermal image matrix at time t. This is the auxiliary test feature vector obtained by encoding the time series data from the auxiliary test instrument at time t. This is a homomorphic mapping operator used to project thermal eigenvectors onto an isomorphic latent space. This is a homomorphic mapping operator used to project auxiliary test feature vectors onto an isomorphic latent space. This is the phase offset compensation matrix. For time-warped smoothing regularization, For regularization terms that constrain the mapping parameters, For the regenerated nucleus Hilbert space.

[0129] The global manifold alignment objective functional is minimized by gradient descent, so that the thermal feature vectors and auxiliary test feature vectors under the same test target and the same detection time window correspond to each other in the isomorphic latent space.

[0130] In the case of aligned multimodal data including acoustic imaging data, acoustic feature vectors are mapped to the isomorphic latent space using the same processing principle as thermal feature vectors, which will not be elaborated here.

[0131] When both thermal and acoustic feature vectors are included, they can be integrated into the main control sensing features first, and then the homomorphic mapping operator can be used to perform isomorphic latent space mapping with the auxiliary test feature vector.

[0132] After completing the isomorphic latent space mapping, the latent space features of each mode are concatenated to form initial integrated features. Subsequently, these initial integrated features are input into a deep cross-attention network. This deep cross-attention network is used to calculate the correlation between thermal gradient anomalies and / or acoustic anomalies and sudden changes in current, voltage, mechanical vibration, or environmental parameters, thereby highlighting auxiliary test features related to the current abnormal state and obtaining multimodal feature data.

[0133] Based on the multimodal feature data, fault diagnosis is performed on the target under test to obtain the fault diagnosis result; the fault diagnosis result includes fault type, fault root cause and / or diagnosis confidence.

[0134] Specifically, the multimodal feature data is used as the observation input and fed into a variational Bayesian diagnostic model that integrates the edge-side expert knowledge graph. The edge-side expert knowledge graph is constructed based on domain expert experience, industry standards, equipment maintenance specifications, and historical fault data, and includes fault phenomena, physical mechanisms, and related parameters.

[0135] For example, the end-side expert knowledge graph may include: abnormal temperature rise—increased contact resistance—contact oxidation or loosening; abnormal chip heating—voltage spike—power supply circuit abnormality; decreased heat exchange efficiency—decreased airflow—fan bearing wear.

[0136] The edge expert knowledge graph can be stored as a graph database or rule base and used as a prior regularization constraint for the variational Bayesian diagnostic model.

[0137] The variational Bayesian diagnostic model introduces a latent variable Z to characterize the deep joint distribution of multimodal data and utilizes an end-point expert knowledge graph as a prior regularization constraint. The optimization objective for its lower bound of evidence is: In the formula, As the lower bound of evidence, These are the thermal and / or acoustic features obtained through feature integration. To assist in testing features, This represents the approximate posterior distribution of the latent variables. This represents the conditional probability distribution of multimodal features under the condition of latent variables. For the prior distribution constrained by the edge expert knowledge graph, It is the Kulbeck-Leibler divergence.

[0138] By optimizing the lower bound of evidence, the posterior distribution of latent variables is constrained by the physical mechanisms and fault association rules represented by the end-side expert knowledge graph, thereby reducing diagnostic results that do not conform to thermodynamic, electromagnetic, or mechanical operating laws.

[0139] The final probability output of the fault type is obtained by Monte Carlo integration over the latent variables: In the formula, This is a fault type with a very high posterior probability. For the i-th candidate fault type, Candidate fault types under latent variable Z The probability of.

[0140] In the edge device, Monte Carlo integration can be used to solve the integral of the aforementioned latent variables. To balance diagnostic accuracy and edge processing latency, importance sampling can be used, with the prior distribution corresponding to the edge expert knowledge graph serving as the proposal distribution.

[0141] The sampling frequency is 50 to 100 times, and the sampling calculation is performed in parallel by the NPU to keep the inference latency of a single fusion diagnosis within 20 milliseconds. Candidate fault types with extremely high posterior probabilities are identified as fault types, and the posterior probability corresponding to these candidate fault types is used as the diagnostic confidence level. After determining the fault type, the cause of the fault is determined based on the relationship between the fault type and the corresponding relationship in the edge-side expert knowledge graph.

[0142] For example, when thermal characteristics indicate an abnormal temperature rise at the terminals, but the current collected by the clamp meter is within the rated current range, the model reduces the posterior probability of line overload and identifies contact oxidation or loosening as the cause of the fault by following the knowledge graph path of abnormal temperature rise—increased contact resistance—contact oxidation or loosening.

[0143] In practical applications, let's take collaborative fault diagnosis of circuit boards as an example. The target under test is the main control chip on the circuit board, the multimodal testing equipment is a testing device with visible light imaging and infrared thermal imaging capabilities, and the auxiliary testing instruments include a digital oscilloscope and a digital multimeter.

[0144] The multimodal inspection equipment acquires continuous multi-frame visible light images and infrared thermal images of the circuit board and its surrounding inspection scene.

[0145] The target recognition model identifies the main control chip, digital oscilloscope, and digital multimeter on the circuit board, and outputs the specific instrument model, spatial coordinates, and recognition confidence level of the digital oscilloscope and digital multimeter.

[0146] After the cumulative confidence value of multiple consecutive frames reaches the handshake trigger threshold, the multimodal detection equipment queries the communication protocol and handshake parameters corresponding to the digital oscilloscope and digital multimeter in the preset protocol library according to the instrument model, and establishes wireless communication links respectively.

[0147] The digital oscilloscope uploads the voltage waveform of the main control chip's power supply pins, and the digital multimeter uploads the steady-state voltage value of the main control chip's power network.

[0148] The multimodal detection equipment detected an abnormally high temperature of 85℃ in the infrared thermal imaging area corresponding to the main control chip. Time-domain and time-frequency analysis of the voltage waveform determined that a transient voltage spike existed in the waveform.

[0149] The thermal characteristics, transient voltage spike characteristics, and steady-state voltage offset characteristics of the main control chip region are mapped to a homogeneous latent space and fused. Under the prior constraints of the expert knowledge graph, the variational Bayesian diagnostic model outputs the fault type and cause of abnormal heating of the main control chip due to transient overvoltage in the power supply network, and outputs the corresponding diagnostic confidence level.

[0150] In practical applications, let's take collaborative fault diagnosis of a power distribution cabinet as an example. The target under test is the A-phase air switch in the power distribution cabinet, and the auxiliary testing instrument is a smart clamp meter. The multimodal detection equipment acquires visible light images and infrared thermal images of the power distribution cabinet, identifies the A-phase air switch and the smart clamp meter, and establishes a wireless communication link with the smart clamp meter based on its instrument model.

[0151] Infrared thermal imaging data showed an abnormal hotspot at 85℃ in phase A air switch. The smart clamp meter uploaded phase A current timing data, and the measured phase A load current was 45A, which is within the preset normal load range.

[0152] The multimodal detection equipment synchronizes the thermal characteristics of the A-phase air switch with the A-phase current characteristics in time, spatially correlates them, and maps them in an isomorphic implicit space.

[0153] Since the current in phase A does not exceed the normal load range, the expert knowledge graph reduces the posterior probability of line overload faults and increases the posterior probability of increased contact resistance due to contact oxidation or loosening.

[0154] The fusion diagnostic model ultimately outputs fault diagnosis results indicating that contact oxidation or loosening leads to increased contact resistance and abnormal heating.

[0155] In the description of the above embodiments, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. "And / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0156] In the description of the above embodiments, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0157] The various operations in the specification may be described sequentially as multiple discrete actions or operations in a manner most conducive to understanding the claimed subject matter. However, the order of description should not be construed as implying that these operations must be sequentially related. Specifically, these operations may not be performed in the order presented. The described operations may be performed in a different order than in the described embodiments. Various additional operations may be performed in additional embodiments and / or the described operations may be omitted.

[0158] The above description is merely a preferred embodiment of this application and does not constitute any limitation on this application. Any person skilled in the art can make many possible variations and modifications to the technical solution of this application, or modify it into equivalent embodiments, without departing from the scope of the technical solution of this application. Therefore, all equivalent changes made based on the shape, structure, and principle of this application without departing from the content of the technical solution of this application should be covered within the protection scope of this application.

Claims

1. A collaborative diagnostic method based on multimodal perception, characterized in that, The method includes: Detection data is collected using a multimodal detection device, and the detection data includes visible light images, infrared thermal images, and / or acoustic imaging data. Based on the detection data, identify the target to be tested and at least one auxiliary testing instrument in the detection scene, and obtain the instrument identification information and spatial location information of the auxiliary testing instrument. The instrument identification information includes the instrument category and / or instrument model. Based on the instrument identification information, the communication protocol and handshake parameters corresponding to the auxiliary test instrument are obtained from the preset protocol library, and a wireless handshake is performed with the auxiliary test instrument according to the communication protocol and handshake parameters. Acquire the timing data collected by the auxiliary testing instrument; Based on the spatial location information, the time-series data is synchronized and spatially correlated with the data of the corresponding region of the target in the two-dimensional thermal image matrix and / or two-dimensional acoustic image matrix acquired by the multimodal detection device to obtain aligned multimodal data; Based on the aligned multimodal data, the tested target is fused for diagnosis to obtain the fault diagnosis result.

2. The collaborative diagnostic method based on multimodal perception according to claim 1, characterized in that, The acquisition of detection data through multimodal detection equipment includes: Within the same detection period, multiple consecutive visible light images of the detection scene are acquired, and infrared thermal imaging data and / or acoustic imaging data corresponding to the multiple visible light images are acquired simultaneously; wherein, the visible light images, the infrared thermal imaging data and / or acoustic imaging data are time-stamped to obtain a group of associated data at the same detection time.

3. The collaborative diagnostic method based on multimodal perception according to claim 1, characterized in that, The step of identifying the target object and at least one auxiliary testing instrument in the detection scene based on the detection data, and obtaining the instrument identification information and spatial location information of the auxiliary testing instrument, includes: The detection data is input into a preset target recognition model to perform spatial feature extraction and temporal feature extraction, thereby obtaining the target category and target location of the target being tested, as well as the instrument category, instrument model, spatial location coordinates, and recognition confidence level of the auxiliary testing instrument.

4. The collaborative diagnostic method based on multimodal perception according to claim 3, characterized in that, After obtaining the recognition confidence level of the auxiliary testing instrument, the process includes: Based on the detection data and / or wireless scanning information, determine the environmental noise evaluation value of the detection scene; The handshake trigger threshold is determined based on the environmental noise evaluation value; The recognition confidence scores corresponding to multiple consecutive frames of the detection data are obtained, and a weighted integral is performed to obtain the cumulative recognition confidence score. When the cumulative identification confidence level reaches the handshake trigger threshold, a wireless handshake with the auxiliary testing instrument is triggered.

5. The collaborative diagnostic method based on multimodal perception according to claim 4, characterized in that, The step of determining the environmental noise evaluation value of the detection scene based on the detection data and / or wireless scanning information includes: The visible light noise value is determined based on the image contrast and / or edge sharpness of the visible light image; The infrared thermal noise value is determined based on the inter-frame noise variance of the infrared thermal image data; The radio frequency interference value is determined based on the radio frequency electromagnetic interference intensity in the wireless scanning information; The environmental noise evaluation value is calculated based on the visible light noise value, the infrared thermal noise value, and / or the radio frequency interference value; wherein, the larger the environmental noise evaluation value, the higher the handshake trigger threshold.

6. The collaborative diagnostic method based on multimodal perception according to claim 1, characterized in that, The step of obtaining the communication protocol and handshake parameters corresponding to the auxiliary testing instrument from a preset protocol library based on the instrument identification information includes: Using the instrument category and / or instrument model as an index, the preferred communication protocol, handshake parameters, and data format corresponding to the auxiliary test instrument can be retrieved from the preset protocol library; When the preferred communication protocol fails to perform a wireless handshake, the system switches to an alternative communication protocol based on the preset priority of the communication protocols, and then performs a wireless handshake again based on the alternative communication protocol.

7. The collaborative diagnostic method based on multimodal perception according to claim 1, characterized in that, When multiple auxiliary testing instruments are identified in the testing scenario, a wireless handshake is performed with the multiple auxiliary testing instruments to establish a wireless communication link, including: Assign a corresponding visual instance identifier to each identified auxiliary testing instrument; Send handshake requests to the plurality of auxiliary test instruments and obtain the radio frequency signal strength of the response signals returned by each auxiliary test instrument; The visual recognition objects of each auxiliary testing instrument are determined based on the visual instance identifier and the spatial location information, and each visual recognition object is matched and bound to the corresponding communication object based on the radio frequency signal strength obtained during the wireless handshake process.

8. The collaborative diagnostic method based on multimodal perception according to claim 1, characterized in that, The step of performing a wireless handshake with the auxiliary testing instrument according to the communication protocol and handshake parameters includes: Send a handshake request to the auxiliary testing instrument and determine whether a handshake response is received from the auxiliary testing instrument within a preset handshake timeout period; If the handshake response is not received within the preset handshake timeout period, the backoff waiting time is determined by the binary exponential backoff algorithm, and the handshake request is resent after the backoff waiting time. If the handshake response is received, the auxiliary test instrument is authenticated, and the wireless communication link is established after the authentication is successful. If the wireless communication link is not established after the number of handshake retries reaches the preset maximum number of retries, the corresponding auxiliary test instrument will be marked as unconnectable, and subsequent fusion diagnosis will continue based on the auxiliary test instrument that has successfully established a wireless communication link.

9. The collaborative diagnostic method based on multimodal perception according to claim 1, characterized in that, The step of synchronizing and spatially associating the time-series data with the data of the corresponding region of the target in the two-dimensional thermal image matrix and / or two-dimensional acoustic image matrix acquired by the multimodal detection device based on the spatial location information includes: Using the clock of the multimodal detection device as a reference, timestamp synchronization information is sent to the auxiliary testing instrument; Based on the data timestamps fed back by the auxiliary testing instrument, the time offset of the time series data relative to the two-dimensional thermal imaging matrix and / or the two-dimensional acoustic imaging matrix is ​​determined; Based on the time offset, time compensation is performed on the time-series data; Based on the spatial position information of the auxiliary testing instrument, the compensated time-series data is associated with the area corresponding to the target in the two-dimensional thermal imaging matrix and / or the two-dimensional acoustic imaging matrix; Data packets that do not fall within the preset sliding time window are marked as lost, and the lost data is filled using nearest neighbor interpolation or linear interpolation; If the number of consecutive packet losses reaches a preset threshold, a new wireless handshake with the corresponding auxiliary testing instrument will be initiated.

10. The collaborative diagnostic method based on multimodal perception according to claim 1, characterized in that, The step of performing fusion diagnosis on the target under test based on the aligned multimodal data to obtain fault diagnosis results includes: Spatiotemporally aligned multimodal data is input into a preset fusion diagnostic model, and the two-dimensional thermal image matrix and / or two-dimensional acoustic image matrix in the spatiotemporally aligned multimodal data are feature-encoded to obtain thermal feature vectors and / or acoustic feature vectors. Feature encoding is performed on the temporal data in the spatiotemporally aligned multimodal data to obtain auxiliary test feature vectors; The thermal feature vector and / or acoustic feature vector, and the auxiliary test feature vector are mapped to the isomorphic latent space and the features are integrated to obtain multimodal feature data; Based on the multimodal feature data, fault diagnosis is performed on the target under test to obtain the fault diagnosis result; the fault diagnosis result includes fault type, fault root cause and / or diagnosis confidence.

Images