Multi-mode fusion patrol inspection system and method based on a honkong system

Abstract

The application provides a multi-mode fusion patrol inspection system and method based on a Hongmeng system, and relates to the technical field of inspection.The application realizes hardware-level synchronous collection of multi-modal data through a Hongmeng distributed soft bus, performs intelligent preliminary screening by using multi-modal data consistency, automatically locates abnormal modes and selects verification partners, realizes early and targeted discovery of faults, dynamically generates enhanced sampling instructions according to preliminary characteristics, only performs high-precision synchronous re-collection on key modes, optimizes resources and energy consumption while acquiring key evidence, provides quantitative physical verification for initial assumptions by calculating the correlation confidence between abnormal modes and verification partner data, enhances the ability to distinguish real equipment faults from sensor abnormalities, and finally generates a diagnosis result based on the dynamic weighted fusion of multi-source evidence with the correlation confidence as the core, so that the inspection conclusion is established on the basis of cross-validation and credibility weighting, and the diagnosis reliability of the inspection is comprehensively improved.

Description

Technical Field

[0001] This invention relates to the field of inspection technology, and in particular to a multi-mode fusion inspection system and method based on the HarmonyOS system. Background Technology

[0002] As industries increasingly demand higher precision and efficiency in inspections, traditional inspection methods are gradually revealing limitations in terms of coverage, real-time performance, and data processing capabilities. In the power, transportation, and industrial sectors, there is a widespread need for inspecting large areas or complex facilities, such as power transmission line inspections, urban traffic facility checks, and large factory equipment monitoring. Existing inspection systems often rely on single technologies, making it difficult to integrate data from multiple sensors. This results in incomplete information acquisition and an inability to accurately identify potential problems. Furthermore, in existing technologies, multi-sensor data suffers from a mismatch between fixed sampling strategies and dynamically changing fault characteristics. In complex industrial scenarios, this leads to either insufficient sampling resulting in the loss of critical information or continuous high-precision sampling causing resource waste. Data processing flows are often open-loop, with a linear process from acquisition and simple fusion to alarm generation lacking self-verification and optimization capabilities. The reliability and interpretability of the judgment results are poor, making it impossible to distinguish between actual equipment faults and sensor anomalies, and limiting the level of intelligence. Therefore, there is an urgent need for a new inspection method that can achieve accurate synchronization of multimodal data, intelligent guidance for in-depth exploration, and dynamic decision-making based on the credibility of evidence, so as to fundamentally improve the accuracy, reliability and efficiency of inspection.

[0003] Therefore, it is necessary to provide a multi-mode fusion inspection system and method based on the HarmonyOS system to solve the above-mentioned technical problems. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a multi-mode fusion inspection system and method based on the HarmonyOS system, achieving the beneficial effects of accurate, reliable, and efficient inspection. This invention provides a multi-mode fusion inspection method based on the HarmonyOS system, comprising: S1: Synchronously collect multimodal sensing data of the inspected equipment through the inspection terminal, and extract the primary feature vectors of all modes respectively; S2: Calculate the consistency score of the primary feature vectors between all pairs of modes. Based on the consistency score and the preset consistency threshold, identify the suspected anomalous modes and the second mode with the highest consistency score with the suspected anomalous modes, and generate an enhanced sampling instruction set. S3: Simultaneously send the corresponding instructions from the enhanced sampling instruction set to the sensors of the suspected anomalous mode and the second mode, perform enhanced sampling, and obtain the enhanced sensing data of the suspected anomalous mode and the associated sensing data of the second mode; S4: Extract the fine feature vector of the suspected anomalous mode and the associated feature vector of the second mode, and calculate the association confidence between the fine feature vector and the associated feature vector; S5: Based on the association confidence, the fine feature vector, the association feature vector and the primary feature vector of each modality are weighted and fused to generate the decision feature vector. Based on the decision feature vector and the pre-built fault judgment rule base, the fault inspection result is generated.

[0005] Preferably, in step S2, the identification of suspected anomalous modes includes the following steps: Calculate the consistency scores of the primary eigenvectors between all pairs of modes to obtain the score matrix; Calculate the average consistency score of each mode with all other modes to obtain the consistency index of each mode; The mode with the lowest consistency index below the preset consistency threshold is identified as a suspected abnormal mode.

[0006] Preferably, in step S2, generating the enhanced sampling instruction set includes the following steps: A frequency domain transformation is performed on the primary eigenvectors of the suspected anomalous modes to obtain the spectrum of the anomalous modes; All frequencies whose power spectral density values ​​exceed the effective power spectral threshold are defined as effective frequencies. The highest effective frequency is defined as the highest effective frequency. The continuous frequency range in which the accumulated signal energy reaches the first preset percentage of the total spectral energy is defined as the main energy concentration band. Based on the highest effective frequency, the theoretical minimum sampling rate is calculated using the Nyquist sampling theorem. Based on the theoretical minimum sampling rate and the preset oversampling coefficient, the oversampling rate of the suspected anomalous mode in the enhanced sampling is determined. The lower and upper limits of the main energy concentration frequency band are set as the passband frequency range of the anti-aliasing filter on the signal acquisition link of the suspected anomalous mode in enhanced sampling. An enhanced sampling instruction set is generated based on the oversampling rate and the passband frequency range of the anti-aliasing filter.

[0007] Preferably, in step S2, the enhanced sampling instruction set includes: A first instruction sets an oversampling rate and an anti-aliasing filter for a suspected anomalous mode, and a second instruction performs synchronous sampling of the sensor of the second mode at the same oversampling rate within the same time window.

[0008] Preferably, in step S3, after performing enhanced sampling, the method further includes: Perform timestamp alignment verification on the obtained enhanced sensing data and associated sensing data; If the start time deviation does not exceed the preset deviation tolerance, the enhanced sensor data and associated sensor data will be retained. If the start time deviation exceeds the preset deviation tolerance, then determine whether the number of retries for this enhanced sampling exceeds the preset maximum number of retries; If the number of retries does not exceed the preset maximum number of retries, the data obtained in that retries is discarded, and step S3 is executed again. If the number of retries exceeds the preset maximum number of retries, the enhanced sampling process will be abandoned and a system-level synchronization anomaly diagnosis will be triggered.

[0009] Preferably, in step S4, the method for calculating the association confidence is as follows: Calculate the cross-correlation coefficient between the fine feature vector and the associated feature vector within the same time window, and use the absolute value of the cross-correlation coefficient as the association confidence.

[0010] Preferably, in step S5, the step of generating the decision feature vector includes: If the association confidence is lower than the preset confidence threshold, the first basic importance score of the fine feature vector is set to zero. If the association confidence is not lower than the preset confidence threshold, the association confidence is assigned as the first basic importance score of the fine feature vector; Assign a second basic importance score to the associated feature vectors; Each primary feature vector is assigned a third basic importance score, and the third basic importance score is less than the second basic importance score. The first basic importance score, the second basic importance score, and all third basic importance scores are normalized to obtain the weighted fusion weight; Based on the weighted fusion weights, the fine feature vector, the associated feature vector, and the primary feature vector of each modality are weighted and fused to generate the decision feature vector.

[0011] Preferably, the second basic importance score is negatively correlated with the association confidence score.

[0012] Preferably, the third basic importance score assigned to each primary feature vector is calculated based on the consistency index of the corresponding modality, a preset scaling factor, and the second basic importance score.

[0013] This invention provides a multi-mode fusion inspection system based on the HarmonyOS system, comprising: The multi-source feature extraction module is used to synchronously collect multimodal sensing data of the inspected equipment through the inspection terminal, and extract the primary feature vectors of all modes respectively. The collaborative discrimination module is used to calculate the consistency score of the primary feature vectors between all pairs of modes. Based on the consistency score and the preset consistency threshold, it identifies the suspected anomalous mode and the second mode with the highest consistency score with the suspected anomalous mode, and generates an enhanced sampling instruction set. The enhanced sampling module is used to synchronously send the corresponding instructions in the enhanced sampling instruction set to the sensors of suspected anomalous modes and second modes, perform enhanced sampling, and obtain enhanced sensing data of suspected anomalous modes and associated sensing data of second modes. The correlation verification module is used to extract the fine feature vector of the suspected anomalous mode and the correlation feature vector of the second mode, and to calculate the correlation confidence between the fine feature vector and the correlation feature vector. The fusion decision module is used to weight and fuse the fine feature vector, the associated feature vector and the primary feature vector of each modality based on the association confidence to generate a decision feature vector. Based on the decision feature vector and the pre-built fault judgment rule base, the fault inspection result is generated.

[0014] Compared with related technologies, the multi-mode fusion inspection system and method based on HarmonyOS provided by this invention have the following beneficial effects: This invention improves the overall performance of equipment status perception and diagnosis by constructing a closed-loop "discovery-verification-decision" inspection process. It achieves hardware-level synchronous acquisition of multimodal data through the HarmonyOS distributed soft bus, ensuring a solid data foundation for subsequent analysis. Intelligent initial screening using multimodal data consistency automatically locates anomalous modes and selects verification partners, enabling early and targeted fault detection. Enhanced sampling instructions are dynamically generated based on preliminary characteristics, performing high-precision synchronous re-acquisition only on key modes, optimizing resources and energy consumption while acquiring crucial evidence. By calculating the correlation confidence between anomalous modes and verification partner data, quantitative physical verification is provided for the initial hypothesis, enhancing the ability to distinguish between real equipment faults and sensor anomalies. Finally, multi-source evidence is dynamically weighted and fused using correlation confidence as the core, and diagnostic results are generated through a fault judgment rule base. This ensures that inspection conclusions are based on cross-validation and confidence weighting, thereby comprehensively improving the diagnostic reliability, interpretability, and overall intelligence level of the system. Attached Figure Description

[0015] Figure 1 This is a flowchart of a multi-mode fusion inspection method based on the HarmonyOS system according to the present invention; Figure 2 This is a module structure diagram of a multi-mode fusion inspection system based on the HarmonyOS system according to the present invention. Detailed Implementation

[0016] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the drawings, not all structures. Moreover, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0017] It should also be noted that, for ease of description, the accompanying drawings show only the parts relevant to the invention and not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations (or steps) as sequential processes, many of the operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but it may also have additional steps not included in the drawings. The process may correspond to a method, function, procedure, subroutine, subroutine, etc.

[0018] Example 1 A multi-mode fusion inspection method based on the HarmonyOS system, in specific implementation, such as... Figure 1 As shown, a flowchart of a multi-mode fusion inspection method based on the HarmonyOS system according to the present invention is illustrated, including: Step S1: Synchronously collect multimodal sensing data of the inspected equipment through the inspection terminal, and extract the primary feature vectors of all modes respectively.

[0019] In a specific implementation project, for example, the inspection terminal is a mobile device running the HarmonyOS operating system. Its integrated multimodal sensor group includes, but is not limited to, a three-axis digital accelerometer based on MEMS technology as a vibration sensor, a capacitive microphone as an acoustic sensor, an uncooled focal plane array infrared thermal imager, and a CMOS visible light camera. All sensors are connected to the terminal main control unit via the HarmonyOS distributed soft bus. The soft bus provides a unified, microsecond-level precision hardware clock source. The main control unit broadcasts a global synchronization trigger command to ensure all sensors operate on a unified clock. The system begins acquisition based on a reference point, and assigns a corresponding high-precision timestamp to each data frame to achieve strict spatiotemporal alignment of multimodal data. During acquisition, for example, a vibration sensor continuously acquires a 1-second data block at a fixed sampling rate of 8kHz, an acoustic sensor continuously acquires a 1-second data block at a fixed sampling rate of 16kHz, an infrared thermal imager acquires 10 frames of 320x240 pixel temperature images within 1 second at a frame rate of 10Hz, and a visible light camera triggers a single acquisition of a 1920x1080 pixel RGB image at the start of the 1-second acquisition window. After acquisition, the system... Parallel execution of primary feature vector extraction: For 1 second of vibration data, calculate its time-domain features, including but not limited to RMS value, peak value, peak-to-peak value, waveform index, impulse index, and margin index, and calculate its spectrum at the equipment characteristic frequencies, including but not limited to the amplitude at the octave of the blade passage frequency, forming a 7-dimensional vibration primary feature vector; For 1 second of acoustic data, calculate its A-weighted sound pressure level and 1kHz octave band sound pressure level, and calculate its spectrum in a specific frequency band, for example, the energy proportion in the specific frequency band 4-6kHz, forming a 3-dimensional acoustic ... amplitude at the octave of the blade passage frequency, forming a 7-dimensional vibration primary feature vector; For 1 second of acoustic data, calculate its A-weighted sound pressure level and 1kHz octave band sound pressure level, and calculate its amplitude at the octave of the blade passage frequency, forming a 7-dimensional vibration primary feature vector; For 1 second of acoustic data, calculate its A-weighted sound pressure level and 1kHz octave band sound pressure level, and calculate its amplitude at the octave of the blade passage frequency, forming a 7-dimensional vibration primary feature vector; For 1 second of acoustic data, calculate its A-weighted sound pressure level and 1kHz octave band sound pressure level, and calculate its amplitude at the octave of the For a zero-frame infrared image, calculate the average temperature, maximum temperature, and maximum temperature difference within a specified region of interest, including but not limited to the bearing housing and motor housing, to form a 3D infrared primary feature vector. For a single-frame visible light image, convert it to grayscale and calculate the contrast, correlation, and entropy of the grayscale co-occurrence matrix of the entire region to form a 3D visible light primary feature vector. Finally, concatenate the primary feature vectors of the above four modes in a predetermined order to form a multimodal primary feature vector set containing 16 feature values ​​for consistency analysis in subsequent steps.

[0020] Step S2: Calculate the consistency score of the primary feature vectors between all pairs of modes. Based on the consistency score and the preset consistency threshold, identify the suspected anomalous modes and the second mode with the highest consistency score with the suspected anomalous modes, and generate an enhanced sampling instruction set.

[0021] Specifically, in step S2, the identification of suspected anomalous modes includes the following steps: Calculate the consistency scores of the primary eigenvectors between all pairs of modes to obtain the score matrix; Calculate the average consistency score of each mode with all other modes to obtain the consistency index of each mode; The mode with the lowest consistency index below the preset consistency threshold is identified as a suspected abnormal mode.

[0022] Specifically, in step S2, generating the enhanced sampling instruction set includes the following steps: A frequency domain transformation is performed on the primary eigenvectors of the suspected anomalous modes to obtain the spectrum of the anomalous modes; All frequencies whose power spectral density values ​​exceed the effective power spectral threshold are defined as effective frequencies. The highest effective frequency is defined as the highest effective frequency. The continuous frequency range in which the accumulated signal energy reaches the first preset percentage of the total spectral energy is defined as the main energy concentration band. Based on the highest effective frequency, the theoretical minimum sampling rate is calculated using the Nyquist sampling theorem. Based on the theoretical minimum sampling rate and the preset oversampling coefficient, the oversampling rate of the suspected anomalous mode in the enhanced sampling is determined. The lower and upper limits of the main energy concentration frequency band are set as the passband frequency range of the anti-aliasing filter on the signal acquisition link of the suspected anomalous mode in enhanced sampling. An enhanced sampling instruction set is generated based on the oversampling rate and the passband frequency range of the anti-aliasing filter.

[0023] Specifically, in step S2, the enhanced sampling instruction set includes: A first instruction sets an oversampling rate and an anti-aliasing filter for a suspected anomalous mode, and a second instruction performs synchronous sampling of the sensor of the second mode at the same oversampling rate within the same time window.

[0024] In the specific implementation process, firstly, suspected abnormal modes are identified. For example, the Pearson correlation coefficients between all pairs of extracted primary feature vectors of all modes are calculated, and this is used as a consistency score to obtain a symmetric scoring matrix. Next, for each row in the scoring matrix, the arithmetic mean of the consistency scores of the corresponding mode with all other modes is calculated. This arithmetic mean is the consistency index of that mode. The consistency indices of all modes are compared with a preset consistency threshold. For example, the preset consistency threshold is 0.8. The mode with the smallest consistency index below 0.8 is selected. The mode is identified as a suspected anomalous mode, and the mode with the highest consistency score with this suspected anomalous mode is identified as the second mode. Subsequently, an enhanced sampling instruction set is generated to perform a Fast Fourier Transform on the primary feature vector of the suspected anomalous mode to obtain its spectrum. In this spectrum, for example, the frequency point where the signal power spectral density value first falls below the baseline noise mean by 3dB is determined as the effective power spectral threshold. All frequency points where the power spectral density exceeds the effective power spectral threshold are marked as effective frequencies, with the highest effective frequency being determined as the highest effective frequency. At the same time, the cumulative signal energy of the spectrum is calculated, for example, starting from the highest effective frequency. The continuous frequency range corresponding to when the accumulated energy reaches 95% of the total spectral energy from a low frequency is determined as the main energy concentration band. Then, according to the Nyquist sampling theorem, for example, the highest effective frequency is multiplied by 2 to obtain the theoretical minimum sampling rate. This theoretical minimum sampling rate is then multiplied by a preset oversampling coefficient, for example, set to 2.5, to obtain the actual oversampling rate required for enhanced sampling of the suspected anomalous mode. Simultaneously, the lower and upper limits of the main energy concentration band are set as the passband frequencies of the anti-aliasing filter on the signal acquisition link when driving the suspected anomalous mode sensor. For example, the anti-aliasing filter uses a fourth-order Butterworth low-pass filter. Finally, based on the above calculation results, a structured enhanced sampling instruction set is generated, which contains two explicit instructions: the first instruction is issued to the sensor suspected of having an anomalous mode, setting its sampling rate to the calculated oversampling rate and setting the passband of its anti-aliasing filter to the specified frequency range; the second instruction is issued to the sensor of the second mode, requiring it to start data acquisition at the exact same oversampling rate as the suspected anomalous mode, and within the exact same sampling time window triggered by the system's unified hardware clock.

[0025] Step S3: Simultaneously send the corresponding instructions from the enhanced sampling instruction set to the sensors of the suspected anomalous mode and the second mode, perform enhanced sampling, and obtain the enhanced sensing data of the suspected anomalous mode and the associated sensing data of the second mode.

[0026] Specifically, in step S3, after performing enhanced sampling, the following is also included: The obtained enhanced sensing data and associated sensing data are time-stamp aligned and verified. If the start time deviation does not exceed the preset deviation tolerance, the enhanced sensor data and associated sensor data will be retained. If the start time deviation exceeds the preset deviation tolerance, then determine whether the number of retries for this enhanced sampling exceeds the preset maximum number of retries; If the number of retries does not exceed the preset maximum number of retries, the data obtained in that retries is discarded, and step S3 is executed again. If the number of retries exceeds the preset maximum number of retries, the enhanced sampling process will be abandoned and a system-level synchronization anomaly diagnosis will be triggered.

[0027] In the specific implementation process, after receiving the generated enhanced sampling instruction set, instructions are sent to the target sensor via the HarmonyOS distributed soft bus. The instruction sending is synchronous, that is, an instruction packet containing a precise global start timestamp is created, and the atomic broadcast capability of the soft bus ensures that the data acquisition units of the sensors corresponding to the suspected anomalous mode and the second mode receive the instruction within the same microsecond clock cycle. The instruction packet contains two specific instructions. The first instruction is sent to the sensor of the suspected anomalous mode, which switches its sampling rate to the oversampling rate specified in the instruction set and sets the cutoff frequency of the fourth-order Butterworth low-pass filter in its signal link to be set to... The instruction set specifies the upper limit of the passband frequency; the second instruction is sent to the sensor of the second mode, commanding it to adopt the exact same oversampling rate as in the first instruction, and waits for execution; subsequently, the system main control unit sends a unified hardware trigger pulse, and the sensors of the suspected anomalous mode and the second mode immediately start data acquisition upon receiving the pulse, and continuously acquire data for a preset duration at the new oversampling rate, thereby obtaining the enhanced sensing data stream of the suspected anomalous mode and the associated sensing data stream of the second mode; after the acquisition is completed, the system extracts the high-precision start time based on the unified hardware clock source carried in the header of each data packet of the two data streams. The system calculates the absolute difference between two timestamps, using this as the starting time deviation for the current enhanced sampling. This deviation is set to 100 microseconds. The calculated time deviation is compared to a preset deviation tolerance, exemplarily set to 100 microseconds. If the time deviation is less than 100 microseconds, the sampling synchronization is considered successful, and the two segments of enhanced sensor data and associated sensor data are retained. If the time deviation exceeds 100 microseconds, the sampling synchronization is considered failed. The system immediately queries the retry counter assigned to this inspection task for this suspected abnormal event. This counter is initially set to zero, and the system checks whether the current number of retries is less than the preset maximum number of retries. If the current number of retries is less than the preset maximum number of retries, the number of retries is incremented by one, and the two data streams that failed to acquire this time are completely discarded. Then, the same enhanced sampling instruction set is sent to the two sensors again and a new round of synchronous acquisition is triggered, that is, step S3 is executed again. If the number of retries has reached or exceeded the preset maximum number of retries, the enhanced sampling process for the current suspected abnormal mode is abandoned, and a system-level diagnostic event indicating synchronous acquisition abnormality is triggered through the system log. At the same time, the process will switch to using the multimodal sensor data and its primary feature vector cached in step S1 and directly jump to step S5 to perform fusion and diagnosis.

[0028] Step S4: Extract the fine feature vector of the suspected anomalous mode and the associated feature vector of the second mode, and calculate the association confidence between the fine feature vector and the associated feature vector.

[0029] Specifically, in step S4, the association confidence level is calculated as follows: Calculate the cross-correlation coefficient between the fine feature vector and the associated feature vector within the same time window, and use the absolute value of the cross-correlation coefficient as the association confidence.

[0030] In the specific implementation process, the enhanced sensing data stream of the suspected anomalous mode and the associated sensing data stream of the second mode are received. For example, if the suspected anomalous mode is a vibration sensor, a fourth-order Butterworth bandpass filter is applied to the enhanced sensing data according to the passband frequency range specified for the suspected anomalous mode in the enhanced sampling instruction set to accurately extract the signal components within that frequency band. Then, the effective values ​​of the filtered signal are calculated within ten consecutive time windows, each one second long, resulting in a sequence containing ten effective values. This sequence is used as the fine feature vector of the suspected anomalous mode. For the second mode, based on a pre-set physical association knowledge base, the physical quantities and their characteristics associated with the fault modes of the suspected anomalous mode are searched. Based on this, a calculation method for extracting associated features from the associated sensing data of the second mode is determined. For example, if the second mode is an acoustic sensor, the spectral energy of the associated acoustic data within the corresponding frequency band that has a harmonic relationship with the vibration anomalous frequency band is calculated within the same ten one-second time windows. A sequence containing ten energy values ​​is obtained and used as the associated feature vector of the second mode. Next, the cross-correlation coefficient between the two feature vector sequences is calculated. For example, firstly, the arithmetic mean of the ten effective values ​​of the fine feature vector and the arithmetic mean of the ten energy values ​​of the associated feature vector are calculated respectively. Then, each value in each sequence is subtracted from its own sequence's arithmetic mean to obtain two difference sequences. The differences at the same position in the two difference sequences are multiplied accordingly, and the ten products are summed to obtain a total value. Then, the square of each difference in the two difference sequences is calculated, and the square root of the sum of the ten square values ​​of the two sequences is obtained to obtain two square root values. Finally, the previously obtained total value is divided by the product of the two square root values ​​to obtain a value between -1 and +1. This value is the cross-correlation coefficient. The absolute value of the cross-correlation coefficient is taken to obtain a scalar value between 0 and 1, and this value is determined as the association confidence level of this verification, completing the transformation from specific sensor data to quantitative verification results.

[0031] Step S5: Based on the association confidence, the fine feature vector, the association feature vector and the primary feature vector of each modality are weighted and fused to generate the decision feature vector. Based on the decision feature vector and the pre-built fault judgment rule base, the fault inspection result is generated.

[0032] Specifically, in step S5, the generation of the decision feature vector includes: If the association confidence is lower than the preset confidence threshold, the first basic importance score of the fine feature vector is set to zero. If the association confidence is not lower than the preset confidence threshold, the association confidence is assigned as the first basic importance score of the fine feature vector; Assign a second basic importance score to the associated feature vectors; Each primary feature vector is assigned a third basic importance score, and the third basic importance score is less than the second basic importance score. The first basic importance score, the second basic importance score, and all third basic importance scores are normalized to obtain the weighted fusion weight; Based on the weighted fusion weights, the fine feature vector, the associated feature vector, and the primary feature vector of each modality are weighted and fused to generate the decision feature vector.

[0033] Specifically, the second basic importance score is negatively correlated with the association confidence score.

[0034] Specifically, the third basic importance score assigned to each primary feature vector is calculated based on the consistency index of the corresponding modality, a preset scaling factor, and the second basic importance score.

[0035] In the specific implementation process, the weighted fusion weights are first calculated. For example, a preset confidence threshold of 0.6 is used. The received association confidence is compared with this preset threshold. If the association confidence is lower than 0.6, the refined feature vector is deemed unreliable, and its corresponding first basic importance score is directly set to zero. If the association confidence is not lower than 0.6, the association confidence value itself is directly assigned as the first basic importance score of the refined feature vector. Next, a second basic importance score is assigned to the associated feature vector. Its value is dynamically determined according to a rule negatively correlated with the association confidence. For example… Subtracting the current association confidence from 1 yields a difference. This difference is then multiplied by a preset negative correlation coefficient (e.g., 0.3). Finally, a preset baseline value of 0.2 is added, resulting in the second baseline importance score. Next, a third baseline importance score is assigned to each primary feature vector. For any given primary feature vector, its score is determined by the modality consistency index, a preset scaling factor, and the second baseline importance score. The scaling factor is multiplied by the modality consistency index, then multiplied by the second baseline importance score, where the scaling factor is less than 1. This ensures that any third basic importance score obtained is always less than the current second basic importance score. After all basic importance scores are assigned, the system normalizes all scores to obtain the final weighted fusion weights. The sum of all weights is one. After obtaining the weighted fusion weights, weighted fusion is performed. Each value of the refined feature vector is multiplied by its corresponding weight, each value of the associated feature vector is multiplied by its corresponding weight, and each value of each primary feature vector is multiplied by its corresponding weight. Then, the results of multiplying all weights by values ​​are added element-wise to generate the final decision feature vector. Finally, the generated decision feature vector is input into a pre-built fault judgment rule base, which is a support vector machine classification model trained using historical fault data. The decision feature vector serves as the input to this model, and the model outputs a specific fault type identifier and its diagnostic confidence. A preset diagnostic output threshold is set, and the diagnostic confidence of the model output is compared with the diagnostic output threshold. If the diagnostic confidence is not lower than the diagnostic output threshold, the fault type identifier is output as the confirmed result of this inspection. If the diagnostic confidence is lower than the diagnostic output threshold, the conclusion of uncertain status is output, thus completing a complete fault inspection process.

[0036] The working principle of the multi-mode fusion inspection method based on the HarmonyOS system provided by this invention is as follows: First, utilizing the hardware-level unified clock provided by the HarmonyOS distributed soft bus, microsecond-level synchronous acquisition and primary feature extraction of multimodal sensor data, including vibration, acoustic, and infrared data, are achieved. Then, by calculating the consistency scores between all pairs of primary features across all modes, suspected anomalous modes inconsistent with the performance of most sensors and their most relevant secondary modes are identified, thus completing preliminary anomaly localization and selection of collaborating objects. Subsequently, based on intelligent analysis of the spectrum of the suspected anomalous modes, enhanced sampling instructions, including precise oversampling rates and filtering parameters, are dynamically generated. The sensors of the two modes are then instructed to perform a high-precision data acquisition synchronously with identical parameters, serving as a controlled verification experiment for the initial suspicions. Next, fine-grained and correlated features are extracted from the synchronously acquired paired data, and the correlation confidence between them is calculated, thereby quantitatively verifying the physical correlation of the initial anomaly hypothesis. Finally, using association confidence as the core arbitrator, the fusion weights of fine features, association features, and other primary features in the final decision are dynamically adjusted to generate a confidence-weighted decision feature vector, which is then input into a pre-built fault judgment rule base to output a highly reliable fault diagnosis result. This forms a complete autonomous diagnostic closed loop of inspection, from multimodal data cross-verification to targeted exploration and verification, and then to evidence-weighted decision-making.

[0037] Example 2 A multi-mode fusion inspection system based on the HarmonyOS system, in its specific implementation process, such as... Figure 2 As shown, it illustrates the module structure diagram of a multi-mode fusion inspection system based on the HarmonyOS system according to the present invention, including: The multi-source feature extraction module 100 is used to synchronously collect multimodal sensing data of the inspected equipment through the inspection terminal, and extract the primary feature vectors of all modes respectively. The collaborative discrimination module 200 is used to calculate the consistency score of the primary feature vectors between all pairs of modes. Based on the consistency score and the preset consistency threshold, it identifies the suspected abnormal mode and the second mode with the highest consistency score with the suspected abnormal mode, and generates an enhanced sampling instruction set. The enhanced sampling module 300 is used to synchronously send corresponding instructions from the enhanced sampling instruction set to the sensors of the suspected anomalous mode and the second mode, perform enhanced sampling, and obtain enhanced sensing data of the suspected anomalous mode and the associated sensing data of the second mode. The association verification module 400 is used to extract the fine feature vector of the suspected anomalous mode and the association feature vector of the second mode, and to calculate the association confidence between the fine feature vector and the association feature vector. The fusion decision module 500 is used to weight and fuse the fine feature vector, the associated feature vector and the primary feature vector of each modality based on the association confidence to generate a decision feature vector. Based on the decision feature vector and the pre-built fault judgment rule base, the fault inspection result is generated.

[0038] The working principle of the multi-mode fusion inspection system based on HarmonyOS provided by this invention is as follows: This invention leverages the underlying capabilities of the HarmonyOS distributed soft bus. A multi-source feature extraction module 100 uniformly schedules heterogeneous sensors to achieve hardware-level synchronous acquisition and primary feature calculation of multimodal data, including vibration, acoustic, and infrared data. A collaborative discrimination module 200 receives all primary features and, by calculating an intermodal consistency score matrix, intelligently identifies isolated suspected anomalous modes and their most relevant second modes, generating an enhanced sampling instruction set containing precise parameters. An enhanced sampling module 300 atomically distributes the specific parameters and synchronous trigger commands from the instruction set via the HarmonyOS soft bus, driving the suspected anomalous and second-mode sensors to perform a high-precision, same-time-window collaborative data acquisition to complete a targeted investigation experiment. A correlation verification module 400 extracts fine-grained and correlated features from the synchronously acquired paired data and quantifies the correlation confidence between them by calculating their cross-correlation coefficients, thereby empirically testing the initial anomalous hypothesis. Ultimately, the fusion decision module 500 uses the association confidence level as the arbitration core, dynamically weights and fuses fine features, association features, and various primary features to generate a decision vector, and calls the pre-set fault judgment model to output diagnostic conclusions, thereby realizing full-process autonomous inspection from multimodal perception and cross-validation to intelligent decision-making at the system level.

[0039] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0040] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compactdisc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0041] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

Claims

1. A multi-mode fusion inspection method based on the HarmonyOS system, characterized in that, The inspection method includes: S1: Synchronously collect multimodal sensing data of the inspected equipment through the inspection terminal, and extract the primary feature vectors of all modes respectively; S2: Calculate the consistency score of the primary feature vectors between all pairs of modes. Based on the consistency score and the preset consistency threshold, identify the suspected anomalous modes and the second mode with the highest consistency score with the suspected anomalous modes, and generate an enhanced sampling instruction set. S3: Simultaneously send the corresponding instructions from the enhanced sampling instruction set to the sensors of the suspected anomalous mode and the second mode, perform enhanced sampling, and obtain the enhanced sensing data of the suspected anomalous mode and the associated sensing data of the second mode; S4: Extract the fine feature vector of the suspected anomalous mode and the associated feature vector of the second mode, and calculate the association confidence between the fine feature vector and the associated feature vector; S5: Based on the association confidence, the fine feature vector, the association feature vector and the primary feature vector of each modality are weighted and fused to generate the decision feature vector. Based on the decision feature vector and the pre-built fault judgment rule base, the fault inspection result is generated.

2. The multi-mode fusion inspection method based on HarmonyOS system according to claim 1, characterized in that, In step S2, the identification of suspected anomalous modes includes the following steps: Calculate the consistency scores of the primary eigenvectors between all pairs of modes to obtain the score matrix; Calculate the average consistency score of each mode with all other modes to obtain the consistency index of each mode; The mode with the lowest consistency index below the preset consistency threshold is identified as a suspected abnormal mode.

3. The multi-mode fusion inspection method based on HarmonyOS system according to claim 2, characterized in that, In step S2, generating the enhanced sampling instruction set includes the following steps: A frequency domain transformation is performed on the primary eigenvectors of the suspected anomalous modes to obtain the spectrum of the anomalous modes; All frequencies whose power spectral density values ​​exceed the effective power spectral threshold are defined as effective frequencies. The highest effective frequency is defined as the highest effective frequency. The continuous frequency range in which the accumulated signal energy reaches the first preset percentage of the total spectral energy is defined as the main energy concentration band. Based on the highest effective frequency, the theoretical minimum sampling rate is calculated using the Nyquist sampling theorem. Based on the theoretical minimum sampling rate and the preset oversampling coefficient, the oversampling rate of the suspected anomalous mode in the enhanced sampling is determined. The lower and upper limits of the main energy concentration frequency band are set as the passband frequency range of the anti-aliasing filter on the signal acquisition link of the suspected anomalous mode in enhanced sampling. An enhanced sampling instruction set is generated based on the oversampling rate and the passband frequency range of the anti-aliasing filter.

4. The multi-mode fusion inspection method based on HarmonyOS system according to claim 3, characterized in that, In step S2, the enhanced sampling instruction set includes: A first instruction sets an oversampling rate and an anti-aliasing filter for a suspected anomalous mode, and a second instruction performs synchronous sampling of the sensor of the second mode at the same oversampling rate within the same time window.

5. The multi-mode fusion inspection method based on HarmonyOS system according to claim 4, characterized in that, In step S3, after performing enhanced sampling, the following is also included: Perform timestamp alignment verification on the obtained enhanced sensing data and associated sensing data; If the start time deviation does not exceed the preset deviation tolerance, the enhanced sensor data and associated sensor data will be retained. If the start time deviation exceeds the preset deviation tolerance, then determine whether the number of retries for this enhanced sampling exceeds the preset maximum number of retries; If the number of retries does not exceed the preset maximum number of retries, the data obtained in that retries is discarded, and step S3 is executed again. If the number of retries exceeds the preset maximum number of retries, the enhanced sampling process will be abandoned and a system-level synchronization anomaly diagnosis will be triggered.

6. The multi-mode fusion inspection method based on HarmonyOS system according to claim 5, characterized in that, In step S4, the association confidence level is calculated as follows: Calculate the cross-correlation coefficient between the fine feature vector and the associated feature vector within the same time window, and use the absolute value of the cross-correlation coefficient as the association confidence.

7. The multi-mode fusion inspection method based on HarmonyOS system according to claim 6, characterized in that, In step S5, the steps for generating the decision feature vector include: If the association confidence is lower than the preset confidence threshold, the first basic importance score of the fine feature vector is set to zero. If the association confidence is not lower than the preset confidence threshold, the association confidence is assigned as the first basic importance score of the fine feature vector; Assign a second basic importance score to the associated feature vectors; Each primary feature vector is assigned a third basic importance score, and the third basic importance score is less than the second basic importance score. The first basic importance score, the second basic importance score, and all third basic importance scores are normalized to obtain the weighted fusion weight; Based on the weighted fusion weights, the fine feature vector, the associated feature vector, and the primary feature vector of each modality are weighted and fused to generate the decision feature vector.

8. The multi-mode fusion inspection method based on HarmonyOS system according to claim 7, characterized in that, The second basic importance score is negatively correlated with the association confidence score.

9. A multi-mode fusion inspection method based on HarmonyOS system according to claim 8, characterized in that, The third basic importance score assigned to each primary feature vector is calculated based on the consistency index of the corresponding modality, a preset scaling factor, and the second basic importance score.

10. A multi-mode fusion inspection system based on the HarmonyOS system, characterized in that, The multi-mode fusion inspection method based on the HarmonyOS system, as described in any one of claims 1 to 9, comprises: The multi-source feature extraction module is used to synchronously collect multimodal sensing data of the inspected equipment through the inspection terminal, and extract the primary feature vectors of all modes respectively. The collaborative discrimination module is used to calculate the consistency score of the primary feature vectors between all pairs of modes. Based on the consistency score and the preset consistency threshold, it identifies the suspected anomalous mode and the second mode with the highest consistency score with the suspected anomalous mode, and generates an enhanced sampling instruction set. The enhanced sampling module is used to synchronously send the corresponding instructions in the enhanced sampling instruction set to the sensors of suspected anomalous modes and second modes, perform enhanced sampling, and obtain enhanced sensing data of suspected anomalous modes and associated sensing data of second modes. The correlation verification module is used to extract the fine feature vector of the suspected anomalous mode and the correlation feature vector of the second mode, and to calculate the correlation confidence between the fine feature vector and the correlation feature vector. The fusion decision module is used to weight and fuse the fine feature vector, the associated feature vector and the primary feature vector of each modality based on the association confidence to generate a decision feature vector. Based on the decision feature vector and the pre-built fault judgment rule base, the fault inspection result is generated.

Images