Intelligent port equipment inspection method based on multi-source data

Abstract

The invention provides a port equipment intelligent inspection method based on multi-source data, and the method comprises the steps: enabling an unmanned plane to synchronously collect the multi-modal data, such as infrared thermal imaging, three-axis vibration, visible light and time sequence working conditions, and achieving the heterogeneous data alignment through high-precision space-time marking; performing normalization correction, filtering transformation and enhancement processing on each modal data to form a standard feature tensor; constructing a causal sensitivity initialization weight in combination with a historical fault sample, driving a dynamic weighted fusion network to generate multi-modal joint representation, and outputting an abnormal score and structured attribution explanation in a double-path reasoning network; through user feedback and man-machine collaborative correction, weight closed-loop adaptive evolution is realized, the accuracy and discrimination interpretability of equipment anomaly detection are improved, and maintenance decision and continuous optimization of an inspection system are facilitated.

Description

Technical Field

[0001] This invention relates to the field of intelligent analysis and anomaly identification of multi-source data, and in particular to an intelligent inspection method for port equipment based on multi-source data. Background Technology

[0002] With the rapid development of intelligent operation and maintenance of port equipment, using drones equipped with multi-source sensors for equipment inspection and anomaly identification has become a mainstream technical solution to improve port equipment safety and operational efficiency. Currently, mainstream systems in the industry generally adopt advanced multimodal joint analysis methods, integrating diverse data such as infrared thermal imaging, vibration monitoring, visible light vision, and operating parameters. Leveraging deep learning or data fusion models, they achieve intelligent judgment of equipment health status and anomaly detection. These methods can significantly improve the ability to detect anomalies in port equipment under complex operating conditions and have been deployed and validated in actual port automated inspection and remote fault early warning applications. Existing multi-source data inspection anomaly identification technologies mainly employ end-to-end black-box neural network models or feature-level fusion algorithms to jointly process and identify anomalies in massive amounts of sensor data. These technologies primarily focus on improving the accuracy and coverage of anomaly detection by optimizing model structures and enhancing data fusion capabilities to achieve unified analysis and efficient detection of multimodal inputs such as infrared, vibration, and visible light. In practical applications, they have been deployed in maintenance scenarios such as port container hoisting equipment, rail machinery, and electric drive systems for anomaly identification and early warning. However, in actual industrial operation and maintenance work, traditional anomaly identification systems have significant technical limitations: First, the anomaly detection results output by the system are mostly single scores or warning labels, lacking attribution explanations for specific anomalies, key characteristics, and contributing sensors, making it difficult to provide effective decision support for operation and maintenance personnel. Secondly, existing technologies lack a systematic user feedback mechanism, making it impossible to attribute and correct false or missed events and to adaptively optimize the model. This results in insufficient transparency and credibility of anomaly identification results, affecting the operation and maintenance team's trust in the intelligent judgment results. Furthermore, most systems use static weights or preset fusion parameters, lacking causal experience based on the history of typical port equipment failures, making it difficult to dynamically adjust the discrimination contribution of each sensor mode in different scenarios. Furthermore, some interpretability enhancement solutions rely on external knowledge graphs or complex counterfactual reasoning, resulting in cumbersome model structures that are prone to technical overlap with existing general avoidance mechanisms, making it difficult to achieve efficient integration and build patent barriers. Summary of the Invention

[0003] In order to solve the above-mentioned technical problems, the present invention provides an intelligent inspection method for port equipment based on multi-source data.

[0004] The technical solution of this invention is implemented as follows: A method for intelligent inspection of port equipment based on multi-source data, comprising: S1: Acquire multi-source sensor data collected by UAV during port equipment inspection. The multi-source sensor data includes infrared thermal imaging sequences, triaxial vibration signals, visible light image frames, and time-series operating parameters. Add corresponding spatial location labels and timestamps to each type of data to form an original observation dataset with spatiotemporal alignment markings. S2: Perform modality-specific preprocessing on the original observation dataset, including non-uniformity correction and temperature field normalization on the infrared thermal imaging sequence, bandpass filtering and time-frequency transformation on the triaxial vibration signal to extract short-time Fourier features, and dehazing enhancement and edge sharpening on the visible light image frame to generate normalized feature tensors aligned to each modality. S3: Based on the statistical distribution characteristics of typical abnormal patterns in the historical fault case library, a causal sensitivity initialization weight vector is constructed, in which initial causal weight coefficients are configured for different sensor modes. This weight vector reflects the prior discrimination contribution of each mode in the known fault type and serves as the initial basis for dynamic weighted fusion. S4: Input the standardized feature tensor and the causal sensitivity initialization weight vector into the dynamic weighted fusion network, calculate the real-time confidence index of each modality at the current time, and generate a time-varying fusion weight matrix by combining the initial causal weight coefficients, and output the fused multimodal joint representation vector. S5: Input the multimodal joint representation vector into the dual-path inference network, wherein the main path generates anomaly scores through a lightweight convolution classifier, and the auxiliary path synchronously performs attention gradient backtracking operation to extract the response intensity trajectories of key feature channels and construct a structured attribution graph matrix. S6: Based on the abnormal score value, determine whether it exceeds the adaptive threshold range. If it does, trigger the abnormal event marking and map the attribution map matrix to the predefined semantic template library to generate a structured explanation label containing four-dimensional attributes of abnormal location, dominant mode, evolution trend and related working conditions. S7: Package the structured interpretation labels and raw observation data fragments and upload them to the task management terminal, and open the user feedback interface for operation and maintenance personnel to annotate the accuracy and true cause of the identification results, forming a feedback sample set with manual annotation; S8: The causal sensitivity initialization weight vector is updated by reverse modulation using the feedback sample set, wherein the prior weight coefficients are adjusted according to the actual contribution deviation of each mode in the false alarm or missed alarm samples, thereby realizing an interpretability-driven closed-loop weight evolution mechanism.

[0005] The present invention provides an intelligent inspection method for port equipment based on multi-source data, which has the following beneficial effects: (1) This invention constructs an interpretable endogenous architecture with a three-level linkage of "perception-reasoning-feedback", and for the first time deeply embeds attribution ability into the main process of anomaly identification. Without relying on external knowledge graphs or complex counterfactual generation mechanisms, it realizes the transparent expression of the anomaly judgment process. In particular, it introduces a causal sensitivity initialization module, which assigns physically meaningful initial causal weights to multi-source sensor channels based on a historical fault case library. This breaks through the limitation of traditional feature fusion that only statically weights based on data confidence, enabling the model to have a tendency perception capability for key fault modes in the initial decision-making stage, which significantly improves the accuracy and semantic consistency of early anomaly capture. (2) The present invention designs a dual-path inference network: the main path focuses on high-precision anomaly scoring calculation to ensure that the detection performance is not affected, while the auxiliary path performs attribution analysis based on attention gradient backtracking in parallel, dynamically tracking the contribution trajectory of multimodal features such as infrared thermal imaging, vibration signals, visible light images and time series operating parameters in each decision node, and integrating spatiotemporal evolution laws to generate structured interpretation labels containing four-dimensional attributes of "abnormal location, dominant mode, evolution trend and associated operating conditions"; this mechanism not only avoids the additional delay and logical disconnect problem brought about by traditional posterior interpretation methods, but also ensures the high synchronization and semantic alignment between interpretation information and original discrimination results; at the same time, a lightweight attribution compression algorithm is adopted to map the high-dimensional attention path to a predefined semantic template library, which greatly reduces the computational resource overhead required for interpretation generation and storage, supports real-time visualization on edge devices, and effectively meets the stringent requirements of industrial sites for response sensitivity and human-computer interaction efficiency. Attached Figure Description

[0006] Figure 1 This is a flowchart of an intelligent inspection method for port equipment based on multi-source data according to the present invention; Figure 2 This is a sub-flowchart of an intelligent inspection method for port equipment based on multi-source data according to the present invention; Figure 3 This is another sub-flowchart of the intelligent inspection method for port equipment based on multi-source data according to the present invention. Detailed Implementation

[0007] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0008] The following disclosure provides many different embodiments or examples for implementing different structures of the invention. To simplify the disclosure, specific examples of components and arrangements are described below. Of course, these are merely examples and are not intended to limit the invention. Furthermore, reference numerals and / or letters may be repeated in different examples; such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0009] like Figure 1 As shown, this invention provides an intelligent inspection method for port equipment based on multi-source data, specifically including: S1: Acquire multi-source sensor data collected by UAV during port equipment inspection. The multi-source sensor data includes infrared thermal imaging sequences, triaxial vibration signals, visible light image frames, and time-series operating parameters. Add corresponding spatial location labels and timestamps to each type of data to form an original observation dataset with spatiotemporal alignment markings. S2: Perform modality-specific preprocessing on the original observation dataset, including non-uniformity correction and temperature field normalization on the infrared thermal imaging sequence, bandpass filtering and time-frequency transformation on the triaxial vibration signal to extract short-time Fourier features, and dehazing enhancement and edge sharpening on the visible light image frame to generate normalized feature tensors aligned to each modality. S3: Based on the statistical distribution characteristics of typical abnormal patterns in the historical fault case library, a causal sensitivity initialization weight vector is constructed, in which initial causal weight coefficients are configured for different sensor modes. This weight vector reflects the prior discrimination contribution of each mode in the known fault type and serves as the initial basis for dynamic weighted fusion. S4: Input the standardized feature tensor and the causal sensitivity initialization weight vector into the dynamic weighted fusion network, calculate the real-time confidence index of each modality at the current time, and generate a time-varying fusion weight matrix by combining the initial causal weight coefficients, and output the fused multimodal joint representation vector. S5: Input the multimodal joint representation vector into the dual-path inference network, wherein the main path generates anomaly scores through a lightweight convolution classifier, and the auxiliary path synchronously performs attention gradient backtracking operation to extract the response intensity trajectories of key feature channels and construct a structured attribution graph matrix. S6: Based on the abnormal score value, determine whether it exceeds the adaptive threshold range. If it does, trigger the abnormal event marking and map the attribution map matrix to the predefined semantic template library to generate a structured explanation label containing four-dimensional attributes of abnormal location, dominant mode, evolution trend and related working conditions. S7: Package the structured interpretation labels and raw observation data fragments and upload them to the task management terminal, and open the user feedback interface for operation and maintenance personnel to annotate the accuracy and true cause of the identification results, forming a feedback sample set with manual annotation; S8: The causal sensitivity initialization weight vector is updated by reverse modulation using the feedback sample set, wherein the prior weight coefficients are adjusted according to the actual contribution deviation of each mode in the false alarm or missed alarm samples, thereby realizing an interpretability-driven closed-loop weight evolution mechanism.

[0010] Step S1: Acquire multi-source sensor data collected by the UAV during port equipment inspection. This multi-source sensor data includes infrared thermal imaging sequences, triaxial vibration signals, visible light image frames, and time-series operating parameters. Each type of data is appended with a corresponding spatial location label and timestamp, forming a raw observation dataset with spatiotemporal alignment markings. Specifically, this includes: S1.1: Based on the infrared thermal imager, three-axis vibration sensor, visible light camera and operating condition parameter acquisition unit carried by the UAV, perform multi-source heterogeneous data synchronous acquisition operation to obtain the raw sensing data stream including infrared thermal imaging sequence, three-axis vibration signal, visible light image frame and time-series operating condition parameters, as the initial input for multimodal perception. Based on the infrared thermal imager, three-axis vibration sensor, visible light camera and working condition parameter acquisition unit carried by the UAV, a multi-source heterogeneous sensing synchronous acquisition method is adopted (parameters: the sampling synchronization signal trigger frequency is set to 1-500Hz, preferably 100Hz; the hardware clock synchronization accuracy is less than 1ms) to achieve the simultaneous acquisition of multi-modal sensing data, so as to ensure the strict consistency of multi-source data at the sampling time. Furthermore, by calling the modal data reading functions through the sensor driver interface layer (parameters: infrared thermal imaging sensor output format is 16-bit temperature matrix, resolution 640×480; triaxial vibration sensor output format is ±16g acceleration sequence; visible light camera sampling resolution is 1920×1080; operating condition parameter acquisition unit outputs numerical streams of current, voltage, speed, etc. with a sampling frequency of 10Hz), parallel reading of the modal raw data streams is achieved, and four types of raw sampled data are obtained: infrared thermal imaging sequence, triaxial vibration signal, visible light image frame, and time-series operating condition parameters. Furthermore, a heterogeneous data buffer queue merging algorithm (parameters: queue depth 200 frames, delay compensation strategy is the nearest matching window ±2ms) is used to achieve buffering and time alignment preparation of cross-modal data streams, avoid sampling misalignment caused by differences in bus upload speed, and generate multi-source raw data packets to be uniformly labeled; Furthermore, a data stream integrity detection algorithm (parameters: frame loss threshold is ≥3 consecutive frames lost in a single mode, noise overflow threshold is twice the signal amplitude exceeding the design working range) is used to perform validity screening on the original data packets of each mode, and abnormal data frames that do not meet the sampling quality requirements are removed to ensure that subsequent normalization and tag attaching operations are based on high-quality original data; Through the above-mentioned synchronous acquisition and validity detection processing method, the physical observation results of the previous step are transformed into a raw sensor data set containing four-modal synchronous data streams, achieving the expected technical effect of building a unified time reference and a highly consistent data source for multimodal inspection tasks. For example, in a port quay crane equipment inspection scenario, the infrared thermal imager mounted on the UAV has a sampling frequency of 25Hz and outputs a temperature matrix with a resolution of 640×480; the three-axis vibration sensor samples acceleration signals at 5kHz with a range of ±16g; the visible light camera operates at a resolution of 1920×1080 at 30 frames per second; and the operating parameter acquisition unit samples operating indicators such as equipment current (0-200A), voltage (0-600V), and rotational speed (0-3000rpm) at 10Hz. When implementing the multi-source heterogeneous sensor synchronous acquisition method, the hardware master clock synchronization accuracy is controlled to be less than 1ms. Each modal data acquisition port is driven by a unified trigger signal, and the acquired infrared temperature frames are completely associated with the vibration sequence, visible light image frames, and operating parameters of the corresponding time slices. After matching by the heterogeneous data buffer queue merging algorithm, the four modal data are aligned within a time window of ±2ms, and vibration sampling segments with noise amplitudes exceeding twice the rated operating range and infrared image segments with more than 3 consecutive lost frames are eliminated through data stream integrity detection. The final output of the raw sensor data set contains valid data acquired simultaneously in four modalities in each time slice, providing a stable and consistent basic data source for subsequent spatial location labeling, timestamp assignment, and spatiotemporal registration processing. In the actual test, it significantly improved the robustness and initial accuracy of multimodal fusion anomaly identification. S1.2: Add a spatial location label provided by the UAV positioning system to each type of data item in the original sensing data stream. The spatial location label contains the geographic coordinates and relative pose information corresponding to the device component, and generate a local observation data segment with spatial attribution identifier to establish a mapping relationship between sensor readings and the physical area of ​​the device under test. S1.3: Apply a unified timestamp to all sensor channels based on the high-precision time synchronization module. The timestamp is accurate to the millisecond level and is used to align the differences in sampling time of cross-modal data to generate time-series data units with time synchronization attributes, which serve as the time reference input for realizing spatiotemporal joint alignment. S1.4: The spatiotemporal registration algorithm is used to perform coarse alignment processing on the modal data with spatial location labels and timestamps. The infrared thermal imaging sequence, triaxial vibration signal, visible light image frame and time series operating condition parameters collected within the same time window are matched at the frame level to generate a spatiotemporal association group, eliminating the asynchronous problem caused by transmission delay or sampling frequency difference. For each modal data with spatial location labels and timestamps, a coarse-grained spatiotemporal registration algorithm based on bidirectional nearest neighbor matching is adopted (parameter: time window). (t=50ms, spatial neighborhood radius r=0.5m), to establish the preliminary correspondence between cross-modal observation data within the same time period and spatial region; Furthermore, by using the time-axis interpolation resampling method (parameters: linear interpolation, sampling rate unified to 100Hz), the normalization processing of different modal data on the sampling frequency is achieved, and a multimodal data sequence with a unified time step is obtained; Furthermore, by using a spatial location label matching algorithm (parameters: Euclidean distance metric, tolerance threshold ε=0.2m), the spatial coordinate correspondence of different modal data within the same spatial neighborhood is realized, and a spatial matching index table is generated; Furthermore, a threshold determination method based on frame timestamp differences is adopted, and the time deviation between any two modal frames is calculated using the following formula. :

[0011] in, For the timestamp of mode a, The timestamp of mode b is determined by... Is it less than The t threshold is used to determine temporal proximity and mark registerable frame pairs. Furthermore, multimodal frame-level matching rules are utilized (parameter: temporal proximity is satisfied). (assuming t condition and spatial distance satisfy ε condition), frame-level matching of four types of modes—infrared thermal imaging, triaxial vibration, visible light image, and temporal operating parameters—is achieved within the same time window to generate a spatiotemporal correlation group. Through the above spatiotemporal registration and frame matching algorithms, the timestamp and spatial label information of the previous step are transformed into a set of cross-modal alignable data frames, realizing asynchronous elimination and consistency improvement of multi-source data under the conditions of transmission delay and sampling frequency difference; For example, during the inspection of container cranes in a port, the infrared thermal imaging sequence sampling rate is 60Hz, the triaxial vibration signal sampling rate is 200Hz, the visible light image frame rate is 30Hz, and the temporal operating parameter sampling rate is 20Hz. Each mode is accompanied by a spatial position label (accuracy 0.1m) and a timestamp (accuracy 1ms). A time window is used. With t=50ms and spatial radius r=0.5m, the vibration signal was first resampled to 100Hz. The time steps of the infrared thermal imaging and visible light images were interpolated to 100Hz, and the timing parameters were also interpolated to 100Hz to obtain a multimodal time series with a uniform sampling rate. Data frames of each modality within the same equipment component area were selected by spatial coordinate matching. For example, the infrared thermal imaging frame near coordinates (120.3, 45.6, 8.2) and spatially 0.15m from the visible light image frame satisfied the ε condition. The time difference between the infrared thermal imaging and visible light frames was calculated using the time deviation formula to be 35ms, which is lower than... The threshold t is used to determine the matching frame. Based on this, triaxial vibration and time-series operating parameters are matched to form a spatiotemporal correlation group, which includes one frame of infrared thermal imaging, one frame of visible light, a vibration signal sample segment, and an operating parameter sample segment. In the subsequent modality-specific preprocessing stage, this correlation group enables synchronous analysis of cross-modal data and significantly improves the input consistency of the anomaly identification algorithm. S1.5: Integrate the multi-source sensor data that has completed spatiotemporal correlation matching into a data set in a structured storage format. Each record contains modality type, original data volume, corresponding spatial location label and timestamp fields, generating an original observation dataset with spatiotemporal alignment markers, which serves as a unified input data source for the next stage of modality-specific preprocessing.

[0012] Step S2: Modality-specific preprocessing is performed on the original observation dataset. This includes performing non-uniformity correction and temperature field normalization on the infrared thermal imaging sequence, bandpass filtering and time-frequency transformation on the triaxial vibration signal to extract short-time Fourier features, and dehazing enhancement and edge sharpening on the visible light image frames to generate standardized feature tensors aligned to each modality. Specifically, this includes: S2.1: Perform non-uniformity correction processing on the infrared thermal imaging sequence. Based on the non-uniformity model of blackbody reference source calibration data and pixel response, compensate for the fixed pattern noise caused by manufacturing process differences in the focal plane array, and generate a temperature distribution image sequence after spatial noise suppression. For the infrared thermal imaging sequence with spatiotemporal alignment markers output from step S1.5, a pixel response non-uniformity modeling method based on blackbody reference source calibration data is adopted (parameters: blackbody reference temperature range is set to -20℃ to 120℃, sampling interval is 0.1℃) to achieve response curve fitting and non-uniformity coefficient extraction for each pixel of the focal plane array. Furthermore, by using a fixed-pattern noise separation algorithm (parameter: noise statistics window size is 64×64 pixels), static noise pattern recognition caused by manufacturing process differences in the observed image is achieved, and a static noise template matrix is ​​obtained; Furthermore, by using the frame-by-frame template subtraction method (parameter: noise suppression weight coefficient λ=0.85), the above static noise template matrix is ​​used to perform subtraction operations on the original infrared thermal imaging frame to suppress fixed pattern noise interference and generate a preliminary spatial noise suppression result frame. Furthermore, through a bilateral filtering algorithm (parameter: spatial domain standard deviation) =2.0 pixels, luminance range standard deviation =0.05), which achieves the goal of reducing high-frequency noise intensity while preserving edge details in the noise suppression result frame, and generating a temperature distribution image sequence that balances smoothness and detail preservation; Through the above-mentioned blackbody calibration and non-uniformity correction processing, the original infrared thermal imaging data output in the previous step is transformed into a temperature distribution image sequence after spatial noise suppression, thereby achieving the signal-to-noise ratio improvement and spatial consistency enhancement technical effects required for subsequent temperature field normalization and cross-time period comparability analysis. For example, in a port container yard equipment inspection task, a blackbody reference source is used to calibrate the temperature range from 0℃ to 80℃, with a sampling interval of 0.2℃. The response coefficient and offset value of each pixel in the focal plane array are obtained through a multi-point calibration method, forming a non-uniformity correction coefficient matrix. A fixed-pattern noise template matrix (Q size 128×128) extracted using a static noise separation algorithm is used in the subtraction operation with a noise suppression weight coefficient λ=0.9. Matrix subtraction is then performed on the original thermal imaging frame.

[0013] in, The original image pixel temperature matrix, This is the corrected image matrix after subtraction. For noise suppression weighting coefficients, A fixed noise template matrix. After bilateral filtering ( =1.5, After processing (=0.04), the spatial temperature change curve in the hot spot area of ​​the output temperature distribution image sequence becomes significantly smoother, and the peak amplitude of high-frequency noise is significantly reduced, ensuring the accuracy of subsequent temperature field normalization and the stability of attribution path analysis. S2.2: The infrared thermal imaging sequence after non-uniformity correction is normalized by temperature field. Based on the steady-state temperature statistical characteristics of typical areas on the equipment surface, the original temperature values ​​are mapped to the standard temperature range. The histogram equalization algorithm is used to enhance the contrast of the thermal map and obtain a normalized thermal feature map set with cross-time period comparability. S2.3: Bandpass filtering is applied to the triaxial vibration signal. The filtering frequency band is set based on the natural frequency range of the main transmission components of the port equipment. Rigid body motion interference below 5Hz and high-frequency noise components above 2kHz are filtered out, and effective vibration frequency band signals containing the status information of key components such as bearings and gears are extracted. A bandpass filtering algorithm (parameter: low cutoff frequency) is used for the triaxial vibration signal. Hz, high cutoff frequency Hz, filter type is finite impulse response (FIR)), to remove signals below Hz. rigid body motion interference at Hz and above High-frequency noise components at Hz; Furthermore, through the natural frequency analysis method of transmission components (parameter: typical bearing frequency)... Hz to Hz, typical frequency of gears Hz to Optimize the bandpass filter response curve (Hz) to achieve concentrated retention of the spectrum of vibration signals from key parts of the equipment; Furthermore, a window function weighting technique is employed (parameter: Hanning window length). Points, overlap rate (%), to achieve sidelobe suppression in the filtering process, and obtain the smoothness gain index to evaluate the spectral leakage suppression effect; Furthermore, the power per unit frequency of the effective frequency band signal is calculated based on the signal energy density assessment method and compared with a preset normal operating power threshold to obtain the signal effectiveness determination result; By using bandpass filtering, the original triaxial vibration signal is converted into an effective vibration frequency band signal containing only the state information of key components such as bearings and gears, thereby significantly improving the signal-to-noise ratio of vibration characteristics in high-noise environments. For example, under normal operating conditions, the sampling rate of the raw signals collected by the triaxial vibration sensor of the port quay crane equipment is set to... Hz, axial vibration at Hz to The frequency is distributed within the Hz range. An FIR type bandpass filter is selected, with the low cutoff frequency configured as follows: Hz, high cutoff frequency configuration Hz, the sampling window uses the Hanning window function, length Points, overlap rate %. The results of the spectral characteristic analysis after filtering show that, Low-frequency components below Hz are completely attenuated. The amplitude of high-frequency noise above Hz is lower than the original amplitude. %.

[0014] For bearings Hz vibration modes and gears The Hz mode shows a significant increase in peak amplitude of the filtered spectrum compared to the original signal. This is based on the energy density evaluation formula. ,in For the first Frame bandwidth power Given the total number of frames, the calculated power per unit frequency in the effective frequency band is significantly higher than the set normal operating threshold. This result is output to the subsequent short-time Fourier transform step to achieve high-confidence vibration feature extraction of the equipment's operating status; S2.4: Perform short-time Fourier transform processing on the filtered effective vibration frequency band signal, use the Hanning window function to segment the time-series signal and calculate its time-spectrum matrix, extract the three-dimensional joint features of amplitude-frequency-time, and generate a short-time Fourier feature tensor that reflects the evolution law of local vibration energy. S2.5: Perform joint processing of dehazing enhancement and edge sharpening on visible light image frames. Estimate scene transmittance based on atmospheric scattering model and restore haze-free image. Then apply Laplacian operator combined with adaptive gain control mechanism to enhance image edge details and generate a set of enhanced visible light image frames with clear texture and optimized contrast, which serves as the basic input for visual modality analysis.

[0015] like Figure 2 As shown, step S3 involves constructing a causal sensitivity initialization weight vector based on the statistical distribution characteristics of typical abnormal patterns in the historical fault case library. This vector configures initial causal weight coefficients for different sensor modes, reflecting the prior discriminative contribution of each mode to known fault types, and serves as the initial basis for dynamic weighted fusion. Specifically, this includes: S3.1: Obtain multi-source sensor data records and corresponding fault type labels from the historical fault case library. The multi-source sensor data records include infrared thermal imaging sequences, triaxial vibration signals, visible light image frames, and time-series operating parameters that are isomorphic to the current inspection system. Ensure that each record is accompanied by accurate spatial location labels, timestamps, and manually confirmed fault location and category information to form a structured historical fault sample set. Based on the output of the historical data storage module of the port intelligent inspection system, multi-source sensor data that is isomorphic to the current inspection system is used as the input object; A unified data interface protocol (parameters: supports four modalities: infrared thermal imaging sequence, triaxial vibration signal, visible light image frame, and time-series operating condition parameters) is adopted to realize the batch reading function of the original data of the historical fault case library. Furthermore, by using a spatial location label parsing algorithm (parameters: geographic coordinate system of equipment components, relative pose transformation matrix), the spatial mapping information of each historical data record is decoupled and standardized, and a set of spatial location identifiers that can be directly matched with the current UAV inspection and positioning system is obtained; Furthermore, through a high-precision time synchronization parsing module (parameters: millisecond-level time base, time synchronization protocol), the unified conversion of historical data timestamps is realized, the time synchronization offset across device acquisition ends is eliminated, and a time series index vector is generated; Furthermore, a manual annotation fusion algorithm (parameters: manually confirmed fault location coding table, abnormal category label set) is adopted to align the fields of manual attribution information in historical samples and generate a semantic annotation structure containing fault location and category; By using a combined structured processing approach that combines multimodal data with spatial location labels, timestamps, and artificial attribution labels, the results of the previous step are transformed into a historical fault sample set with complete fields and traceable indexes, thus providing stable, isomorphic, and highly reliable basic data for constructing the initial weights for causal sensitivity. For example, in the inspection of container cranes at ports, the historical fault case database stores 500 isomorphic inspection samples. Each sample includes an infrared thermal imaging sequence of 60 frames (10fps), a triaxial vibration signal sampling length of 1200 points (2kHz sampling frequency), a visible light image resolution of 1920×1080, and a time-series operating condition parameter length of 300 points (1Hz sampling frequency). Spatial location labels are encoded using a combination of geographic coordinates (WGS-84) and relative pose vectors, with a timestamp accuracy of 1 millisecond. The annotation information is provided by port equipment operation and maintenance experts and includes anomaly category labels for parts such as the hoisting motor, gearbox, and pulley assembly. After data reading, the spatial location label parsing algorithm is used to calculate the spatial mapping identifier of the target lifting motor and project it onto the coordinate system of the current UAV inspection and positioning system; the high-precision time synchronization parsing module is used to convert all local timestamps of the equipment into unified UTC millisecond counts; the manual annotation fusion algorithm is used to encode the abnormal categories provided by experts into "gear wear" (number C03) or "bearing jamming" (number B07) in the unified number table and bind them with the corresponding modal data, finally forming a structured historical fault sample set, which includes four types of modal data, spatial location labels, unified millisecond timestamps, and standardized fault location and category information, meeting the technical conditions for subsequent modal contribution analysis in terms of data consistency and label credibility; S3.2: Based on the structured historical fault sample set, perform modal contribution analysis for each type of fault, use mutual information calculation method to evaluate the nonlinear correlation strength between each sensor mode and the fault label during the fault occurrence period, and obtain the original discrimination contribution score of each mode under each type of fault, as the initial basis for quantifying the prior importance of each mode; Based on a structured historical fault sample set, infrared thermal imaging sequences, triaxial vibration signals, visible light image frames, and time-series operating condition parameters within the fault occurrence period are selected as the analysis objects. A mutual information calculation method (parameters: modal feature tensor, fault label) is used to quantify the nonlinear correlation strength between multi-source sensor modes and fault types. Furthermore, through a discretization preprocessing algorithm (parameter: equal frequency binning values), the discrete value mapping of continuous modal features in mutual information calculation is realized, and a feature state sequence suitable for discrete mutual information calculation is obtained; Furthermore, by using the conditional probability density estimation method (parameter: bandwidth selection rule based on kernel density estimation), the joint distribution model of modal features and fault labels is realized, and a joint probability matrix is ​​generated; Furthermore, the mutual information value between each mode and the fault label is calculated using the mutual information calculation method, thereby realizing the correlation value measurement between each mode and the fault label; The mutual information values ​​of each mode under different fault types are calculated using the above formula, and the mutual information values ​​are used as the original discrimination contribution scores. By using a mutual information value filtering algorithm (parameter: setting the minimum mutual information threshold), low-correlation modes are eliminated, and a set of effective contribution scores for each mode under the current fault type is obtained. Through the above processing methods, the mutual information analysis results are transformed into quantitative raw discriminant contribution scores, thereby realizing the prior discriminant importance measurement of multi-source sensor modes in known fault types; For example, in a historical failure case set of port equipment, crane mechanical failure types containing 1000 samples were selected. The temperature field characteristics of the infrared thermal imaging sequence were discretized into 10 state intervals, the frequency domain peak characteristics of the triaxial vibration signal were discretized into 8 state intervals, the texture gradient characteristics of the visible light image frame were discretized into 12 state intervals, and the load change rate of the time-series operating parameters were discretized into 6 state intervals. The joint probability of each mode and fault label was calculated using the kernel density estimation method, and the bandwidth was automatically adjusted according to the Silverman rule. The mutual information value between each mode and the fault label was calculated using the mutual information calculation method for the discretized state sequence. The mutual information value of the infrared thermal imaging mode was... The mutual information value of the triaxial vibration modes is The visible light modal mutual information value is The modal mutual information value of the time-series operating condition parameters is Set the minimum mutual information threshold to... Modes with mutual information values ​​below a threshold are removed, while the original discrimination contribution scores of infrared thermal imaging, triaxial vibration, and time-series operating condition parameter modes are retained for subsequent normalization processing and weight vector construction. In practical verification, this method can accurately characterize the prior importance of different modes in mechanical fault identification, significantly improving the discrimination efficiency and stability of subsequent dynamic weighted fusion. S3.3: The original discrimination contribution score is normalized and weighted by combining the experience rules of port equipment maintenance experts to generate a standardized modal contribution factor matrix, where each row corresponds to a typical fault type, each column corresponds to a sensor mode, and the matrix elements represent the prior discrimination influence strength of the mode on the fault type. S3.4: Based on the standardized modal contribution factor matrix, calculate the weighted average contribution value of each sensor mode on all fault types, optimize the weight distribution using the entropy weight method, suppress the tendency of overconfidence under high uncertainty, output the global causal sensitivity benchmark value of each mode, and form the basic components of the causal sensitivity initialization weight vector. Based on the standardized modal contribution factor matrix, a weighted average calculation method is used (parameter: the weight coefficient of each fault type comes from the sample proportion and importance assessment) to summarize the average discrimination contribution of each sensor mode under all fault types. Furthermore, the contribution values ​​of each mode are calculated using a weighted summation formula:

[0016] in, For fault type weighting coefficients, For the modal contribution factor matrix elements, Indicates modal index, This represents the fault type index, enabling accurate calculation of the weighted average contribution value. Furthermore, the entropy weight method (parameter: modal contribution value vector) is used to optimize the weighted average results above, and the entropy value of each mode is calculated to reflect the uncertainty of the data distribution. The confidence distribution is adjusted in reverse by the entropy value to suppress the overconfidence tendency of high uncertainty modes. Furthermore, based on the adjustment coefficients output by the entropy weighting method, they are multiplied by the weighted average contribution value to obtain the corrected global contribution value for each mode; Furthermore, the corrected contribution value is normalized to generate a global causal sensitivity benchmark vector. The elements of this vector correspond to four modes: infrared thermal imaging, triaxial vibration signal, visible light image, and time-series operating condition parameters, which serve as the basic components of the causal sensitivity initialization weight vector. Through the above weighted averaging and entropy weight optimization methods, the standardized modal contribution factor matrix result of the previous step is transformed into a stable and interpretable global causal sensitivity benchmark value, thereby realizing the robustness and rationality of the dynamic weighted fusion network initialization. For example, in a port equipment inspection scenario, let the total number of historical fault types be... The infrared thermal imaging mode is categorized into four types, with contribution factors of 0.8, 0.6, 0.4, and 0.5 for each fault type, and fault sample weighting coefficients of 30, 25, 25, and 20, respectively. To calculate the weighted average, each contribution factor is multiplied by its corresponding weighting coefficient, and the sum is calculated. The result is then divided by the total weights of 100, yielding an initial global contribution value of 0.615 for the infrared thermal imaging mode. After normalizing the contribution ratios for each type of this mode, the values ​​are {0.325, 0.243, 0.162, 0.270}. Substituting these values ​​into the entropy formula yields an entropy value of approximately 0.98, corresponding to an entropy weighting coefficient of 0.34. Multiplying 0.615 by 0.34 yields a corrected contribution value of 0.209, which is then normalized to generate a four-modal causal sensitivity baseline vector {0.209, 0.278, 0.292, 0.221}. Verification shows that this vector can significantly improve the stability and interpretability of cross-modal anomaly identification in the subsequent dynamic weighted fusion network. S3.5: The causal sensitivity initialization weight vector is mapped to the initial connection weight layer of the dynamic weighted fusion network in modal order as a priori guiding parameter for multimodal joint representation learning, so that the model has a discriminative bias based on historical experience before seeing new data, thereby improving the interpretability and convergence stability of the anomaly identification process.

[0017] like Figure 3 As shown, step S4 involves jointly inputting the standardized feature tensor and the causal sensitivity initialization weight vector into a dynamic weighted fusion network, calculating the real-time confidence index of each modality at the current time, and generating a time-varying fusion weight matrix by combining the initial causal weight coefficients, outputting the fused multimodal joint representation vector. Specifically, this includes: S4.1: Based on the causal sensitivity initialization weight vector constructed in the previous step S3, the weight vector reflects the prior discrimination contribution of infrared thermal imaging, triaxial vibration signal, visible light image and time series operating condition parameters in typical fault types. The prior weight is used as the basis for the initial parameter configuration of the dynamic weighted fusion network to introduce modal importance bias driven by historical fault knowledge and generate an initial weighted basis with interpretable prior guidance. The causal sensitivity initialization weight vector constructed based on the preceding step S3 is input to the prior discrimination contribution of four types of sensor modes—infrared thermal imaging, triaxial vibration signal, visible light image, and time-series operating condition parameters—in typical fault types. The weight vector mapping method (parameters: modality order mapping table, network initial connection layer index) is adopted to load each element in the causal sensitivity initialization weight vector into the initial connection weight layer of the dynamic weighted fusion network, thereby realizing the direct association and binding between modalities and network parameters. Furthermore, the loaded weights are normalized using a normalization algorithm (parameters: L2 norm constraint, weight normalization coefficient) to prevent the initial causal weights from causing gradient calculation offset on the numerical scale, thus obtaining a structured modal initial weight distribution matrix. Furthermore, an interpretable bias injection method (parameters: causal weight bias coefficient β, modal importance threshold τ) is adopted to embed the historical fault discrimination contribution as a bias term in the forward calculation into the fusion network, so that the model can generate a response reinforcement effect on high contribution modes in the feature fusion stage. Furthermore, by using the correlation preservation constraint algorithm (parameters: modal cross-correlation coefficient γ, constraint penalty factor λ), intermodal correlation constraints are applied in the initial stage of the network to ensure that the weight distribution maintains the historical causal relationship pattern while allowing the network to adaptively update the weights based on real-time data in subsequent stages. Through the above weight initialization and bias injection processing, the causal sensitivity information of the previous step is transformed into an interpretable prior guiding basis for the dynamic weighted fusion network, so that the fusion process has a modal importance distribution driven by historical experience at the initial moment. For example, in a port equipment inspection task, four modes are selected: infrared thermal imaging, triaxial vibration, visible light, and operating parameters. Historical fault sample analysis yields an initial weight vector for causal sensitivity of [0.35, 0.25, 0.30, 0.10]. Using a weight vector mapping method, this vector is loaded into the initial weights of the network's weighted fusion layer in modal order. The loaded weights are then normalized, and the normalization coefficient is calculated as the reciprocal of the L2 norm. After normalization, a weight distribution matrix of [0.648, 0.463, 0.555, 0.185] is obtained. A bias coefficient β is set to 0.05, and bias injection is applied to modes with weights greater than the threshold τ=0.50, increasing the weights of infrared thermal imaging and visible light modes to 0.673 and 0.580, respectively. When applying correlation preservation constraints, a penalty factor λ is set to 0.02 to suppress the difference in weights between infrared thermal imaging and visible light modes, keeping their importance close. During the initial network operation, the weight base processed as described above enables the fusion network to prioritize infrared thermal imaging and visible light information in multimodal characterization, thereby enhancing the ability to distinguish temperature and appearance anomalies while retaining the comprehensive discrimination function for vibration and operating parameters. S4.2: Local quality assessment is performed on the normalized feature tensors of each modality output from S2. Based on the signal-to-noise ratio, gradient change entropy, and time domain stability index of the data sequence within the sliding window, the real-time confidence index of each sensor mode at the current moment is calculated. The confidence of the infrared thermal imaging sequence is determined by the spatial continuity of the temperature field, the confidence of the triaxial vibration signal is measured by the frequency domain energy concentration, the confidence of the visible light image frame is evaluated by the edge structure sharpness, and the confidence of the time series operating parameters is jointly determined by the sampling integrity and trend smoothness, thus obtaining a four-dimensional real-time confidence vector. S4.3: Perform nonlinear coupling operation on the real-time confidence vector generated in S4.2 and the causal sensitivity initialization weight vector constructed in S4.1. Use the Sigmoid weighted gating mechanism to perform the fusion of the two to balance the relationship between the current observation quality and the historical discrimination value, avoid the low-quality but high-causal weight mode from dominating the decision or the high-confidence but low-correlation mode from interfering with the decision, and output the dynamically modulated intermediate mode weight distribution. The four-dimensional real-time confidence vector generated by S4.2 and the causal sensitivity initialization weight vector constructed by S4.1 are coupled nonlinearly using a Sigmoid weighted gating mechanism to achieve the function of balancing the current observation quality and historical discrimination value. By using a normalization mapping method, the components of the real-time confidence vector are... Mapping to the [-1,1] interval unifies the numerical range of confidence scores for different modalities and reduces coupling bias caused by differences in dimensions, resulting in a normalized confidence vector. ; Furthermore, the components of the weight vector W are initialized using causal sensitivity, and then compared with the normalized confidence vector. Perform product coupling at the element level according to the corresponding modal order to generate an initial weight activation vector A. This vector synchronously reflects the coupling effect between the real-time data quality and the historical discrimination contribution of each modality. A is nonlinearly compressed and mapped using the Sigmoid gate function, in the following mathematical form:

[0018] in, To activate vector elements for initial weighting, It is an exponentially decaying term to suppress weight saturation caused by extreme large values; Furthermore, the Sigmoid output vector Perform gated modulation, set the gate vector G, whose components represent the degree of activation of a specific mode in the current system, and achieve the gate effect through element-wise multiplication to form the dynamic modal weight distribution M; Through this Sigmoid weighted gating mechanism, The fusion result with W is transformed into an intermediate mode weight distribution after nonlinear compression and gating adjustment, achieving the expected technical effect of avoiding the over-dominance of low-quality but high-causal weight modes or the interference of high-confidence but low-correlation modes in the discrimination. For example, in a port crane inspection scenario, the system calculates in real time the normalized confidence level of the infrared thermal imaging mode as 0.85, the triaxial vibration mode as 0.65, the visible light mode as 0.92, and the time-series operating condition parameter mode as 0.78. The corresponding initial weight vectors for causal sensitivity are 0.70, 0.80, 0.60, and 0.50, respectively. Element-wise multiplication yields the initial weight activation vector A as [0.595, 0.520, 0.552, 0.390]. Inputting A into the Sigmoid function yields... The values ​​are [0.644, 0.627, 0.634, 0.596]. The gate vector G is set to [1, 1, 0.9, 0.95]. Element-wise multiplication yields M as [0.644, 0.627, 0.571, 0.566]. This intermediate mode weight distribution significantly improves the balance of multimodal fusion during subsequent normalization and weighted summation, ensuring the joint representation vector maintains high sensitivity to infrared thermal imaging and visible light modes. Simultaneously, it reasonably controls the influence range of vibration and operating condition modes, enhancing the stability and interpretability of anomaly identification. S4.4: Based on the intermediate mode weight distribution output by S4.3, a normalized time-varying fusion weight matrix is ​​generated through Softmax normalization. Each element of this matrix corresponds to the relative contribution ratio of each sensor mode in the fusion process at the current time, and ensures that the sum of all mode weights is 1, thereby realizing the differentiable and traceable weighted aggregation of multimodal information in the joint representation space. S4.5: Using the time-varying fusion weight matrix generated in S4.4, a weighted summation operation is performed on the standardized feature tensor output by S2. Specifically, the feature tensor of each modality is multiplied by its corresponding weight coefficient and then superimposed element by element to generate a multimodal joint representation vector with dynamic perception capability. This vector retains the key state information of the original modality and integrates the dual guidance of real-time credibility and historical causal prior, serving as the unified input of the dual-path inference network in the next stage.

[0019] Step S5: The multimodal joint representation vector is input into a dual-path inference network. The main path generates anomaly scores using a lightweight convolutional classifier, while the auxiliary path simultaneously performs attention gradient backtracking operations to extract response intensity trajectories from key feature channels and construct a structured attribution graph matrix. Specifically, this includes: S5.1: Based on the multimodal joint representation vector, it is input into the main path of the dual-path inference network. A lightweight one-dimensional convolutional neural network is used to perform a local receptive domain sliding convolution operation on the joint representation vector to extract cross-modal temporal dependency features and generate a high-order latent feature mapping tensor as the input representation for anomaly score calculation. Based on the multimodal joint representation vector as input, a lightweight one-dimensional convolutional neural network structure is adopted (parameters: kernel size is 3, stride is 1, and the number of channels is set according to the number of input modalities). Local receptive field sliding convolution operation is performed on the input vector along the time dimension to capture short-term cross-modal correlation features. Furthermore, by using multi-scale one-dimensional convolutional groups (parameters: convolutional kernel sizes of 3, 5, and 7, respectively, and a shared weight strategy), parallel extraction of modal interaction patterns within different time spans is achieved, and the outputs of each scale convolution are concatenated in the channel dimension to generate a multi-resolution feature mapping tensor. Furthermore, batch normalization (parameters: momentum coefficient 0.9, epsilon value 1e-5) is used to perform channel normalization on the concatenated feature maps to eliminate the influence of differences in feature distributions of different modalities on the convolutional response and improve the network convergence speed and stability. Furthermore, a nonlinear transformation is applied to the batch-normalized feature map using the ReLU activation function to enhance the discriminative ability of cross-modal features and suppress the low-amplitude response introduced by input noise; By using the residual connection mechanism, the output after ReLU transformation is added element-wise to the initial convolution output, which preserves the original structural characteristics of cross-modal features in the temporal dimension and prevents feature degradation caused by excessive convolution. By using the above combination of convolution, concatenation, normalization, nonlinearity, and residual processing, the multimodal joint representation vector of the previous step is transformed into a high-order latent feature mapping tensor containing cross-modal temporal dependency information, thereby providing a robust and high-fidelity input representation for anomaly scoring calculation. For example, in a port equipment inspection scenario, the input multimodal joint representation vector has a length of 128 and 4 modal channels, including infrared thermal imaging features (32-dimensional), triaxial vibration features (32-dimensional), visible light image features (32-dimensional), and operating condition parameter features (32-dimensional). In this step, the convolution kernel size is set to 3, the stride is 1, and the number of output channels is set to 64. The three sets of convolution kernels in the multi-scale one-dimensional convolution group have sizes of 3, 5, and 7, respectively. After parallel convolution of the input vector, three 64-channel feature tensors are obtained, and they are concatenated in the channel dimension to obtain a 192-channel multi-scale feature map. The batch normalization layer evaluates the mean and variance of the 192-channel data channel by channel and uses a momentum coefficient of 0.9 for exponential weighting updates. ReLU activation truncates negative responses to zero and retains the amplitude of positive responses. The residual connection adds the corresponding elements of the output of the multi-scale convolution to the output of the initial single-scale convolution to generate the final 192-channel high-order latent feature map tensor. In performance testing, the structure can significantly improve the separability of anomaly patterns while maintaining the original structure of the fused features. The discriminative power of higher-order latent features in subsequent anomaly scoring calculations is significantly improved, and the computation latency remains within an acceptable range in multiple rounds of testing. S5.2: Perform global average pooling on the high-order latent feature mapping tensor to compress the spatiotemporal dimension and retain the channel-level semantic response strength to obtain a compact feature vector, and input it into the fully connected classification head. Calculate the anomaly probability distribution based on the Softmax activation function, and output an anomaly score value between 0 and 1 to quantify the degree to which the current device state deviates from the normal mode. S5.3: The multimodal joint representation vector is synchronously input into the auxiliary path of the dual-path inference network to construct an attention mechanism layer. Based on the self-attention weight matrix, the response intensity of each sensor modality in the time-channel dimension is weighted and mapped to generate the original attention heatmap, and the relative importance distribution of different modalities and time nodes to the discrimination process is recorded. Based on the multimodal joint representation vector output by S4.5, it is synchronously input into the auxiliary path feature processing channel of the dual-path inference network as the original input data carrier for attention mechanism calculation. Employing a multi-head self-attention mechanism (parameters: number of attention heads h≥4, embedding dimension) (Consistent with the dimension of multimodal joint representation), the correlation score matrix between modal feature elements is calculated in the time-channel two-dimensional feature plane to realize the correlation modeling of cross-time step and cross-modal features; Furthermore, based on the scaled dot product attention formula, the query matrix Q, key matrix K, and value matrix V are calculated respectively. The query matrix and key matrix are generated from the input joint representation vector via a linear mapping. The attention weights are then calculated using the following formula:

[0020] in, This represents the attention weight value. The dimension of the key vector. Represents matrix multiplication. This is the matrix transpose. Furthermore, by using Softmax normalization, the correlation score between each time point and the modal channel is converted into a non-negative weight, thereby achieving a weighted mapping of different features on the response intensity. The weighted mapping result is multiplied element-wise with the original value matrix V and summed to generate a preliminary weighted feature representation vector, which reflects the distribution of the response contribution of each mode in different time segments; Furthermore, the weighted feature representation vector is rearranged and mapped to a two-dimensional tensor on the modality-time two-dimensional plane, and normalized to the 0-1 interval to form the original attention heatmap, which is used to record the relative importance of each sensor modality and time node in the discrimination process; The original attention heatmap is spatially continuous by using a two-dimensional tensor interpolation smoothing method to eliminate abrupt changes caused by sparse sampling or noise, thereby improving the stability of the visualization interpretation. Through the above self-attention weight calculation and spatiotemporal weighted mapping processing, the multimodal joint representation vector is transformed into an original attention heatmap with high-resolution importance distribution, realizing the visualization effect of contribution measurement in both modality and time dimensions. For example, for a port gantry crane inspection task, the multimodal joint representation vector dimension is set to 128, including 48 dimensions of infrared thermal imaging features, 32 dimensions of triaxial vibration signal features, 32 dimensions of visible light image features, and 16 dimensions of time-series operating condition parameter features. An 8-head self-attention mechanism is constructed using auxiliary paths, embedding the dimensions... =128, key vector dimension =16. After obtaining the Q, K, and V matrices through linear mapping, the attention weights are calculated using the scaling dot product formula, and Softmax normalization maps the weights to the 0-1 interval. During the heatmap generation process, the 128-dimensional weight vector is rearranged into a two-dimensional matrix of modality × time (4 modalities, 32 time slices), and after smoothing by bicubic interpolation, the original attention heatmap with a resolution of 4×32 is obtained. In this figure, it can be seen that the response intensity of the infrared thermal imaging modality is significantly higher than that of other modalities in the 10th to 15th time slices, indicating that the temperature anomaly in this interval has the highest contribution to the final discrimination, effectively supporting the subsequent gradient backtracking attribution analysis; S5.4: Perform gradient backtracking analysis on the original attention heatmap, use the backpropagation algorithm to calculate the gradient magnitude of the abnormal score value relative to each input feature channel, extract the key feature channels with significant contributions and their corresponding time segments, form a gradient sensitive region mask, and use it as the core evidence chain of the attribution path. S5.5: Based on the gradient-sensitive region mask, the original features in the multimodal joint representation vector are selectively reconstructed to generate a structured attribution map matrix. This matrix contains four-dimensional attributes: spatial location index, dominant modality identifier, temporal evolution interval, and feature response polarity, which are used to support the semantic mapping of subsequent interpretable labels and human-computer interaction presentation.

[0021] Step S6: Based on the anomaly score, determine whether it exceeds the adaptive threshold range. If it does, trigger an anomaly event labeling and map the attribution map matrix to a predefined semantic template library to generate structured explanation labels containing four-dimensional attributes: anomaly location, dominant mode, evolution trend, and associated working conditions. Specifically, this includes: S6.1: Based on the abnormal score value, calculate its relative offset from the adaptive threshold interval, wherein the adaptive threshold interval is dynamically updated according to the statistical distribution of abnormal score values ​​under historical normal operating conditions to reflect the context sensitivity of the current equipment operating status and obtain dynamic judgment results. S6.2: If the dynamic criterion result indicates that the abnormal score value exceeds the upper limit of the adaptive threshold range, then an abnormal event marker is triggered, and the attribution graph parsing process is activated to start the semantic decoding processing of the attribution graph matrix output by the auxiliary path. Based on the input conditions of dynamic criterion results, the abnormal score value and the corresponding adaptive threshold interval boundary parameter are loaded as the judgment data source; An interval comparison algorithm (parameters: anomaly score value a, upper limit of threshold interval b) is used to compare the deviation of the current device status with the upper limit value that does not require alarm, and generate an out-of-bounds judgment signal. Furthermore, through a logical threshold triggering mechanism (parameter: out-of-bounds judgment signal = 1), the abnormal event marking signal is set, and the signal is synchronously transmitted to the entry port of the attribution graph analysis module to activate the module's running status; Furthermore, by using a data channel switching algorithm (parameter: secondary path data cache index), the attribution graph matrix output by the secondary path of the dual-path inference network is quickly scheduled and loaded, and the attribution graph matrix is ​​completely imported into the temporary storage area of ​​the semantic decoding processing subsystem to ensure the integrity and correctness of data association in the subsequent decoding process. Furthermore, through attribution matrix parsing initialization processing (parameters: spatial location index set, dominant mode identifier set, temporal evolution interval set, feature response polarity set), the independent extraction and formatting of each dimension of the attribution matrix attributes are realized, generating a structured parsing unit that adapts to semantic template matching; Through semantic decoding algorithms, the structured parsing units of the previous step are transformed into four types of initial descriptions of attributes that are compatible with the predefined fault semantic model, thereby achieving the semantic expression initiation effect of the attribution graph. For example, in port equipment inspection tasks, the input condition for the dynamic judgment result is the anomaly score value. = Adaptive threshold range upper limit = The interval comparison algorithm outputs an out-of-bounds judgment signal 1, which, through a logical threshold trigger mechanism, marks the abnormal event and activates the attribution graph parsing module. The dimension of the attribution graph matrix loaded by the secondary path cache index is... × Includes spatial location index Coordinate set (unit: meters), three dominant mode identifiers (infrared, vibration, visible light), time evolution interval (length: (seconds), characteristic response polarity (range: / After the attribution matrix is ​​parsed and initialized, a structured parsing unit is formed for semantic decoding. During the semantic decoding stage, it is matched with predefined templates such as blockage, leakage, and temperature rise in the port equipment fault database, which greatly improves the matching degree. The initial description of the four types of attributes output can provide high-precision input for the subsequent channel importance aggregation in S6.3. S6.3: Perform channel importance aggregation operation on the attribution map matrix, identify the dominant contributing sensor mode by using the inter-modal attention weight distribution, and locate the device region with the strongest abnormal response by combining spatial location labels, and generate preliminary abnormal location and dominant mode determination results. S6.4: Based on the continuous attribution map matrix sequence within the time-series sliding window, extract the evolution trend vector of key feature channels, identify the change pattern of response intensity by fitting linear slope and detecting abrupt change points, and generate evolution trend attributes containing rising, stable or decaying characteristics. S6.5: The preliminary judgment result and the evolution trend attribute are jointly input into a predefined semantic template library for pattern matching. The semantic template library stores standardized description entries of four-dimensional attribute combinations under typical fault scenarios and outputs the structured interpretation label with the highest matching degree, which includes four-dimensional attributes such as abnormal location, dominant mode, evolution trend and related working conditions, to complete the generation of interpretability information.

[0022] Step S7: Package the structured interpretation labels and raw observation data fragments together and upload them to the task management terminal. Provide a user feedback interface for operations and maintenance personnel to annotate the accuracy and true attribution of the identification results, forming a feedback sample set with manual annotations. Specifically, this includes: S7.1: Obtain the structured interpretation labels generated by the dual-path inference network and the corresponding original observation data fragments. The structured interpretation labels include four-dimensional attributes: anomaly location, dominant mode, evolution trend, and associated operating conditions. The original observation data fragments include infrared thermal imaging sequences, triaxial vibration signals, visible light image frames, and time-series operating condition parameters. Perform data fragment truncation operation based on spatiotemporal alignment markers to ensure that the interpretation labels and the original data are accurately matched in timestamps and spatial locations. S7.2: Perform data packaging processing on the matched structured interpretation labels and the original observation data fragments, and compress them into a unified data transmission unit using a lightweight encoding protocol to generate an inspection event package with a metadata header. The metadata header records the event trigger time, UAV ID, device number and data modality type to support efficient parsing and classified storage by the task management terminal. S7.3: Upload the inspection event package to the anomaly review queue of the task management terminal, and simultaneously render the semantic content of the structured interpretation label and the time-series waveform or image snapshot of the key sensor data in the visualization interface. Based on the front-end interaction framework, build a user feedback interface to allow maintenance personnel to confirm or correct the accuracy of the anomaly judgment result by checking boxes, text input, or hot spot annotation. Upon receiving the inspection event packet containing structured interpretation tags and fragments of raw observation data, a secure transmission protocol module (parameters: TLS1.3 encryption suite, two-way certificate authentication mechanism) is used to establish an encrypted channel between the UAV and the task management terminal for the inspection event packet. The metadata field in the header of the data packet is length-checked and signature-verified to ensure the integrity and tamper-proof nature of the data in the upload path. Furthermore, through the event scheduler module inside the task management terminal (parameters: queue priority rule = abnormal events first, maximum concurrent uploads = 8), the orderly entry of inspection event packages into the abnormal review queue is realized, and a unique event identifier is created at the same time for tracking index in subsequent review and feedback stages; Furthermore, a visualization rendering engine (parameters: GPU acceleration switch = enabled, rendering frame rate ≥ 60fps) is adopted to realize the partitioned layout display of the "four-dimensional attributes" in the structured explanation label on the front-end review interface. Abnormal parts are highlighted by a three-dimensional spatial model, the dominant mode is distinguished by color coding, the evolution trend is presented by a dynamic line graph, and the related working conditions are presented in a mixed mode of text + chart. It is also rendered in parallel with the infrared thermal imaging sequence, three-axis vibration time series curve, visible light image and working condition parameter curve in the original observation data segment in multiple windows. Furthermore, a user feedback interface (parameters: event response latency <50ms, supports multi-touch operation) is built based on the front-end interaction framework to implement a checkbox-style accuracy confirmation module (with preset "correct judgment" and "incorrect judgment" options), a text input module (supports UTF-8 encoding storage of maintenance reason explanations), and a hotspot labeling module (allowing users to click and mark specific abnormal locations or signal segments in the device's 3D model or sensor data curves). The operation events of each module are transmitted back to the task management terminal backend in real time via WebSocket so as to generate feedback records immediately. Through the above-mentioned transmission, scheduling, rendering and interactive processing methods, the inspection event package is transformed from the data upload stage into an interactive feedback input that can be visualized and manually annotated on the task management terminal, so as to realize the intuitive presentation of inspection results and efficient manual verification. For example, in a port quay crane equipment inspection task, the abnormal location field in the inspection event package points to the gearbox area of ​​the quay crane trolley drive system. The dominant mode is a three-axis vibration signal, the evolution trend is a continuous increase in vibration amplitude, and the associated operating condition is low-speed no-load operation. The event package is uploaded via the TLS 1.3 protocol, the metadata header records a timestamp of 2024-05-18 14:32:15, the UAV ID is UAV-025, the equipment number is QC-102, and the data mode types include IR, VIB, VIS, and PARAM. The task management terminal adds the event package to the abnormal review queue, and highlights the gearbox model on the front-end interface, marks the rising segment of the vibration signal curve in red, shows no obvious abnormalities in the temperature field, and the visible light image shows that the gearbox appears normal. The maintenance personnel select "Correct Judgment" in the feedback interface, enter "Gearbox bearing may be lacking lubrication" in the text box, and click on the abnormal period at the hot spot mark on the vibration curve. The system backend saves the feedback in real time and binds it with the event package to form a complete triplet containing the joint representation of the input, the explanation label of the output, and the human feedback. This triplet is used for the reverse modulation learning of the causal sensitivity weights, which significantly improves the recognition reliability and response speed of such gearbox anomalies in the re-inspection task. S7.4: Based on the user's operation behavior in the feedback interface, extract the real attribution information of manual annotation, including false alarm / missed alarm judgment, actual abnormal part correction, dominant mode redistribution and working condition impact description. Use structured parsing rules to transform unstructured feedback into standardized feedback vectors and generate manual annotation feedback items corresponding to the original structured explanation labels. S7.5: The manually labeled feedback items are reverse-associated and bound with the corresponding original observation data segments, structured interpretation labels and inspection event packages to construct a feedback sample set with multimodal context information. Each sample contains a complete causal link input-output-feedback triplet, which serves as the learning basis for subsequent causal sensitivity initialization weight vector inverse modulation.

[0023] Step S8: The causal sensitivity initialization weight vector is updated by reverse modulation using the feedback sample set. This involves adjusting the prior weight coefficients based on the actual contribution deviation of each mode in the false positive or false negative samples, thus achieving an interpretability-driven closed-loop weight evolution mechanism. Specifically, this includes: S8.1: Obtain a set of feedback samples with manual annotations returned by the task management terminal. The set of feedback samples includes the accuracy judgment of the abnormal event marking results by the operation and maintenance personnel and the corresponding true attribution labels. The true attribution labels clearly indicate the actual dominant contribution status of each sensor mode in the case of false alarm or missed alarm. S8.2: Based on the true attribution labels labeled in the feedback sample set, construct a modal contribution deviation matrix, where each row corresponds to a feedback sample, and each column corresponds to four types of sensor modes: infrared thermal imaging, triaxial vibration signal, visible light image, and time-series operating condition parameters. The element value indicates whether the mode should be identified as the dominant discriminant mode in the current sample, which serves as the basis for calculating the prior weight adjustment direction. S8.3: Compare the modal contribution deviation matrix with the cause-effect sensitivity initialization weight vector element by element, calculate the difference index between the historical prior judgment contribution degree and the actual manually labeled contribution state of each modality, and generate a modal-level weight correction gradient vector to quantify the degree of systematic overestimation or underestimation of each type of sensor modality. S8.4: Based on the modal-level weight correction gradient vector, an incremental modulation is performed on the causal sensitivity initialization weight vector using an accumulation update strategy with a decay factor. A moving average mechanism is introduced to suppress the drastic fluctuations caused by a single feedback, and an updated version of the causal sensitivity initialization weight vector with enhanced stability is output. S8.5: Write the updated causal sensitivity initialization weight vector into the system knowledge base and mark the version number. At the same time, trigger the initialization parameter synchronization mechanism of the dynamic weighted fusion network to ensure that the multimodal joint representation generation process in subsequent inspection tasks can inherit the latest human-machine collaborative learning results and realize interpretability-driven closed-loop weight evolution.

[0024] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

[0025] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and rules of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for intelligent inspection of port equipment based on multi-source data, characterized in that, Includes the following steps: S1: Acquire multi-source sensor data collected by UAVs during port equipment inspection, and attach corresponding spatial location labels and timestamps to each type of data to form a raw observation dataset with spatiotemporal alignment markings; S2: Perform modality-specific preprocessing on the original observation dataset to generate standardized feature tensors aligned to each modality; S3: Based on the statistical distribution characteristics of typical abnormal modes in the historical fault case library, construct the causal sensitivity initialization weight vector, in which the initial causal weight coefficients are configured for different sensor modes respectively; S4: Input the standardized feature tensor and the causal sensitivity initialization weight vector into the dynamic weighted fusion network, calculate the real-time confidence index of each modality at the current time, and generate a time-varying fusion weight matrix by combining the initial causal weight coefficients, and output the fused multimodal joint representation vector. S5: Input the multimodal joint representation vector into the dual-path inference network, wherein the main path generates anomaly scores through a convolution classifier, and the auxiliary path synchronously performs attention gradient backtracking operation to extract the response intensity trajectories of key feature channels and construct a structured attribution graph matrix. S6: Based on the abnormal score value, determine whether it exceeds the adaptive threshold range. If it does, trigger the abnormal event marking and map the attribution graph matrix to the predefined semantic template library to generate structured explanation labels.

2. The intelligent inspection method for port equipment based on multi-source data according to claim 1, characterized in that, Following step S6, the following is also included: S7: Package the structured interpretation labels and raw observation data fragments and upload them to the task management terminal, and open the user feedback interface for operation and maintenance personnel to annotate the accuracy and true cause of the identification results, forming a feedback sample set with manual annotation; S8: The causal sensitivity initialization weight vector is back-modulated and updated using the feedback sample set, wherein the prior weight coefficients are adjusted according to the actual contribution deviation of each mode in the false alarm or missed alarm samples.

3. The intelligent inspection method for port equipment based on multi-source data according to claim 1, characterized in that, Step S1 specifically includes: Based on the infrared thermal imager, three-axis vibration sensor, visible light camera and operating condition parameter acquisition unit carried by the UAV, a multi-source heterogeneous data synchronous acquisition operation is performed to obtain raw sensor data streams including infrared thermal imaging sequences, three-axis vibration signals, visible light image frames and time-series operating condition parameters. Each data item in the original sensor data stream is appended with a spatial location tag provided by the UAV positioning system. The spatial location tag contains the geographic coordinates and relative pose information of the device component, generating a local observation data fragment with a spatial attribution identifier. A unified timestamp is applied to all sensor channels based on a high-precision time synchronization module to generate time-series data units with time synchronization attributes; The spatiotemporal registration algorithm is used to perform coarse alignment processing on the modal data with spatial location labels and timestamps. The infrared thermal imaging sequence, triaxial vibration signal, visible light image frame and time series operating condition parameter collected within the same time window are matched at the frame level to generate spatiotemporal association group. The spatiotemporal correlation groups are integrated into a data set in a structured storage format to generate a raw observation dataset with spatiotemporal alignment markers.

4. The intelligent inspection method for port equipment based on multi-source data according to claim 3, characterized in that, The multi-source sensor data acquisition adopts a multi-source heterogeneous sensor synchronous acquisition method. The trigger frequency of the data sampling synchronization signal for each mode is 1-500Hz, and the hardware clock synchronization accuracy is less than 1ms.

5. The intelligent inspection method for port equipment based on multi-source data according to claim 1, characterized in that, Step S2 specifically includes: Non-uniformity correction processing is performed on the infrared thermal imaging sequence. Based on the non-uniformity model of blackbody reference source calibration data and pixel response, the fixed pattern noise caused by manufacturing process differences in the focal plane array is compensated to generate a temperature distribution image sequence after spatial noise suppression. The temperature distribution image sequence is normalized by temperature field. Based on the steady-state temperature statistical characteristics of typical areas on the device surface, the original temperature values ​​are mapped to the standard temperature range. Histogram equalization algorithm is used to enhance the contrast of the thermal map and obtain a normalized thermal feature map set. Bandpass filtering is applied to the triaxial vibration signal. The filtering frequency band is set based on the natural frequency range of the main transmission components of the port equipment. Rigid body motion interference below a specific low-frequency threshold and high-frequency noise components above a specific high-frequency threshold are filtered out to extract the effective vibration frequency band signal of key component status information. The effective vibration frequency band signal is subjected to short-time Fourier transform processing. The time-series signal is segmented using the Hanning window function and its time-spectrum matrix is ​​calculated. The amplitude-frequency-time three-dimensional joint features are extracted to generate a short-time Fourier feature tensor. Visible light image frames are subjected to joint processing of dehazing enhancement and edge sharpening. The scene transmittance is estimated based on the atmospheric scattering model and the haze-free image is restored. Then, the Laplacian operator combined with the adaptive gain control mechanism is applied to enhance the edge details of the image and generate an enhanced set of visible light image frames.

6. The intelligent inspection method for port equipment based on multi-source data according to claim 1, characterized in that, Step S3 specifically includes: Obtain multi-source sensor data records and corresponding fault type labels from the historical fault case library to form a structured historical fault sample set; Based on the structured historical fault sample set, modal contribution analysis is performed for each type of fault. The mutual information calculation method is used to evaluate the nonlinear correlation strength between each sensor mode and the fault label during the fault occurrence period, and the original discrimination contribution score of each mode under each type of fault is obtained. The original discrimination contribution score is normalized and then weighted and corrected in combination with the experience rules of port equipment maintenance experts to generate a standardized modal contribution factor matrix. Based on the standardized modal contribution factor matrix, the weighted average contribution value of each sensor mode on all fault types is calculated. The weight distribution is optimized using the entropy weight method to suppress the tendency of overconfidence under high uncertainty. The global causal sensitivity benchmark value of each mode is output, which constitutes the basic components of the causal sensitivity initialization weight vector. The causal sensitivity initialization weight vector is mapped to the initial connection weight layer of the dynamic weighted fusion network in modal order, serving as a priori guiding parameter for multimodal joint representation learning.

7. The intelligent inspection method for port equipment based on multi-source data according to claim 6, characterized in that, In the standardized modal contribution factor matrix, each row corresponds to a typical fault type, each column corresponds to a sensor mode, and the matrix elements represent the prior discriminative influence of the mode on the fault type.

8. The intelligent inspection method for port equipment based on multi-source data according to claim 1, characterized in that, Step S4 specifically includes: The constructed causal sensitivity initialization weight vector serves as the basis for initial parameter configuration of the dynamic weighted fusion network. A modal importance bias driven by historical fault knowledge is introduced to generate the initial weighted basis. Local quality assessments are performed on the normalized feature tensors after each modality alignment. Based on the signal-to-noise ratio, gradient change entropy, and time-domain stability index of the data sequence within the sliding window, the real-time confidence index of each sensor modality at the current moment is calculated to obtain the real-time confidence vector. The real-time confidence vector and the causal sensitivity initialization weight vector are nonlinearly coupled and fused using a Sigmoid weighted gating mechanism to output the dynamically modulated intermediate mode weight distribution. Based on the intermediate mode weight distribution, a normalized time-varying fusion weight matrix is ​​generated through Softmax normalization. The time-varying fusion weight matrix is ​​used to perform a weighted summation operation on the standardized feature tensor. Each modal feature tensor is multiplied by its corresponding weight coefficient and then superimposed element by element to generate a multimodal joint representation vector.

9. The intelligent inspection method for port equipment based on multi-source data according to claim 8, characterized in that, Among the real-time confidence indices, the confidence of infrared thermal imaging sequences is determined by the spatial continuity of the temperature field, the confidence of triaxial vibration signals is measured by the frequency domain energy concentration, the confidence of visible light image frames is evaluated by the edge structure sharpness, and the confidence of time-series operating parameters is jointly determined by the sampling integrity and trend smoothness.

10. The intelligent inspection method for port equipment based on multi-source data according to claim 1, characterized in that, Step S5 specifically includes: Based on the multimodal joint representation vector, it is input into the main path of the dual-path inference network. A one-dimensional convolutional neural network is used to perform a local receptive domain sliding convolution operation on the multimodal joint representation vector to extract cross-modal temporal dependent features and generate a high-order latent feature mapping tensor. Global average pooling is performed on the higher-order latent feature mapping tensor to obtain a compact feature vector, which is then input into a fully connected classification head. The anomaly probability distribution is calculated based on the Softmax activation function, and an anomaly score value between 0 and 1 is output. The multimodal joint representation vector is synchronously input into the auxiliary path of the dual-path inference network to construct an attention mechanism layer. Based on the self-attention weight matrix, the response intensity of each sensor modality in the time-channel dimension is weighted and mapped to generate the original attention heatmap. Gradient backtracking analysis is performed on the original attention heatmap. The gradient magnitude of the abnormal score value relative to each input feature channel is calculated using the backpropagation algorithm. Key feature channels with significant contributions and their corresponding time segments are extracted to form a gradient-sensitive region mask. Based on the gradient-sensitive region mask, the original features in the multimodal joint representation vector are selectively reconstructed to generate a structured attribution map matrix.

Images