Proportional servo valve intelligent diagnosis method based on multi-modal feature fusion

Abstract

The invention discloses a proportional servo valve intelligent diagnosis method based on multi-modal feature fusion, and the method comprises the steps: collecting a multi-modal time domain signal through a plurality of sensors according to the superposition and hidden characteristics of a fault of a proportional servo valve; the time domain signal is converted into a GASF / GADF image through GAF transformation, and time-frequency coupling features are enhanced; constructing a time domain feature extraction branch and an image feature extraction branch to realize double-domain feature complementation; designing a cross-dimensional attention fusion-multi-modal weighted fusion layering strategy, generating multi-dimensional fusion features, and inputting the multi-dimensional fusion features into a classifier to complete 16 types of working condition recognition; and finally, retrieving the hidden fault knowledge base, and outputting fault location, risk levels and a stepped maintenance method. Weak fault signals with the amplitude smaller than 5% can be captured, the average recognition precision of 16 kinds of working conditions reaches 97.5% or above, the anti-interference capacity is high, diagnosis-maintenance integration is achieved, and the method is suitable for operation and maintenance of hydraulic systems in the fields of engineering machinery, aerospace and the like.

Description

Technical Field

[0001] This invention relates to the field of hydraulic transmission control and fault diagnosis technology, specifically to an intelligent diagnostic method for proportional servo valves based on multimodal feature fusion. Background Technology

[0002] Electro-hydraulic proportional servo valves, as core actuators in high-end hydraulic control systems, serve as the "nerve center" for critical scenarios such as aerospace servo actuation, temperature control in precision manufacturing equipment, and flow regulation in energy equipment. Through precise conversion of electrical signals to hydraulic energy, they achieve millisecond-level response speeds and micron-level control accuracy, directly determining the operational stability and precision of the equipment. However, these components integrate multiple disciplines, including electromagnetic drives, precision mechanical spool valve pairs, and embedded drive circuits. Their internal structure is highly coupled, and they operate under harsh conditions of high pressure (up to 35MPa), wide temperature range (-40℃~120℃), and strong vibration (acceleration up to 10g), leading to frequent failures such as spool valve core jamming, spool valve pair wear, inter-turn short circuits in the proportional electromagnet, and parameter drift in the drive circuit components. According to industrial statistics, servo valve failures account for more than 35% of all hydraulic system failures, making them a core bottleneck restricting equipment reliability.

[0003] During the entire lifecycle of a proportional servo valve, fault evolution exhibits a three-tiered characteristic of "latent-accumulation-outburst": early minor anomalies (such as slight wear of the valve core or a slight decrease in the permeability of the electromagnet) do not directly cause system performance inaccuracies, but only manifest as latent characteristics such as increased dynamic response lag and amplified control signal fluctuations. If such "latent faults" are not identified in time, under long-term alternating loads and environmental stresses, they can easily evolve into fatal faults such as valve core jamming and coil burnout, causing major accidents such as attitude control failure in aerospace equipment and scrapping of precision rolling mill products. According to data from a heavy industry company, the downtime losses caused by sudden servo valve failures can reach hundreds of thousands of yuan per hour. Therefore, achieving early and highly sensitive identification of latent faults has become a core technical challenge in the health management system of intelligent hydraulic equipment.

[0004] Currently, proportional servo valve fault diagnosis technology mainly follows two major technical routes: The first is a model-based diagnostic method, which establishes a coupled mathematical model of the servo valve's flow-pressure-displacement relationship and uses residual analysis and Kalman filtering to identify parameter deviations. However, the servo valve contains complex factors such as hysteresis, nonlinear hydraulic forces, and time-varying oil viscosity, making accurate modeling extremely difficult, and the diagnostic accuracy in practical engineering applications is less than 70%. The second is a data-driven diagnostic method, which utilizes multi-source sensor signals such as pressure, flow, current, and vibration, and uses deep learning algorithms to automatically extract and classify fault features, which has become the mainstream research approach. For example, schemes such as using attention convolutional capsule networks to enhance the correlation features of current-displacement signals, using multimodal deep residual contraction networks to achieve vibration-pressure signal noise reduction and fusion, and using grasshopper optimization algorithms to optimize support vector machines to improve pressure signal classification accuracy have all achieved high diagnostic performance in specific scenarios.

[0005] Despite significant progress in existing data-driven methods, three major technical shortcomings remain for diagnosing latent faults:

[0006] 1. Insufficient sensitivity to early features: Existing models are mostly designed for obvious faults such as valve core jamming and coil breakage, and have limited ability to extract features from early weak degradation signals (such as valve core wear <0.02mm). Experimental data shows that the recognition rate for wear-related latent faults is only 58%.

[0007] 2. Insufficient utilization of modal information: Feature extraction often relies on single modal signals (such as using only time-domain current signals or frequency-domain vibration signals), which makes it difficult to comprehensively characterize the dynamic behavior of the servo valve's multi-field coupling of "electromagnetic-mechanical-hydraulic", resulting in the lack of fault feature dimensions;

[0008] 3. Poor robustness to strong noise: In industrial settings, noise such as vibration and electromagnetic interference can easily mask local anomalies in time-domain signals. For example, when the signal-to-noise ratio is below 5dB, the diagnostic accuracy of traditional convolutional networks can drop by more than 30%.

[0009] To overcome the aforementioned technical bottlenecks, it is urgent to construct a multimodal collaborative feature mining framework. On the one hand, time-domain signals such as pressure, flow, and vibration are converted into image formats such as GAF (Gramian Angular Field) and MTF (Markov Transition Field) to highlight time-frequency domain correlation features and fully leverage the spatial feature learning advantages of two-dimensional convolutional networks. On the other hand, through cross-modal attention fusion, residual shrinkage and noise reduction techniques, deep collaboration of time-domain, frequency-domain, and image-domain features is achieved, enhancing the separability of latent faults in the feature space. Based on this, this invention proposes an intelligent diagnostic method for proportional servo valves based on cross-dimensional multimodal feature fusion. By collaborative modeling of multi-source multimodal signals and multi-dimensional feature fusion, it achieves early and accurate identification of latent faults, providing technical support for the full life-cycle health management of servo valves. Summary of the Invention

[0010] The purpose of this invention is to address the shortcomings of existing technologies by proposing a method for diagnosing latent faults in proportional servo valves based on Gramian Angular Field (GAF) transform, cross-dimensional feature extraction, and multimodal fusion. This innovative method converts time-domain signals into image-domain signals and combines multi-dimensional feature extraction with a multimodal hierarchical fusion strategy. It is applicable to early latent fault detection, accurate identification, and maintenance decision support for proportional servo valves in fields such as engineering machinery, aerospace, and intelligent manufacturing, and has significant advantages, especially for latent fault scenarios with weak signals and coupled features. This invention aims to overcome the deficiencies of existing methods for diagnosing latent faults in proportional servo valves, including insufficient signal representation, unutilized image advantages, coarse fusion mechanisms, and a disconnect between fault and maintenance.

[0011] The objective of this invention is achieved through the following technical solution: a proportional servo valve intelligent diagnostic method based on multimodal feature fusion, the method comprising the following steps:

[0012] (1) Collect multimodal time-domain signals of the proportional servo valve under normal operating conditions, independent latent fault operating conditions and their combination under composite operating conditions;

[0013] (2) The time-domain signal acquired in step (1) is transformed into image data through GAF;

[0014] (3) After denoising the time domain signal, extract the time domain features. For image data, use a capsule neural network to extract image features.

[0015] (4) For the temporal and image features of the same data, a multi-head attention mechanism is used to learn the association weights of different feature subspaces in parallel, retaining the feature components that are sensitive to single or compound faults; then, based on the cross-dimensional features output by multi-head attention, weighted feature splicing is performed to achieve multimodal feature fusion and optimize modal weights.

[0016] (5) The probability distribution of the corresponding working condition is obtained by inputting the multimodal fusion feature classifier, thereby realizing fine-grained identification of latent faults;

[0017] (6) Based on the identification results of step (5), a knowledge base of latent faults is constructed to realize the automatic association between fault types and operation and maintenance information, and output maintenance solutions that can be directly implemented.

[0018] Furthermore, in step (1), the multimodal time-domain signals include time-domain signals of valve orifice pressure data, flow rate data, coil drive current data, valve body vibration acceleration data, and valve internal noise data.

[0019] Furthermore, in step (2), the specific steps are as follows:

[0020] (2.1) Normalize all time-domain signals in step (1);

[0021] (2.2) Map the normalized time-domain signal to a polar coordinate system represented by modulus and angle, where modulus represents the relative intensity of the amplitude of the time-domain signal and angle represents the temporal correlation of the time-domain signal;

[0022] (2.3) Based on the differences in characteristics of different modal signals, Gramian angle sum field GASF matrix and Gramian angle difference field GADF matrix are constructed differently and downsampled to obtain GAF image set; among them, the GASF matrix is ​​adapted to pressure / flow / current signals, and the cosine value of the sum of the angles of two sampling points is calculated to highlight the synergistic correlation of amplitude; the GADF matrix is ​​adapted to vibration / noise signals, and the sine value of the angle difference between two sampling points is calculated to highlight the abrupt change characteristics of amplitude.

[0023] Furthermore, in step (3), for the time domain signal, a 3-layer wavelet decomposition is used for denoising, and after denoising, a clean time domain signal is obtained by wavelet reconstruction; then, deep feature extraction is performed based on the 1DCNN network.

[0024] Furthermore, in step (3), for the image data, the local gray-scale abnormal texture corresponding to the latent fault in the GAF image is captured by the capsule neural network, and the association weight between the main capsule and the digital capsule is updated by the dynamic routing algorithm.

[0025] Further, in step (4), the temporal feature dimension of the image features is matched, and then the feature weight is calculated through multi-head attention. Specifically, the temporal features and the unified image features are linearly transformed to generate query Q, key K, and value V matrices. For each head, the mutual attention weight between the temporal features and the image features is calculated, and then multi-head splicing and linear transformation are performed to fuse them into cross-dimensional features.

[0026] Furthermore, in step (4), the modality weight vector is determined by statistically analyzing the contribution of each modality to the identification of a single fault or a compound fault through 5-fold cross-validation; and the final fusion feature is formed by weighted splicing of cross-dimensional fusion features to form full-modal-multi-subspace feature association information that includes multiple working conditions.

[0027] Furthermore, in step (6), a relational database is used to store the structured information of latent faults, and a two-stage retrieval strategy combining fault type and semantic retrieval is adopted based on the fault type to ensure accurate matching between compound faults and maintenance solutions.

[0028] Furthermore, the first stage of the two-stage retrieval strategy is precise retrieval, which directly searches the database by standard fault name based on the clear fault type and outputs the corresponding fault location, risk level and maintenance method.

[0029] Furthermore, the second stage of the two-stage retrieval strategy is semantic retrieval. For non-standard fault names and fault names in the knowledge base, fuzzy retrieval is achieved by using BERT semantic encoding combined with cosine similarity matching. Specifically, non-standard fault names and fault names in the knowledge base are encoded into text vectors through a BERT pre-trained model, and the semantic consistency of the vectors is measured by cosine similarity for matching.

[0030] The beneficial effects of this invention: Compared with the prior art, this invention has the following significant innovative advantages, especially in achieving breakthroughs in the sensitivity and accuracy of latent fault diagnosis:

[0031] 1. Significantly improved sensitivity to latent fault characteristics

[0032] By converting time-domain signals into images through GAF transformation, weak fault features are transformed into "visible image grayscale textures." Combined with CapsNet's spatial feature extraction capabilities, the sensitivity of fault feature capture is stronger than that of traditional time-domain methods, and early latent faults that traditional methods cannot detect can be identified.

[0033] 2. High fault diagnosis accuracy and strong anti-interference capability.

[0034] Dual-domain feature extraction (temporal domain + image) and a hierarchical fusion mechanism achieve complementary enhancement of heterogeneous features. Experimental verification is as follows: Figure 6 As shown (this method is abbreviated as GTMF), under normal operating conditions and 15 types of latent fault conditions, the average identification accuracy reaches over 97.5%; Figure 7 The proposed method is more than 15% better than current advanced methods.

[0035] 3. Reuse image algorithm advantages to lower technical barriers

[0036] Innovatively converting time-domain signals into GAF images allows for the direct reuse of mature image feature extraction algorithms in the field of deep learning (such as CapsNet and ResNet), eliminating the need to redesign complex networks for time-domain signals, shortening the algorithm development cycle, and reducing the difficulty of technology implementation.

[0037] 4. Integrated diagnosis and maintenance, highly practical for engineering applications.

[0038] By using a latent fault knowledge base, a closed loop of "fault identification → damaged part location → risk assessment → maintenance guidance" is achieved. The output maintenance methods are specific to the part model and operation steps, which can directly guide on-site maintenance personnel to work, thereby shortening the average fault handling time and avoiding the problems of "maintenance lag" or "over-maintenance".

[0039] 5. Wide applicability and strong scalability

[0040] This method does not rely on a specific model of proportional servo valve. It can be extended to the diagnosis of hidden faults in other hydraulic components such as hydraulic pumps, hydraulic cylinders, and electro-hydraulic proportional valves by simply adjusting the sensor placement and knowledge base parameters. Its application scope covers multiple fields such as engineering machinery, aerospace, and intelligent manufacturing. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a flowchart of an intelligent diagnostic method for latent faults in hydraulic valves based on multimodal feature fusion, provided by the present invention.

[0043] Figure 2 This is a schematic diagram of a cross-dimensional, multi-modal feature fusion fault diagnosis information processing structure.

[0044] Figure 3 This is a schematic diagram illustrating the principle and effect of GAF transformation.

[0045] Figure 4 This is a diagram of a dual-domain feature extraction network architecture.

[0046] Figure 5 This is a schematic diagram of a cross-dimensional multimodal fusion mechanism.

[0047] Figure 6 This is a diagram showing the results of the fault diagnosis confusion matrix.

[0048] Figure 7 This is a comparison chart of fault diagnosis accuracy. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described below with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are merely illustrative and not intended to limit the invention.

[0050] This invention provides an intelligent diagnostic method for proportional servo valves based on multimodal feature fusion. Based on a six-step core process of "multimodal data acquisition → GAF image conversion (enhancing time-frequency characteristics) → dual-domain feature extraction → cross-dimensional multimodal fusion → fault classification → knowledge base linkage," it achieves high-precision diagnosis of latent faults through an innovative logic of "signal dimensionality enhancement - dual-domain feature extraction - hierarchical feature fusion." The following core technologies are employed:

[0051] This invention constructs a mapping relationship between "time domain signal → image signal" through data processing methods, converting 1D time domain signals such as pressure / flow / vibration / noise into high-dimensional images, strengthening the time-frequency coupling characteristics of the signal, and providing adaptive input for high-dimensional feature extraction algorithms;

[0052] This invention designs a dual-path feature extraction architecture of "time domain + image domain", which retains dynamic response information from the original time domain signal and mines spatial correlation information from the GAF image, thereby realizing "dual-domain complementary capture" of weak features of latent faults.

[0053] This invention constructs a cross-dimensional multimodal hierarchical fusion mechanism, first solving the dimensional heterogeneity problem of "temporal features - image features", and then optimizing the weight allocation of "multi-sensor modalities" to improve the discriminativeness and anti-interference ability of the fused features;

[0054] This invention establishes a "hidden fault knowledge base" to automatically associate fault types with damaged parts, risk levels, and maintenance methods, forming a "diagnosis-maintenance" closed loop and improving the engineering applicability of diagnostic results.

[0055] like Figure 1 and Figure 2 As shown, the specific steps of the intelligent diagnostic method for proportional servo valves based on cross-dimensional multimodal feature fusion provided by this invention are as follows:

[0056] Step 1: Multimodal time-domain signal acquisition

[0057] For the normal operating condition (None fault) of the proportional servo valve, four independent typical latent fault conditions (fault1: slight oil contamination, fault2: minor wear of the spool valve pair, fault3: initial degradation of the electromagnet, fault4: zero bias of the drive circuit), and eleven composite faults (six dual faults, four triple faults, and one quad fault) composed of the four fault combinations, a multi-physics signal acquisition system was built to ensure comprehensive capture of fault-related time-domain information. The specific acquisition parameters and signal types are as follows:

[0058] Pressure signal: A piezoelectric pressure sensor (accuracy ±0.1%FS) is used to collect pressure data from port P (oil inlet), port A (working oil port 1), and port B (working oil port 2), with a sampling frequency of... Sampling duration The time-domain sequences corresponding to the three pressure data were obtained. ( (Number of sampling points)

[0059] Flow signal: A turbine flow meter (range 0~100L / min, accuracy ±0.5%) is used to collect flow data from port P. sampling frequency ;

[0060] Current signal: A Hall current sensor (range 0~5A, accuracy ±0.2%) is used to acquire coil drive current data. sampling frequency ;

[0061] Vibration signal: A piezoelectric accelerometer (range 0~500g, frequency response 0.1~10kHz) is used, attached to the top of the proportional servo valve body (directly above the valve core axis) to collect vibration acceleration data. Original sampling frequency The data dimension is unified by downsampling to 1kHz through linear interpolation;

[0062] Noise signal: An electret microphone (frequency response 20Hz~20kHz, sensitivity -40dB) was used, positioned on the side of the proportional servo valve body (5cm away from the valve body) to collect data on fluid noise and mechanical noise inside the valve. Original sampling frequency Downsampling to 1kHz.

[0063] Define a multimodal time-domain signal X 1D The set is:

[0064]

[0065] (Note: The sensors need to be calibrated before data acquisition. Pressure sensors are calibrated using a standard pressure source, and current sensors are calibrated using a standard current source to ensure data accuracy.)

[0066] Step 2: GAF Transformation

[0067] like Figure 3 As shown, this step is one of the core innovations: converting the 1D time-domain signal into a 2D image through GAF transformation, mapping the signal's "time-amplitude" information to the image's "spatial-grayscale" information, while preserving the signal's temporal correlation and time-frequency coupling characteristics, laying the foundation for subsequent image feature extraction algorithms. The specific process consists of three steps: "data normalization → polar coordinate transformation → GAF matrix construction."

[0068] Step 2.1: Data Normalization

[0069] Because the dimensions of different modal signals differ significantly (e.g., pressure is measured in MPa, and current in A), Min-Max normalization must first be used to map all time-domain signals to the [0, 1] interval to avoid interference from the dimensions in subsequent transformations. The normalization formula is as follows:

[0070]

[0071] in, For any original time-domain signal, Let x be the kth sampling point. The normalized value. , Let x be the minimum and maximum values, respectively. The normalized signal set is denoted as... .

[0072] Step 2.2: Polar coordinate transformation

[0073] The normalized time-domain signal Mapping to polar coordinates ,in:

[0074] Modulus r: Characterizes the relative intensity of the signal amplitude, directly taken as... (because The modulus range is consistent with the normalization result, ensuring that the amplitude information is not distorted.

[0075] Angle θ: Characterizes the temporal correlation of the signal, and transforms the sampling index k (time dimension) into a linear mapping. The angle of the interval is given by the formula:

[0076]

[0077] Step 2.3: GAF Matrix Construction

[0078] Based on the differences in characteristics of different modal signals, Gramian Angle Sum Field (GASF) and Gramian Angle Difference Field (GADF) matrices are constructed differently to maximize the preservation of the core time-frequency characteristics of various signals: GASF matrix (adapted to pressure / flow / current signals): Pressure, flow, and current signals have strong time-domain waveform smoothness, and fault characteristics are mainly manifested as "coordinated amplitude changes" (such as the synchronous fluctuation of pressure and flow when the valve core is stuck). The GASF matrix highlights the coordinated correlation of amplitudes by calculating the cosine value of the sum of the angles of two sampling points. The matrix elements are defined as:

[0079]

[0080] GADF Matrix (for vibration / noise signals): Vibration and noise signals contain high-frequency components, and fault characteristics are mainly manifested as "abrupt amplitude differences" (such as high-frequency pulses in vibration signals when seals are worn). The GADF matrix highlights the abrupt amplitude changes by calculating the sine value of the angle difference between two sampling points. The matrix elements are defined as follows:

[0081]

[0082] To reduce the computational load of subsequent image processing, the GASF / GADF matrix is ​​downsampled (using bilinear interpolation), reducing the original 6×10 matrix. 4 ×6×10 4 The matrix is ​​compressed into a 128×128 grayscale image (grayscale value range) (Mapped to pixel values ​​from 0 to 255). Define the GAF image set as:

[0083]

[0084] In the formula , , Gram angles and field images corresponding to the pressure signals from ports P, A, and B, respectively. These are the Gram angle and field image of the P-port flow signal. These are the Gram angle and field image of the coil current signal. and These are Gram angle difference field images of the valve body vibration signal and the valve internal noise signal, respectively.

[0085] Step 3: Dual-domain feature extraction

[0086] like Figure 4As shown, this step is the second core innovation: constructing a parallel architecture of "temporal feature extraction branch" and "image feature extraction branch," while simultaneously... and By extracting features, the "dynamic response information" and "spatial correlation information" can be complemented, significantly improving the sensitivity of latent fault features.

[0087] Step 3.1: Temporal Feature Extraction

[0088] To address the dynamic characteristics of time-domain signals, a combined approach of "wavelet transform denoising → 1DCNN feature extraction" is adopted. This first eliminates noise interference, and then extracts deep time-domain features.

[0089] Wavelet transform denoising: First, a db4 wavelet basis adapted to the characteristics of hydraulic signals is selected as the decomposition basis function. The normalized one-dimensional time-domain signal is subjected to the first-level wavelet decomposition to obtain the first-level approximation coefficients containing the main low-frequency features. Compared with the first layer detail factor containing high-frequency noise Then with Perform a second-level wavelet decomposition on the input to obtain the second-level approximation coefficients. With detail coefficient Finally, with The input is used to complete the third level wavelet decomposition, and the final output is the third level approximation coefficients. Soft thresholding is used to denoise the detail coefficients. The threshold calculation formula is:

[0090]

[0091] in, The standard deviation of noise (based on median estimation, which is more robust). This is a median function. After denoising, a clean time-domain signal is obtained through wavelet reconstruction.

[0092]

[0093] In the formula This represents the detail coefficient after noise reduction.

[0094] 1DCNN Feature Extraction: Constructing a Lightweight 1DCNN Network For deep feature extraction, the network structure is as follows:

[0095] Table 1 1DCNN Network Structure

[0096]

[0097] The mathematical expression for a convolutional layer is:

[0098]

[0099] in, This is the output of the l-th convolutional layer. This indicates the size of the convolution kernel used in the l-th convolution operation. The weights of the m-th convolutional kernel in the l-th layer are... For bias, This is the input for the (l-1)th layer.

[0100] Define the set of temporal features for all modalities as:

[0101]

[0102] in This represents the multimodal time-domain feature vector.

[0103] Step 3.2: Image Feature Extraction

[0104] To address the spatial characteristics of GAF images, a Capsule Neural Network (CapsNet) is employed for feature extraction. Compared to traditional CNNs, CapsNet uses "capsule vectors" to represent global information such as the "position, pose, and scale" of features, accurately capturing the "local gray-level anomaly texture" corresponding to latent faults in GAF images, thus avoiding the limitation of traditional CNNs in losing spatial topological relationships. The specific network structure is as follows:

[0105] Table 2 CapsNet Network Architecture (Single GAF Image Input)

[0106]

[0107] Core activation function and routing algorithm: Squash function: used for capsule activation, enhancing high-confidence features while suppressing low-confidence noise. The formula is:

[0108]

[0109] Where v is the capsule output vector, and ||v|| is the vector magnitude (the larger the magnitude, the higher the feature confidence).

[0110] Dynamic routing algorithm: used to update the association weights between the main capsule and the digital capsule to ensure that fault characteristics are accurately transmitted. The steps are as follows:

[0111] ① Initialize routing coefficients (i is the primary capsule index, j is the numeric capsule index);

[0112] ② Calculate the prediction vector ( It is an 8×16 transformation matrix. (Main capsule output);

[0113] ③ Normalized routing coefficients: (Softmax normalization);

[0114] ④ Update capsule output: ;

[0115] ⑤ Update routing coefficients: ;

[0116] ⑥ Repeat steps ② to ⑤, iterate 3 times, and then output the digital capsule vector.

[0117] Multi-image parallel processing logic: Features are independently extracted from each of the 7 GAF images using the above architecture, ultimately outputting 7 sets of features. Image feature vectors:

[0118]

[0119] in This is a single-modal image feature vector (16-dimensional confidence features).

[0120] Step 4: Cross-dimensional multimodal feature fusion

[0121] like Figure 5 As shown, this step is the third core innovation: In response to the dimensional heterogeneity of "temporal features (256 dimensions) - image features (16 dimensions)" and the feature complexity of 16 types of working conditions (including single faults and compound faults), a multi-head attention mechanism is used to learn feature associations in parallel across multiple subspaces, thereby improving the ability to capture fine-grained features of compound faults. A hierarchical strategy of "cross-dimensional multi-head fusion → multi-modal weighted fusion" is designed to achieve feature complementarity enhancement.

[0122] Step 4.1 Cross-dimensional integration

[0123] Temporal characteristics of the same sensor Image features A multi-head attention mechanism is employed to learn the association weights of different feature subspaces in parallel, prioritizing the retention of feature components sensitive to single / compound faults (such as the coupling features of oil temporal fluctuations and valve image texture in "oil contamination + valve wear"). Specific steps include:

[0124] Dimensional unification:

[0125] Image features (16-dimensional) is mapped to 256-dimensional through a fully connected layer to ensure matching with the temporal feature dimension, resulting in The mapping formula is:

[0126]

[0127] in For mapping weight matrix (adapted to 16-dimensional image features). This is used as a bias (by training to optimize and enhance the feature mapping of complex faults).

[0128] Multi-head attention weight calculation:

[0129] Introducing 8-head attention (number of heads h=8, feature dimension per head) ,satisfy The attention weights of eight feature subspaces are learned in parallel to capture feature associations in different dimensions. Specific steps include:

[0130] Linear projection: for time-domain features With unified image features Perform linear transformations to generate query (Q), key (K), and value (V) matrices:

[0131]

[0132]

[0133] in and This is the projected weight matrix (optimized through training).

[0134] Multi-head split: and Divide the matrix into 8 sub-matrices, each with a dimension of 1×32, i.e.:

[0135]

[0136] Single-head attention calculation: The i-th head calculates the mutual attention weights between temporal features and image features, using the following formula:

[0137]

[0138] in This is a scaling factor (to avoid excessively large weight values). The attention output for the i-th head (capturing the feature associations of the i-th subspace).

[0139] Multi-head concatenation and linear transformation: The outputs of 8 heads are concatenated into 256-dimensional features, and then fused into cross-dimensional features through a linear layer.

[0140]

[0141]

[0142] in This is a multi-head fusion weight matrix. For bias (optimized through training to enhance the complementarity of the 8-head features).

[0143] Cross-dimensional fusion feature set: Define the cross-dimensional fusion feature set of the 7 sensors as follows:

[0144]

[0145] in It is a single-modal cross-dimensional fusion feature vector (containing feature association information of 8 subspaces, which can distinguish between "oil contamination" and "oil contamination + coil aging" and other composite faults).

[0146] Step 4.2 Multimodal Fusion

[0147] Based on the cross-dimensional features of multi-head attention output, weighted feature concatenation is used to achieve the fusion of 7-channel sensor modes. To address the differences in feature sensitivity across 16 operating conditions, the mode weights are optimized through "5-fold cross-validation + fault type contribution analysis." Specific steps are as follows:

[0148] Modal weights determination:

[0149] Based on sample data from 16 operating conditions, the contribution of each mode to the identification of single / compound faults was statistically analyzed using 5-fold cross-validation (e.g., vibration mode has a high contribution to "slide valve wear + oil contamination", and current mode has a high contribution to "coil aging + circuit drift"). The modal weight vector was then determined. (Corresponding to 7 sensors), meeting the requirements The example optimization result is as follows:

[0150] (P-port pressure) (Pressure at Port A) (Pressure at Port B) (P port flow) (Coil current) (vibration), (noise).

[0151] Multimodal fusion features:

[0152] The final fused feature is formed by weighted splicing of 7 cross-dimensional features:

[0153]

[0154] in It contains full-modal-multi-subspace feature association information for 16 types of operating conditions, which can accurately distinguish between single faults such as "oil contamination" and compound faults such as "oil contamination + valve wear".

[0155] Step 5: Fault Classification

[0156] Final fusion features Input a Softmax classifier and output the probability distribution of 16 operating conditions (15 fault conditions + 1 normal condition) to achieve fine-grained identification of latent faults. Specific steps: Classifier prediction: The Softmax classifier outputs the probabilities of the 16 operating conditions, using the following formula:

[0157]

[0158] Where y=k indicates that the k-th type of operating condition k=0 is normal, and k=1~F correspond to 15 types of faults, such as slight valve core jamming, moderate valve core jamming, slight coil aging, severe coil aging, minor wear of seals, and aging of seals, etc. This is the classification weight matrix for the k-th type of work condition (learned through training using feature discrimination patterns for 16 types of work conditions). For bias. Fault type determination: The operating condition with the highest probability is taken as the final diagnostic result, and the formula is:

[0159]

[0160] To verify the diagnostic performance of the aforementioned cross-dimensional multimodal feature fusion model (GTMF), a comprehensive evaluation of the model was conducted based on test set data. For example... Figure 6 The figure shows the fault diagnosis confusion matrix for this embodiment under 16 operating conditions (including 1 normal operating condition, 4 single faults, and 11 compound faults). As can be seen from the figure, thanks to the enhancement of time-frequency features by the GAF transformation in Step 2 and the feature complementarity of cross-dimensional attention in Step 4, the predicted labels of the vast majority of test samples completely overlap with the true labels (distributed along the diagonal), and the model achieves an average diagnostic accuracy of 97.5% for various latent faults. Especially in the identification of strongly coupled compound faults such as 'oil contamination + valve wear', no obvious misclassification phenomenon was observed, verifying the high discriminative power of the fused features.

[0161] Furthermore, to highlight the advanced nature of the GTMF method proposed in this invention, it was compared with five mainstream deep learning diagnostic algorithms: ResNet-20, LeNet-5, Visual Geometry Group + Long Short-Term Memory (VGG+LSTM), VGG-16, and Inception-V4. Figure 7 As shown, the scores of various methods on four core performance indicators—accuracy, recall, precision, and F1 score—are compared. Existing comparative methods are limited by single-modal input or insufficient feature extraction capabilities, with their scores mainly concentrated in the 65%–80% range, making it difficult to meet the requirements of high-reliability diagnosis. In contrast, the GTMF method proposed in this invention demonstrates significant advantages in all four indicators, with scores consistently above 97%. This fully demonstrates that by constructing a dual-branch feature extraction and hierarchical fusion mechanism in the 'time domain + image domain', it is possible to effectively overcome strong noise and weak signal interference, significantly improving the diagnostic efficiency of latent faults in proportional servo valves.

[0162] Step 6: Linking the Hidden Fault Knowledge Base

[0163] Step 6.1: Construct a "Hidden Fault Knowledge Base"

[0164] Build a "latent fault knowledge base" to automatically associate fault types with maintenance information and output readily implementable maintenance solutions. Specific steps include: Knowledge base construction: Use a relational database (such as MySQL) to store structured information about latent faults. The data structure is defined as follows:

[0165] Table 3. Correspondence between knowledge bases for latent faults

[0166]

[0167] Step 6.2 Knowledge Base Retrieval

[0168] For the fault types output in Step 5 (0-F) Employs a "two-stage retrieval strategy" to ensure accurate matching of complex faults with maintenance solutions. Specific steps include:

[0169] Step 6.2.1 First-level precise search

[0170] like For a specific fault type ID (such as For cases involving "mild oil contamination and sticking + slight wear of the valve assembly", the system directly retrieves the corresponding "fault location", "risk level", and "maintenance method" from the database using the "fault type ID". Its core advantage is its millisecond-level precise matching for standard single and compound faults, requiring no additional semantic calculations.

[0171] Step 6.2.2 Secondary semantic retrieval

[0172] If the fault name is a non-standard expression, fuzzy retrieval is achieved using "BERT semantic encoding + cosine similarity matching":

[0173] Text Encoding: Non-standard fault names (such as "oil contamination + valve wear + coil aging") and 16 types of fault names in the knowledge base are both encoded into 768-dimensional text vectors using a BERT pre-trained model, denoted as follows: (The vector to be retrieved) and (Knowledge base vector, );

[0174] Similarity calculation: Cosine similarity is used to measure the semantic consistency of vectors. The formula is:

[0175]

[0176] Results filtering: Calculate similarity The entries (adapting to the semantic redundancy of compound faults) are selected based on the risk level. If multiple matching entries exist, the entry with the highest risk level is selected first.

[0177] The above embodiments are used to explain and illustrate the present invention, but not to limit the present invention. Any modifications and changes made to the present invention within the spirit and scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A method for intelligent diagnosis of proportional servo valves based on multimodal feature fusion, characterized in that, The method includes the following steps: (1) Collect multimodal time-domain signals of the proportional servo valve under normal operating conditions, independent latent fault operating conditions and their combination under composite operating conditions; (2) The time-domain signal acquired in step (1) is transformed into image data through GAF; (3) After denoising the time domain signal, extract the time domain features. For image data, use a capsule neural network to extract image features. (4) For the temporal and image features of the same data, a multi-head attention mechanism is used to learn the association weights of different feature subspaces in parallel, retaining the feature components that are sensitive to single or compound faults; then, based on the cross-dimensional features output by multi-head attention, weighted feature splicing is performed to achieve multimodal feature fusion and optimize modal weights. (5) The probability distribution of the corresponding working condition is obtained by inputting the multimodal fusion feature classifier, thereby realizing fine-grained identification of latent faults; (6) Based on the identification results of step (5), a knowledge base of latent faults is constructed to realize the automatic association between fault types and operation and maintenance information, and output maintenance solutions that can be directly implemented.

2. The intelligent diagnostic method for a proportional servo valve based on multimodal feature fusion according to claim 1, characterized in that, In step (1), the multimodal time-domain signals include time-domain signals of valve port pressure data, flow data, coil drive current data, valve body vibration acceleration data, and valve internal noise data.

3. The intelligent diagnostic method for a proportional servo valve based on multimodal feature fusion according to claim 1, characterized in that, In step (2), the specific steps are as follows: (2.1) Normalize all time-domain signals in step (1); (2.2) Map the normalized time-domain signal to a polar coordinate system represented by modulus and angle, where modulus represents the relative intensity of the amplitude of the time-domain signal and angle represents the temporal correlation of the time-domain signal; (2.3) Based on the differences in characteristics of different modal signals, Gramian angle sum field GASF matrix and Gramian angle difference field GADF matrix are constructed differently and downsampled to obtain GAF image set; among them, the GASF matrix is ​​adapted to pressure / flow / current signals, and the cosine value of the sum of the angles of two sampling points is calculated to highlight the synergistic correlation of amplitude; the GADF matrix is ​​adapted to vibration / noise signals, and the sine value of the angle difference between two sampling points is calculated to highlight the abrupt change characteristics of amplitude.

4. The intelligent diagnostic method for a proportional servo valve based on multimodal feature fusion according to claim 1, characterized in that, In step (3), for the time domain signal, a 3-layer wavelet decomposition is used for denoising, and after denoising, a clean time domain signal is obtained by wavelet reconstruction; then, deep feature extraction is performed based on a 1DCNN network.

5. The intelligent diagnostic method for a proportional servo valve based on multimodal feature fusion according to claim 1, characterized in that, In step (3), for image data, the local gray-scale abnormal texture corresponding to latent faults in GAF images is captured by capsule neural network, and the association weight between the main capsule and digital capsule is updated by dynamic routing algorithm.

6. The intelligent diagnostic method for a proportional servo valve based on multimodal feature fusion according to claim 1, characterized in that, In step (4), the temporal feature dimension of the image features is matched, and then the feature weight is calculated through multi-head attention. Specifically, the temporal features and the unified image features are linearly transformed to generate query Q, key K, and value V matrices. For each head, the mutual attention weight between the temporal features and the image features is calculated, and then multi-head splicing and linear transformation are performed to fuse them into cross-dimensional features.

7. The intelligent diagnostic method for a proportional servo valve based on multimodal feature fusion according to claim 1, characterized in that, In step (4), the contribution of each mode to the identification of a single fault or a compound fault is statistically analyzed by 5-fold cross-validation to determine the modal weight vector; and the cross-dimensional fusion features are weighted and spliced ​​to form the final fusion feature of full-modal-multi-subspace feature association information covering multiple working conditions.

8. The intelligent diagnostic method for a proportional servo valve based on multimodal feature fusion according to claim 1, characterized in that, In step (6), a relational database is used to store the structured information of latent faults. A two-stage retrieval strategy combining fault type and semantic retrieval is adopted based on the fault type to ensure accurate matching between compound faults and maintenance solutions.

9. The intelligent diagnostic method for a proportional servo valve based on multimodal feature fusion according to claim 8, characterized in that, The first stage of the two-stage retrieval strategy is precise retrieval, which directly searches the database by standard fault name based on the clear fault type and outputs the corresponding fault location, risk level and maintenance method.

10. The intelligent diagnostic method for a proportional servo valve based on multimodal feature fusion according to claim 8, characterized in that, The second stage of the two-stage retrieval strategy is semantic retrieval. For non-standard fault names and fault names in the knowledge base, fuzzy retrieval is achieved by using BERT semantic encoding combined with cosine similarity matching. Specifically, non-standard fault names and fault names in the knowledge base are encoded into text vectors by a BERT pre-trained model, and the semantic consistency of the vectors is measured by cosine similarity for matching.

Images