An electronic and electrical product automatic detection system based on multi-modal sensor fusion

Abstract

The application discloses a kind of electronic and electrical product automation detection systems based on multi-modal sensing fusion, it is related to electronic and electrical product detection technical field, solve the technical problem that existing detection scheme waveform separation is not accurate, fault location is not clear, corrosion prediction is not accurate, life evaluation is unreliable, cannot satisfy the demand of high-speed accurate detection of industrial production line;Through the improved K-SVD algorithm, the prior information of load type is introduced, the sparsity of dictionary atom is constrained, so that the dictionary atom is more suitable for the actual load characteristics.Through the sparsity constraint, noise interference can be suppressed, and the key features of the load are highlighted, providing a more reliable data basis for subsequent fault diagnosis or type identification.By aligning the separated independent load waveform with the device operating state captured by the visual sensor in time and space, single sensor misjudgment is eliminated;By introducing the entropy weight method to dynamically adjust the weight of parameters such as corrosion current density and salt fog concentration, the model is more suitable for complex environment.

Description

Technical Field

[0001] This invention belongs to the field of electronic and electrical product testing, specifically an automated testing system for electronic and electrical products based on multimodal sensor fusion. Background Technology

[0002] Multimodal sensor fusion refers to the integration of data from multiple different types of sensors, using algorithms to achieve cross-modal information complementarity and enhancement, thereby improving the accuracy, robustness, and environmental adaptability of the sensing system. A single sensor may fail to detect defects due to lighting, occlusion, or noise. By combining visual, infrared, and acoustic data, multimodal fusion can comprehensively capture product defect characteristics. For example, in motor bearing inspection, visual identification identifies surface scratches, infrared detection detects localized high temperatures, and acoustic detection locates abnormal friction sounds; the complementarity of these three methods can prevent missed detections. Multimodal sensor fusion is also used in the automated inspection of electronic and electrical products. By combining machinery, sensors, and algorithms with multimodal sensor fusion technology, it replaces manual labor in inspecting product performance, function, and appearance, achieving quality control throughout the production process.

[0003] Most existing testing solutions for electronic and electrical products rely on a single waveform separation algorithm that depends solely on time-domain / frequency-domain features. This fails to verify whether the separated waveform corresponds to the actual operating electronic or electrical appliance. For example, it may misinterpret an air conditioner compressor waveform as a refrigerator motor waveform, or omit waveforms of low-power standby devices; the number of separated loads may not match the actual number of operating devices, resulting in inflated loads or missed loads; fault monitoring and classification may completely fail if the separated waveform does not match the actual equipment. Fault parameters extracted based on waveform features will lose their physical meaning: waveforms of normal equipment may be misjudged as faulty waveforms; the actual faulty device cannot be located, only a circuit fault can be detected, but it cannot be associated with a specific appliance ID, making fault tracing impossible.

[0004] Meanwhile, in existing technologies, corrosion rate calculations often employ single-parameter or equal-weighted multi-parameter superposition methods, without introducing an improved grey relational analysis algorithm to determine parameter weights. This leads to parameter weight imbalance and large deviations in corrosion rate prediction. Corrosion of metal components is the result of the coupled effects of multiple factors such as corrosion current density, corrosion potential, salt spray concentration, temperature, and humidity. The degree of influence of each parameter varies significantly under different environments. Furthermore, the corrosion model cannot be modified in conjunction with fault conditions, resulting in unrealistic life predictions. Existing technologies do not incorporate fault classification results into corrosion rate modeling, while equipment failures accelerate metal corrosion. Summary of the Invention

[0005] This invention aims to solve at least one of the technical problems existing in the prior art. To this end, this invention proposes an automated testing system for electronic and electrical products based on multimodal sensor fusion, which solves the technical problems of existing testing schemes, such as inaccurate waveform separation, unclear fault location, inaccurate corrosion prediction, and unreliable life assessment. These schemes cannot meet the needs of high-speed and accurate testing in industrial production lines, nor can they achieve reliability management of electronic and electrical products throughout their entire life cycle.

[0006] To address the aforementioned problems, a first aspect of the present invention provides an automated inspection system for electronic and electrical products based on multimodal sensor fusion, comprising:

[0007] Atomic Library Building Module: Constructs a load waveform atomic library, optimizes the load waveform atomic library by improving the K-SVD algorithm, uses standard waveform atoms of electronic and electrical products as dictionary atoms, introduces prior information of load type, and constrains the sparsity of dictionary atoms;

[0008] Waveform analysis module: The sparse coefficients are solved by the OMP orthogonal matching pursuit algorithm, and the number of iterations is set. In each iteration, the dictionary atom that best matches the residual is selected, the sparse coefficients and residuals are updated, and each independent load waveform is reconstructed by the sparse coefficients and the load waveform atom library.

[0009] Fault Classification Module: The separated independent load waveforms are verified together with visual sensor data and electrical parameters to ensure that the separated waveforms match the actual operating electronic and electrical products. The extracted independent load waveform feature data is used for fault monitoring and fault classification of electronic and electrical products.

[0010] Remaining service life prediction module: Based on corrosion current density, corrosion potential, salt spray concentration, temperature and humidity parameters, and fault classification results, a corrosion rate prediction model is constructed, and an improved grey relational analysis algorithm is used to determine the weight of each parameter; based on corrosion rate and metal component thickness, combined with the Arrhenius equation to correct the accelerating effect of temperature on corrosion, the remaining service life is predicted.

[0011] Optionally, in one example of the above aspects, the atom library building module constructs a load waveform atom library, using standard waveform atoms of electronic and electrical products as dictionary atoms, including the following steps:

[0012] Waveform signals of different load types are collected from electronic and electrical products, and the single-channel mixed waveforms are represented as a linear superposition of the independent load waveforms.

[0013] Based on the collected single-channel mixed waveform, solve for the independent load waveform of the electronic and electrical products;

[0014] The independent load waveforms of historical data of electronic and electrical products are preset as a priori load types. Combined with the independent load waveforms collected in real time from electronic and electrical products, a load waveform atom library is constructed, and the standard waveform atoms of electronic and electrical products are used as dictionary atoms.

[0015] Optionally, in one example of the above aspects, the load waveform atom library is optimized by improving the K-SVD algorithm, introducing prior information on load types, and constraining the sparsity of dictionary atoms, including the following steps:

[0016] The load waveform atom library is optimized by improving the K-SVD algorithm, extracting feature data of prior load types, constraining the sparsity of dictionary atoms, and setting an optimization objective function.

[0017] Optionally, in one example of the above aspects, the sparse coefficients are solved using the OMP orthogonal matching pursuit algorithm, and the number of iterations is set. In each iteration, the dictionary atom that best matches the residual is selected, the sparse coefficients and residuals are updated, and each independent load waveform is reconstructed using the sparse coefficients and the load waveform atom library, including the following steps:

[0018] The sparse coefficients are solved by the OMP orthogonal matching pursuit algorithm. The number of iterations is set to P, which is the upper limit of the number of loads. It can be adaptively adjusted by voltage fluctuations. In each iteration, the dictionary atom that best matches the residual is selected, and the sparse coefficients and residuals are updated.

[0019] The iteration stops when the Frobenius norm of the updated residual is less than or equal to ε.

[0020] Set up an atomic update formula for the dictionary to update the dictionary matrix D;

[0021] Reconstruct the waveforms of each independent load based on the sparsity coefficients and the updated dictionary matrix D. .

[0022] Optionally, in one example of the above aspects, the separated independent load waveforms are jointly verified with visual sensor data and electrical parameters to ensure that the separated waveforms match the actual operating electronic and electrical products. The extracted independent load waveform feature data is used for fault monitoring and fault classification of electronic and electrical products, including the following steps:

[0023] Set the reference timestamp for load waveform acquisition, and simultaneously extract independent load waveforms, visual sensor data and electrical parameters at the same moment as a verification data set;

[0024] Each electronic and electrical product is assigned a unique identifier (ID). The separated independent load waveform carries an ID tag, which is bound to the product ID identified by vision and the product ID collected by electrical parameters.

[0025] A three-level feature collaborative verification strategy is set up to perform progressive verification of waveform feature verification, electrical feature verification, and visual feature verification.

[0026] After completing the three-level feature co-verification, the fault-sensitive features extracted from the independent load waveform include: time domain features: peak current deviation rate, frequency domain features: harmonic distortion rate, and time-frequency domain features: wavelet entropy.

[0027] Acquire the temperature and acoustic characteristics of electronic and electrical products. The temperature characteristics include external temperature and the temperature of key internal components. The acoustic characteristics include sound intensity, frequency, and MFCC Mel-frequency cepstral coefficients.

[0028] After co-verification with the three-level features, obtain the product type of the electronic and electrical products, the historical data of the corresponding type of electronic and electrical products, and analyze the temperature characteristics, acoustic characteristics and fault sensitivity characteristics.

[0029] Based on the historical data on whether electrical products are faulty and their fault classification at the corresponding feature data collection time, temperature features, acoustic features, and fault-sensitive features are labeled, and a feature-weighted random forest algorithm is trained to identify whether electrical products are faulty and their fault classification.

[0030] Optionally, in one example of the above aspects, the three-level feature collaborative verification strategy includes:

[0031] Level 1 Verification: Based on the independent load waveform, waveform frequency, peak current, and waveform harmonic content, the features of the separated waveform are compared with the standard waveform feature library for different product types to calculate the feature similarity. The waveform feature similarity calculation uses the cosine similarity algorithm to calculate the feature similarity between the separated waveform and the standard waveform. The product type corresponding to the standard waveform with the highest feature similarity is the Level 1 verification identification type.

[0032] Secondary verification: Based on electrical parameters including operating power, power factor, and rated voltage, the theoretical power is calculated based on waveform characteristics, and the power deviation rate δP is calculated. If δP≤5%, the secondary verification is qualified, and the tertiary verification is performed. Otherwise, the independent load waveform is reacquired and verified again, and an error warning is issued.

[0033] Level 3 verification: This involves detecting the product type, operating status, and number of different operating statuses using visual sensing data. Operating statuses include power on and power off. The verification process checks whether the number of isolated independent load waveforms matches the number of powered-on appliances identified by the vision system, and whether the identification type of the Level 1 verification matches the visual identification type. If the number of isolated independent load waveforms matches the number of powered-on appliances identified by the vision system, and the types match, then the Level 3 verification is successful. Otherwise, the independent load waveforms are reacquired for Level 1 verification, and an error warning is issued.

[0034] Optionally, in one example of the above aspects, it also includes:

[0035] Classification verification module:

[0036] By extracting leakage parameters from the collected leakage signals, key parameters are calculated, including: the effective value of leakage current, the three-electrode potential difference, and the harmonic content of the leakage signal.

[0037] Leakage current parameters are spliced ​​and fused with temperature data and acoustic data to form fused data;

[0038] The isolated independent load waveforms are verified together with visual sensor data and electrical parameters. After matching the isolated waveforms with the actual operating electrical appliances, the cosine similarity between the corresponding fused data and the fused data of equipment with different fault types is calculated.

[0039] Parameters with cosine similarity greater than a threshold are selected. The fault type corresponding to the parameter with the highest cosine similarity among the selected parameters is taken as the verification fault type. The verification fault type is checked to see if it is consistent with the classification result of the fault classification module. If they are consistent, the verification passes; otherwise, an instruction to reclassify the fault is sent to the fault classification module.

[0040] Optionally, in one example of the above aspects, based on corrosion current density, corrosion potential, salt spray concentration, temperature and humidity parameters, and fault classification results, and combined with an improved grey relational analysis algorithm, the weights of each parameter are determined, and a corrosion rate prediction model is constructed, including the following steps:

[0041] By integrating corrosion current density, corrosion potential, salt spray concentration, temperature and humidity parameters, and using an improved grey relational analysis algorithm, the correlation between each parameter and the corrosion rate is calculated.

[0042] By employing an improved grey relational analysis algorithm to calculate the correlation between each parameter and the corrosion rate, the weights of each parameter are determined, and the parameter weights are calculated based on the correlation. The sum of the correlation between all influencing parameters;

[0043] Based on parameter weights, and combined with corrosion current density, corrosion potential, salt spray concentration, temperature and humidity parameters, and fault classification results, a corrosion rate prediction model is constructed.

[0044] Optionally, in one example of the above aspects, a corrosion rate prediction model is constructed based on parameter weights, combined with corrosion current density, corrosion potential, salt spray concentration, temperature and humidity parameters, and fault classification results, including the following steps:

[0045] Experts classify the fault levels of different fault types for the same type of products, and assign values ​​to different fault levels. The fault classification results are then assigned values ​​after being classified into levels.

[0046] Set baseline values ​​for corrosion current density, corrosion potential, salt spray concentration, and relative humidity, and calculate the final corrosion rate.

[0047] The calculated final corrosion rate is normalized, and the corrosion level is classified according to the normalized final corrosion rate.

[0048] Optionally, in one example of the above aspects, the remaining service life is predicted based on the corrosion rate and the thickness of the metal component, combined with the Arrhenius equation to correct for the accelerating effect of temperature on corrosion, including the following steps:

[0049] Based on the corrosion rate and the thickness of the metal components, and by combining the Arrhenius equation to correct for the accelerating effect of temperature on corrosion, the remaining service life is predicted.

[0050] The three-level early warning mechanism is set up as follows: light corrosion triggers a blue warning, moderate corrosion triggers a yellow warning, and severe corrosion or remaining service life ≤ 6 months triggers a red warning, while generating a corrosion trend curve.

[0051] Compared with the prior art, the beneficial effects of the present invention are:

[0052] This invention introduces prior information about the load type through an improved K-SVD algorithm, constraining the sparsity of dictionary atoms to make them more closely reflect actual load characteristics. The OMP algorithm iteratively selects the atoms that best match the residuals, gradually approximating the true signal. The resulting reconstructed independent load waveform has significantly reduced errors and retains more complete waveform details. Sparsity constraints suppress noise interference, highlight key load characteristics, and provide a more reliable data foundation for subsequent fault diagnosis or type identification.

[0053] This invention eliminates misjudgments by spatiotemporally aligning the separated independent load waveforms with the equipment operating status captured by a visual sensor; it further verifies the rationality of the waveform separation results by combining electrical parameters such as power factor and three-phase imbalance; and it achieves fault type subdivision by extracting waveform features and fusing them with visual features to train a classification model.

[0054] This invention introduces the entropy weight method to dynamically adjust the weights of parameters such as corrosion current density and salt spray concentration, making the model more adaptable to complex environments and avoiding the problem of traditional grey relational analysis being easily affected by extreme values. It also combines temperature and humidity parameters to correct the corrosion potential. For example, in high temperature and high humidity environments, the salt spray corrosion rate may increase exponentially, and the model captures this coupling effect through nonlinear fitting. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a schematic diagram of the system framework of the present invention;

[0057] Figure 2 This is a schematic diagram of the workflow of the classification and verification module of the present invention. Detailed Implementation

[0058] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] Please see Figures 1-2 The first aspect of this invention provides an automated inspection system for electronic and electrical products based on multimodal sensor fusion, comprising:

[0060] Atomic Library Building Module: Constructs a load waveform atomic library, optimizes the load waveform atomic library by improving the K-SVD algorithm, uses standard waveform atoms of electronic and electrical products as dictionary atoms, introduces prior information of load type, and constrains the sparsity of dictionary atoms;

[0061] Waveform analysis module: The sparse coefficients are solved by the OMP orthogonal matching pursuit algorithm, and the number of iterations is set. In each iteration, the dictionary atom that best matches the residual is selected, the sparse coefficients and residuals are updated, and each independent load waveform is reconstructed by the sparse coefficients and the load waveform atom library.

[0062] Fault Classification Module: The separated independent load waveforms are verified together with visual sensor data and electrical parameters to ensure that the separated waveforms match the actual operating electronic and electrical products. The extracted independent load waveform feature data is used for fault monitoring and fault classification of electronic and electrical products.

[0063] Remaining service life prediction module: Based on corrosion current density, corrosion potential, salt spray concentration, temperature and humidity parameters, and fault classification results, a corrosion rate prediction model is constructed, and an improved grey relational analysis algorithm is used to determine the weight of each parameter; based on corrosion rate and metal component thickness, combined with the Arrhenius equation to correct the accelerating effect of temperature on corrosion, the remaining service life is predicted.

[0064] Specifically, in this embodiment, a priori information about the load type is introduced through an improved K-SVD algorithm to constrain the sparsity of dictionary atoms, making the dictionary atoms more closely match the actual load characteristics. The OMP algorithm iteratively selects the atoms that best match the residuals, gradually approximating the real signal. The resulting reconstructed independent load waveform has significantly reduced errors and retains more complete waveform details. Sparsity constraints suppress noise interference, highlight key load characteristics, and provide a more reliable data foundation for subsequent fault diagnosis or type identification.

[0065] The improved K-SVD reduces the number of iterations and computational complexity by initializing a dictionary to match fault features or introducing a harmonic ratio index to optimize the atom selection strategy. The OMP algorithm employs orthogonalization to avoid redundant iterations in traditional matching pursuit, resulting in faster convergence and suitability for real-time detection scenarios. Combined with data fusion processing from multimodal sensors, the algorithm can run in parallel, fully utilizing multi-core processors or GPU acceleration to meet the real-time analysis needs of large-scale data such as high-voltage transmission line detection.

[0066] Sparse representation models reconstruct signals through linear combinations of a small number of atoms, naturally possessing noise resistance. Improved K-SVD further optimizes dictionary atoms, making it insensitive to noise; the OMP algorithm, through a residual update mechanism, gradually eliminates noise components, improving the signal-to-noise ratio of the reconstructed signal. Fusion of multi-sensor data allows for cross-validation of load waveform reconstruction results, avoiding misjudgments from a single sensor and improving the system's robustness in complex environments.

[0067] In power monitoring, the system can extract fault features based on reconstructed waveforms and train fault diagnosis models using historical data to achieve early fault warnings and reduce unplanned power outages. The sparse representation model uses a small number of atoms and coefficients to represent signals, significantly reducing data storage requirements.

[0068] By spatiotemporally aligning the isolated load waveforms with the equipment operating status captured by visual sensors, misjudgments from a single sensor are eliminated. For example, if the waveform displays abnormal harmonics but no arc is detected visually, it can be determined as interference rather than a real fault. The rationality of the waveform separation results is further verified by combining electrical parameters such as power factor and three-phase imbalance. For instance, when a motor is overloaded, current waveform distortion and power factor decline occur simultaneously, improving the confidence level of fault diagnosis. By extracting waveform features and fusing them with visual features to train a classification model, fault type subdivision is achieved.

[0069] Traditional grey relational analysis is susceptible to extreme values. The improved model dynamically adjusts the weights of parameters such as corrosion current density and salt spray concentration by introducing the entropy weight method, making the model more adaptable to complex environments. Corrosion potential is corrected by incorporating temperature and humidity parameters. For example, in high-temperature and high-humidity environments, the salt spray corrosion rate may increase exponentially; the model captures this coupling effect through nonlinear fitting.

[0070] Traditional life prediction often overlooks the accelerating effect of temperature on corrosion. By introducing the Arrhenius equation, this paper quantifies the influence of temperature and corrects the relationship between corrosion rate and metal thickness. Combined with historical equipment operating data, life degradation models under different operating conditions are established. For example, motors that are frequently started and stopped corrode more rapidly due to thermal expansion and contraction; the model can specifically shorten their predicted lifespan.

[0071] In one embodiment of the present invention, the atom library building module constructs a load waveform atom library, using standard waveform atoms of electronic and electrical products as dictionary atoms, including the following steps:

[0072] Waveform signals of different load types are collected from electronic and electrical products, and the single-channel mixed waveform is represented as a linear superposition of the individual load waveforms, i.e.:

[0073]

[0074] Where y(t) is the single-channel mixed waveform at time t, ai is the weighting coefficient of the i-th load (representing the load operating power ratio), si(t) is the independent waveform of the i-th load at time t, i∈(1,2,…,n), n is the number of loads, and Ns(t) is the measurement noise at time t.

[0075] Based on the collected single-channel mixed waveform y(t), solve si(t) to obtain the independent load waveform of the electronic and electrical products;

[0076] The independent load waveforms of historical data of electronic and electrical products are preset as a priori load types. Combined with the independent load waveforms collected in real time from electronic and electrical products, a load waveform atom library is constructed, and the standard waveform atoms of electronic and electrical products are used as dictionary atoms.

[0077] In one embodiment of the present invention, the load waveform atom library is optimized by improving the K-SVD algorithm, introducing prior information on load types, and constraining the sparsity of dictionary atoms, including the following steps:

[0078] The load waveform atom library is optimized by improving the K-SVD algorithm, extracting feature data of prior load types, including waveform frequency and peak characteristics of different loads, constraining the sparsity of dictionary atoms, and setting the optimization objective function as follows:

[0079]

[0080] Where Y is the observation data matrix, each column of which typically represents an observation signal or sample, and the dimension depends on the number of features of the signal and the number of samples; D is the dictionary matrix, which consists of multiple dictionary atoms. These dictionary atoms are used to represent the signal in a linear combination. The number of columns corresponds to the number of atoms, and the number of rows is related to the dimension of the signal; X is the sparse coefficient matrix, each column of which is the sparse representation coefficient of the column vector in the corresponding sample Y. These coefficients determine the linear combination method of the atoms in the dictionary. Denotes the square of the Frobenius norm. Used to measure the error between the observed data matrix Y and the data DX reconstructed from the dictionary matrix D and the sparse coefficient matrix X;

[0081] λ is the sparsity penalty factor. The value of the sparsity penalty factor needs to be adjusted according to the specific problem. In this embodiment, the sparsity penalty factor is 0.01-0.1, which is used to balance sparsity and goodness of fit. It controls the requirement for sparsity of the sparse coefficient matrix X. The larger λ is, the fewer non-zero elements there will be in X, that is, the sparser it is; xj is the sparse coefficient vector corresponding to the j-th sample, j∈(1,2,…,m), and m is the number of columns of the sparse coefficient matrix X, that is, the number of samples;

[0082] γ is the dictionary atom constraint factor. The atom constraint factor is used to balance the reconstruction error and the consistency between the dictionary atoms and the standard waveform. Its value needs to be determined according to the actual situation. In this embodiment, the dictionary atom constraint factor is set to 0.05-0.2 to ensure the consistency between the atoms and the standard load waveform.

[0083] si(t) is the independent waveform of the i-th load at time t, i∈(1,2,…,n), and n is the number of loads;

[0084] This is a submatrix composed of dictionary atoms that select the prior load type whose index belongs to the i-th load from the dictionary matrix D.

[0085] This is used to constrain dictionary atoms related to various electrical appliances to be as close as possible to their corresponding standard waveform templates, in order to ensure the rationality and representativeness of dictionary atoms.

[0086] In one embodiment of the present invention, the sparse coefficients are solved using the OMP orthogonal matching pursuit algorithm, and the number of iterations is set. In each iteration, the dictionary atom that best matches the residual is selected, the sparse coefficients and the residual are updated, and each independent load waveform is reconstructed using the sparse coefficients and the load waveform atom library. The process includes the following steps:

[0087] The sparse coefficients are solved using the OMP (Orthogonal Matching Pursuit) algorithm, with the number of iterations set to P, where P is the upper limit of the number of loads and can be adaptively adjusted based on voltage fluctuations. In each iteration, the dictionary atom that best matches the residual is selected, and the sparse coefficients and residuals are updated. The residual update formula is as follows:

[0088]

[0089] in, For the (k+1)th iteration residual, Let the residual be the value of the k-th iteration. The sparse coefficients are obtained in the k-th iteration. The dictionary atoms selected for the k-th iteration;

[0090] When the Frobenius norm of the updated residual is ≤ ε, in this embodiment, ε = 0.001, which is set as the noise threshold, the iteration stops;

[0091] Set up an atomic update formula for the dictionary to update the dictionary matrix D:

[0092]

[0093]

[0094] Where dk is the k-th dictionary atom to be updated, and xk is the sparse coefficient row vector of the k-th atom. T Let Ek be the transpose of the coefficients, and Ek be the residual matrix. Denotes the square of the Frobenius norm;

[0095] Y1k is a subset of training samples using the k-th atom, and Dh is the h-th dictionary atom (j≠k). Let h be the coefficient of the h-th atom in the sample subset. , where c is the weight coefficient of the c-th class of samples, and c is the load type, i.e., the load category index (such as air conditioner, refrigerator, etc.).

[0096] Based on the sparse coefficients and the updated dictionary matrix D, reconstruct the waveforms of each independent load: sk(t) = dk·xk.

[0097] In one embodiment of the present invention, the separated independent load waveforms are jointly verified with visual sensor data and electrical parameters to ensure that the separated waveforms match the actual operating electronic and electrical products. The extracted independent load waveform feature data is used for fault monitoring and fault classification of electronic and electrical products, including the following steps:

[0098] Set the reference timestamp for load waveform acquisition, and simultaneously extract independent load waveforms, visual sensor data and electrical parameters at the same moment as a verification data set;

[0099] Each electronic and electrical product is assigned a unique identifier (ID). The separated independent load waveform carries an ID tag, which is bound to the product ID identified by vision and the product ID collected by electrical parameters.

[0100] A three-level feature collaborative verification strategy is set up to perform progressive verification of waveform feature verification, electrical feature verification, and visual feature verification.

[0101] After completing the three-level feature co-verification, the fault-sensitive features extracted from the independent load waveform include: time domain features: peak current deviation rate, frequency domain features: harmonic distortion rate, and time-frequency domain features: wavelet entropy.

[0102] Acquire the temperature and acoustic characteristics of electronic and electrical products. The temperature characteristics include external temperature and the temperature of key internal components. The acoustic characteristics include sound intensity, frequency, and MFCC Mel-frequency cepstral coefficients.

[0103] After co-verification with the three-level features, obtain the product type of the electronic and electrical products, the historical data of the corresponding type of electronic and electrical products, and analyze the temperature characteristics, acoustic characteristics and fault sensitivity characteristics.

[0104] Based on the historical data on whether electrical products are faulty and their fault classification at the corresponding feature data collection time, temperature features, acoustic features, and fault-sensitive features are labeled, and a feature-weighted random forest algorithm is trained to identify whether electrical products are faulty and their fault classification.

[0105] In one embodiment of the present invention, the three-level feature collaborative verification strategy includes:

[0106] Level 1 Verification: Based on the independent load waveform, waveform frequency, peak current, and waveform harmonic content, the features of the separated waveform are compared with the standard waveform feature library for different product types to calculate the feature similarity. The waveform feature similarity calculation uses the cosine similarity algorithm to calculate the feature similarity between the separated waveform and the standard waveform. The product type corresponding to the standard waveform with the highest feature similarity is the Level 1 verification identification type.

[0107] Secondary verification: Based on electrical parameters including operating power, power factor, and rated voltage, theoretical power is calculated based on waveform characteristics. Where Un is the rated voltage; I(gp,gn) is the standard waveform with the highest feature similarity detected in the first-level verification, corresponding to the current value under the working condition gp and condition gn of the data to be verified; This refers to the power factor considering the relevant parameter n.

[0108] Calculate the power deviation rate by comparing the actual collected power Pmeas. Where δP represents the relative error of power; Pcalc is the theoretical power; and Pmeas is the acquired power. If δP ≤ 5%, the second-level verification is qualified, and the third-level verification is performed. Otherwise, the independent load waveform is reacquired and verified again, and an error warning is issued.

[0109] Level 3 verification: This involves detecting the product type, operating status, and number of different operating statuses using visual sensing data. Operating statuses include power on and power off. The verification process checks whether the number of isolated independent load waveforms matches the number of powered-on appliances identified by the vision system, and whether the identification type of the Level 1 verification matches the visual identification type. If the number of isolated independent load waveforms matches the number of powered-on appliances identified by the vision system, and the types match, then the Level 3 verification is successful. Otherwise, the independent load waveforms are reacquired for Level 1 verification, and an error warning is issued.

[0110] In one embodiment of the present invention, it further includes:

[0111] Classification verification module:

[0112] By extracting leakage parameters from the collected leakage signals, key parameters are calculated, including: the effective value of leakage current, the three-electrode potential difference, and the harmonic content of the leakage signal.

[0113] Leakage current parameters are spliced ​​and fused with temperature data and acoustic data to form fused data;

[0114] The isolated independent load waveforms are verified together with visual sensor data and electrical parameters. After matching the isolated waveforms with the actual operating electrical appliances, the cosine similarity between the corresponding fused data and the fused data of equipment with different fault types is calculated.

[0115] Parameters with cosine similarity greater than a threshold are selected. The fault type corresponding to the parameter with the highest cosine similarity among the selected parameters is taken as the verification fault type. The verification fault type is checked to see if it is consistent with the classification result of the fault classification module. If they are consistent, the verification passes; otherwise, an instruction to reclassify the fault is sent to the fault classification module.

[0116] In this embodiment, leakage parameters can be extracted from the collected leakage signal, and key parameters including the effective value of leakage current, the three-electrode potential difference and the harmonic content of leakage signal can be calculated to construct a leakage feature vector and set leakage judgment rules to determine whether electronic and electrical products have leakage risks.

[0117] The leakage current determination rule is as follows:

[0118] Based on the GB 12358-2022 standard, the following grading thresholds are set:

[0119] When the first-level verification identification type is Class I electrical appliance, the effective value of the leakage current is ≤30mA; or when the first-level verification identification type is Class II electrical appliance, the effective value of the leakage current is ≤0.75mA. The leakage current is qualified, and proceed to the next step.

[0120] If the harmonic content of the leakage signal is ≤5% and the potential difference is within the allowable range (potential difference ≤5V), it is considered qualified; if it exceeds the threshold, it is considered a leakage fault, and the fault type (such as poor grounding contact or insulation damage) is located by the potential difference and harmonic content.

[0121] In this embodiment, the leakage current determination rule result can also be corrected through Bayesian inference: the Bayesian posterior probability formula is set as follows: Where F is the leakage fault event, X is the multimodal feature vector including leakage parameters, temperature features and acoustic features; P(X|F) is the likelihood probability of the feature under fault conditions, P(F) is the prior probability of leakage fault based on historical data statistics, and P(X) is the marginal probability of the feature.

[0122] In one embodiment of the present invention, a corrosion rate prediction model is constructed by determining the weights of each parameter based on corrosion current density, corrosion potential, salt spray concentration, temperature and humidity parameters, and fault classification results, combined with an improved grey relational analysis algorithm. The model includes the following steps:

[0123] By integrating corrosion current density, corrosion potential, salt spray concentration, and temperature and humidity parameters, and using an improved grey relational analysis algorithm, the correlation between each parameter and the corrosion rate is calculated.

[0124]

[0125] in, The correlation between the *a*th influencing parameter and the corrosion rate reflects the closeness of the relationship between the parameter and the corrosion rate. The greater the correlation, the greater the relative influence of the parameter on the corrosion rate. *ra* represents the *ra*th sampling, where *ra* ∈ (1, 2, ..., R), and R is the number of samplings, i.e., the number of times data is acquired. *a* is the index of the influencing parameter, where *a* = 1, 2, 3, 4, corresponding to: corrosion current density, corrosion potential, salt spray concentration, and relative humidity (RH), respectively.

[0126] xa(ra): the rath sampled value of the ath influencing parameter, y(ra): the rath sampled value of the corrosion rate, and ξ: the resolution coefficient, which is generally between 0 and 1. In this embodiment, it is 0.5. Its function is to adjust the resolution of the correlation coefficient. The smaller ξ is, the greater the difference between the correlation coefficients and the stronger the discrimination ability.

[0127] for The two-stage minimum difference method involves first finding the minimum difference between the sampled value and the corrosion rate among all influencing parameters, and then finding the minimum of this minimum value among all sampling times. for The two-level maximum difference; that is, first find the maximum value of the difference between the sampled value and the corrosion rate among all the influencing parameters, and then find the maximum value of this maximum value among all the sampling times.

[0128] It is the absolute value of the difference between the ra-th sampled value of the a-th influencing parameter and the ra-th sampled value of the corrosion rate.

[0129] By employing an improved grey relational analysis algorithm to calculate the correlation between each parameter and the corrosion rate, the weights of each parameter are determined, and the parameter weights are calculated based on the correlation. The sum of the correlation between all influencing parameters;

[0130] Based on parameter weights, and combined with corrosion current density, corrosion potential, salt spray concentration, temperature and humidity parameters, and fault classification results, a corrosion rate prediction model is constructed.

[0131] In one embodiment of the present invention, a corrosion rate prediction model is constructed based on parameter weights, combined with corrosion current density, corrosion potential, salt spray concentration, temperature and humidity parameters, and fault classification results, including the following steps:

[0132] Experts classify the fault levels of different fault types for the same type of products, assign values ​​to different fault levels, and assign the fault classification results to Fz after classifying the fault levels.

[0133] Set baseline values ​​for corrosion current density, corrosion potential, salt spray concentration, and relative humidity, and calculate the final corrosion rate Vcorr using the formula: Vcorr = ω1·Icorr / Icorr0 + ω2·|Ecorr| / |Ecorr0| + ω3·Csalt / Csalt0 + ω4·RH / RH0 + f·Fz

[0134] Where Icorr is the corrosion current density, Ecorr is the corrosion potential, Csalt is the salt spray concentration, and RH is the relative humidity; Icorr0 is the baseline value of corrosion current density, Ecorr0 is the baseline value of corrosion potential, Csalt0 is the baseline value of salt spray concentration, RH0 is the baseline value of relative humidity, and f is the parameter assigned after the fault level classification, which is set to 0.1 in this embodiment.

[0135] The calculated corrosion rate Vcorr is normalized, and the corrosion level is classified according to the normalized final corrosion rate.

[0136] In this embodiment, the final corrosion rate Vcorr is normalized, and the corrosion level is divided according to the normalized final corrosion rate: Vcorr < 0.3 is classified as mild, 0.3-0.6 as moderate, and Vcorr > 0.6 as severe.

[0137] In this embodiment, experts classify the fault levels of the same type of products according to different fault types. Based on the severity of the fault, the experts divide the fault into Level 1 fault, Level 2 fault and Level 3 fault, and assign a value of 0.7 to Level 1 fault, 0.5 to Level 2 fault and 0.2 to Level 3 fault.

[0138] In one embodiment of the present invention, based on the corrosion rate and the thickness of the metal component, and combined with the Arrhenius equation to correct for the accelerating effect of temperature on corrosion, the remaining service life is predicted, including the following steps:

[0139] Based on corrosion rate and metal component thickness, and by correcting for the accelerating effect of temperature on corrosion using the Arrhenius equation, the remaining service life (Lremaining) is predicted.

[0140] Lremaining=(d0-dmin) / [Vcorr·exp(-Q / (R·T))]; where d0 is the initial thickness of the metal component, dmin is the minimum thickness at the safety threshold, Q is the activation energy of the corrosion reaction, which is set according to different materials, for example, 105kJ / mol for copper; R is the gas constant, set to 8.314J / (mol·K); and T is the absolute temperature.

[0141] The three-level early warning mechanism is set up as follows: light corrosion triggers a blue warning, moderate corrosion triggers a yellow warning, and severe corrosion or remaining service life ≤ 6 months triggers a red warning, while generating a corrosion trend curve.

[0142] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. An automated testing system for electronic and electrical products based on multimodal sensor fusion, characterized in that, include: Atomic Library Building Module: Constructs a load waveform atomic library, optimizes the load waveform atomic library by improving the K-SVD algorithm, uses standard waveform atoms of electronic and electrical products as dictionary atoms, introduces prior information of load type, and constrains the sparsity of dictionary atoms; Waveform analysis module: The sparse coefficients are solved by the OMP orthogonal matching pursuit algorithm, and the number of iterations is set. In each iteration, the dictionary atom that best matches the residual is selected, the sparse coefficients and residuals are updated, and each independent load waveform is reconstructed by the sparse coefficients and the load waveform atom library. Fault Classification Module: The separated independent load waveforms are verified together with visual sensor data and electrical parameters to ensure that the separated waveforms match the actual operating electronic and electrical products. The extracted independent load waveform feature data is used for fault monitoring and fault classification of electronic and electrical products. Remaining service life prediction module: Based on corrosion current density, corrosion potential, salt spray concentration, temperature and humidity parameters and fault classification results, a corrosion rate prediction model is constructed, and an improved grey relational analysis algorithm is used to determine the weight of each parameter; based on corrosion rate and metal component thickness, combined with the Arrhenius equation to correct the accelerating effect of temperature on corrosion, the remaining service life is predicted. The construction of the load waveform atom library includes the following steps: Waveform signals of different load types are collected from electronic and electrical products, and the single-channel mixed waveform is represented as a linear superposition of the independent load waveforms, i.e.: ; Where y(t) is the single-channel mixed waveform at time t, ai is the weighting coefficient of the i-th load, si(t) is the independent waveform of the i-th load at time t, i∈(1,2,…,n), n is the number of loads, and Ns(t) is the measurement noise at time t; Based on the collected single-channel mixed waveform y(t), solve si(t) to obtain the independent load waveform of the electronic and electrical products; The independent load waveforms of historical data of electronic and electrical products are preset as a priori load types. Combined with the independent load waveforms collected in real time from electronic and electrical products, a load waveform atomic library is constructed. The improvement of the K-SVD algorithm to optimize the load waveform atom library, the introduction of prior information on load type, and the constraint of the sparsity of dictionary atoms include the following steps: The load waveform atom library is optimized by improving the K-SVD algorithm, extracting feature data of prior load types, constraining the sparsity of dictionary atoms, and setting the optimization objective function as follows: ； Where Y is the observation data matrix, D is the dictionary matrix (composed of multiple dictionary atoms), and X is the sparse coefficient matrix. Denotes the square of the Frobenius norm. Used to measure the error between the observed data matrix Y and the data DX reconstructed from the dictionary matrix D and the sparse coefficient matrix X; λ is the sparsity penalty factor, xj is the sparse coefficient vector corresponding to the j-th sample, j∈(1,2,…,m), and m is the number of columns of the sparse coefficient matrix X, which is the number of samples; γ is the dictionary atom constraint factor, si(t) is the independent waveform of the i-th load at time t, i∈(1,2,…,n), and n is the number of loads; This is a submatrix composed of dictionary atoms that select the prior load type whose index belongs to the i-th load from the dictionary matrix D.

2. The automated testing system for electronic and electrical products based on multimodal sensor fusion according to claim 1, characterized in that, The sparse coefficients are solved using the OMP (Orthogonal Matching Pursuit) algorithm, with a set number of iterations. In each iteration, the dictionary atom that best matches the residual is selected, and the sparse coefficients and residuals are updated. The independent load waveforms are then reconstructed using the sparse coefficients and the load waveform atom library, including the following steps: The sparse coefficients are solved using the Orthogonal Matching Pursuit (OMP) algorithm, with P iterations. In each iteration, the dictionary atom that best matches the residual is selected, and the sparse coefficients and residuals are updated. The residual update formula is as follows: ; in, For the (k+1)th iteration residual, Let the residual be the value of the k-th iteration. The sparse coefficients are obtained in the k-th iteration. The dictionary atoms selected for the k-th iteration; The iteration stops when the Frobenius norm of the updated residual is less than or equal to ε. Set up an atomic update formula for the dictionary to update the dictionary matrix D: ； ； Where dk is the k-th dictionary atom to be updated, and xk is the sparse coefficient row vector of the k-th atom. T Let Ek be the transpose of the coefficients, and Ek be the residual matrix. Denotes the square of the Frobenius norm; Y1k is the subset of training samples using the k-th atom, and Dh is the h-th dictionary atom. Let h be the coefficient of the h-th atom in the sample subset. represents the weighting coefficient for the c-th class of samples, where c is the load type; Based on the sparse coefficients and the updated dictionary matrix D, reconstruct the waveforms of each independent load: sk(t) = dk·xk.

3. The automated testing system for electronic and electrical products based on multimodal sensor fusion according to claim 1, characterized in that, The separated independent load waveforms are jointly verified with visual sensor data and electrical parameters to ensure that the separated waveforms match the actual operating electronic and electrical products. The extracted independent load waveform feature data is used for fault monitoring and fault classification of electronic and electrical products, including the following steps: Set the reference timestamp for load waveform acquisition, and simultaneously extract independent load waveforms, visual sensor data and electrical parameters at the same moment as a verification data set; Each electronic and electrical product is assigned a unique identifier (ID). The separated independent load waveform carries an ID tag, which is bound to the product ID identified by vision and the product ID collected by electrical parameters. A three-level feature collaborative verification strategy is set up to perform progressive verification of waveform feature verification, electrical feature verification, and visual feature verification. After completing the three-level feature co-verification, the fault-sensitive features extracted from the independent load waveform include: time domain features: peak current deviation rate, frequency domain features: harmonic distortion rate, and time-frequency domain features: wavelet entropy. Acquire the temperature and acoustic characteristics of electronic and electrical products. The temperature characteristics include external temperature and the temperature of key internal components. The acoustic characteristics include sound intensity, frequency, and MFCC Mel-frequency cepstral coefficients. After co-verification with the three-level features, obtain the product type of the electronic and electrical products, the historical data of the corresponding type of electronic and electrical products, and analyze the temperature characteristics, acoustic characteristics and fault sensitivity characteristics. Based on the historical data on whether electrical products are faulty and their fault classification at the corresponding feature data collection time, temperature features, acoustic features, and fault-sensitive features are labeled, and a feature-weighted random forest algorithm is trained to identify whether electrical products are faulty and their fault classification.

4. The automated testing system for electronic and electrical products based on multimodal sensor fusion according to claim 3, characterized in that, The three-level feature collaborative verification strategy includes: Level 1 Verification: Based on the independent load waveform, waveform frequency, peak current, and waveform harmonic content, the features of the separated waveform are compared with the standard waveform feature library for different product types to calculate the feature similarity. The waveform feature similarity calculation uses the cosine similarity algorithm to calculate the feature similarity between the separated waveform and the standard waveform. The product type corresponding to the standard waveform with the highest feature similarity is the Level 1 verification identification type. Secondary verification: Based on electrical parameters including operating power, power factor, and rated voltage, the theoretical power is calculated based on waveform characteristics, and compared with the actual collected power Pmeas to calculate the power deviation rate. Where δP represents the relative error of power; Pcalc is the theoretical power; and Pmeas is the acquired power. If δP ≤ 5%, the second-level verification is qualified, and the third-level verification is performed. Otherwise, the independent load waveform is reacquired and verified again, and an error warning is issued. Level 3 verification: This involves detecting the product type, operating status, and number of different operating statuses using visual sensing data. Operating statuses include power on and power off. The verification process checks whether the number of isolated independent load waveforms matches the number of powered-on appliances identified by the vision system, and whether the identification type of the Level 1 verification matches the visual identification type. If the number of isolated independent load waveforms matches the number of powered-on appliances identified by the vision system, and the types match, then the Level 3 verification is successful. Otherwise, the independent load waveforms are reacquired for Level 1 verification, and an error warning is issued.

5. The automated testing system for electronic and electrical products based on multimodal sensor fusion according to claim 3, characterized in that, Also includes: Classification verification module: By extracting leakage parameters from the collected leakage signals, key parameters are calculated, including: the effective value of leakage current, the three-electrode potential difference, and the harmonic content of the leakage signal. Leakage current parameters are spliced ​​and fused with temperature data and acoustic data to form fused data; The isolated independent load waveforms are verified together with visual sensor data and electrical parameters. After matching the isolated waveforms with the actual operating electrical appliances, the cosine similarity between the corresponding fused data and the fused data of equipment with different fault types is calculated. Parameters with cosine similarity greater than a threshold are selected. The fault type corresponding to the parameter with the highest cosine similarity among the selected parameters is taken as the verification fault type. The verification fault type is checked to see if it is consistent with the classification result of the fault classification module. If they are consistent, the verification passes; otherwise, an instruction to reclassify the fault is sent to the fault classification module.

6. The automated testing system for electronic and electrical products based on multimodal sensor fusion according to claim 1, characterized in that, Based on corrosion current density, corrosion potential, salt spray concentration, temperature and humidity parameters, and fault classification results, and combined with an improved grey relational analysis algorithm, the weights of each parameter are determined, and a corrosion rate prediction model is constructed, including the following steps: By integrating corrosion current density, corrosion potential, salt spray concentration, and temperature and humidity parameters, and using an improved grey relational analysis algorithm, the correlation between each parameter and the corrosion rate is calculated. ； in, Let ra be the correlation between the a-th influencing parameter and the corrosion rate, ra be the a-th sampling, ra∈(1,2,…,R), R be the number of samplings, and a be the index of the influencing parameter, a=1,2,3,4, which correspond to corrosion current density, corrosion potential, salt spray concentration, and relative humidity RH, respectively. xa(ra) is the ra-th sampled value of the a-th influencing parameter, y(ra) is the ra-th sampled value of the corrosion rate, and ξ is the resolution coefficient. for The two minimum differences, for The maximum difference between the two levels; By employing an improved grey relational analysis algorithm to calculate the correlation between each parameter and the corrosion rate, the weights of each parameter are determined, and the parameter weights are calculated based on the correlation. The sum of the correlation between all influencing parameters; Based on parameter weights, and combined with corrosion current density, corrosion potential, salt spray concentration, temperature and humidity parameters, and fault classification results, a corrosion rate prediction model is constructed.

7. The automated testing system for electronic and electrical products based on multimodal sensor fusion according to claim 6, characterized in that, Based on parameter weights, and combining corrosion current density, corrosion potential, salt spray concentration, temperature and humidity parameters, and fault classification results, a corrosion rate prediction model is constructed, including the following steps: Experts classify the fault levels of different fault types for products of the same type, assign values ​​to different fault levels, and assign values ​​Fz to the fault classification results after classifying the faults. Set the baseline values ​​for corrosion current density, corrosion potential, salt spray concentration, and relative humidity, and calculate the final corrosion rate Vcorr using the formula: Vcorr=ω1·Icorr / Icorr0+ω2·|Ecorr| / |Ecorr0|+ω3·Csalt / Csalt0+ω4·RH / RH0+f·Fz; Where Icorr is the corrosion current density, Ecorr is the corrosion potential, Csalt is the salt spray concentration, and RH is the relative humidity; Icorr0 is the baseline value of corrosion current density, Ecorr0 is the baseline value of corrosion potential, Csalt0 is the baseline value of salt spray concentration, RH0 is the baseline value of relative humidity, and f is the parameter assigned after the fault level classification. The calculated corrosion rate Vcorr is normalized, and the corrosion level is classified according to the normalized final corrosion rate.

8. The automated testing system for electronic and electrical products based on multimodal sensor fusion according to claim 7, characterized in that, Based on corrosion rate and metal component thickness, and by correcting for the accelerating effect of temperature on corrosion using the Arrhenius equation, the remaining service life is predicted, including the following steps: Based on corrosion rate and metal component thickness, and by correcting for the accelerating effect of temperature on corrosion using the Arrhenius equation, the remaining service life (Lremaining) is predicted. Lremaining=(d0-dmin) / [Vcorr·exp(-Q / (R·T))]; where d0 is the initial thickness of the metal component, dmin is the minimum thickness at the safety threshold, Q is the activation energy of the corrosion reaction, R is the gas constant, and T is the absolute temperature; The three-level early warning mechanism is set up as follows: light corrosion triggers a blue warning, moderate corrosion triggers a yellow warning, and severe corrosion or remaining service life ≤ 6 months triggers a red warning, while generating a corrosion trend curve.

Images