Multi-sensor fusion data processing method and system

Abstract

The invention discloses a multi-sensor fusion data processing method and system, and the method comprises the steps: receiving the signal information of a plurality of sensors, extracting original data and manual features, carrying out the combined learning of a sensor specific auto-encoder through output label knowledge, fusing the weight of the auto-encoder and the weight of a classifier, so as to learn discriminative features, and further classifying the test data. The system comprises a memory, a communication interface and a hardware processor and is used for executing the operations. According to the method, discriminative features are learned through a joint optimization technology, compared with the prior art, the performance is better, the complexity is lower, good accuracy and robustness are shown in scenes such as bearing fault detection, and the method can be applied to multiple fields such as remote sensing and industrial automation.

Description

Technical Field

[0001] This invention relates to the field of multi-sensor fusion technology, specifically to a data processing method and system for multi-sensor fusion. Background Technology

[0002] Existing system performance monitoring technologies typically rely on a single sensor as a key performance indicator (KPI). Systems based on a single sensor are inefficient because inference using a single sensor always involves uncertainty. However, when multiple sensors are available, combining complementary, competitive, or cooperative information from various sensors can achieve more robust and accurate inference. Multi-sensor fusion is a technique that effectively utilizes data from multiple sensors, creating a unified picture with improved information, and offers significant advantages over existing single-sensor technologies. This technology can address various challenges such as incomplete data, data dimensionality, and conflicting data.

[0003] Multi-sensor fusion has made some progress in improving reasoning and decision-making due to its advantages. Various fusion structures can be adopted depending on the nature of the problem and the information provided by the sensors. The current trend is to use handcrafted sensor features for information fusion, which largely depends on the application domain. Recently, due to the increasing complexity of sensing environments and the massive amounts of data generated, processing such complex and massive data often exceeds human capabilities. Therefore, compared to handcrafted feature design, learning representations directly from data using representation learning techniques has received much attention. Generally, representation learning techniques are part of a two-stage network: the first stage is used to learn representations from sensor data, and then the second stage learns a classifier / regressor for reasoning.

[0004] In existing technologies, system inference based on a single sensor suffers from high uncertainty and low efficiency. When using handcrafted sensor features for information fusion, it is highly dependent on the application domain and exceeds human processing capabilities when faced with complex and massive data. Two-stage networks have certain limitations in feature learning and classification inference, making it difficult to efficiently learn more discriminative features. Therefore, it is necessary to develop a data processing method and system for multi-sensor fusion. Summary of the Invention

[0005] The purpose of this invention is to provide a data processing method and system for multi-sensor fusion to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a multi-sensor fusion data processing method, comprising the following steps:

[0007] A. Signal information reception and training preparation;

[0008] B. Data Feature Extraction:

[0009] C. Collaborative learning process;

[0010] D. Classification of test data.

[0011] Preferably, step A specifically involves receiving signal information from multiple sensors (X1, X2, ..., Xn), which will be used to train sensor-specific autoencoders (Ai) and fusion autoencoders (Af) for multi-label classification.

[0012] Preferably, step B specifically involves: extracting raw sensor data and domain-specific manual features from the received signal information from multiple sensors.

[0013] Preferably, step C specifically involves: using one or more hardware processors to jointly learn sensor-specific autoencoders (Ai), fusion autoencoders (Af), and classifier weights (M) based on the knowledge of the output label (Y); for the i-th sensor (i = 1, ..., n) and j-th layer depth (j = 1, ..., K), learning sensor-specific autoencoder weights, including encoder weights and decoder weights, as well as hidden representations; during the learning process, applying a nonlinearity of the modified linear unit type to the latent representation by forcing the representation of each layer to be non-negative, stacked autoencoders are used to learn deep representations from multiple sensors, and fusion is performed using a fusion autoencoder; the latent representation of the fusion autoencoder is mapped to a linear classifier to perform classification.

[0014] Preferably, step D specifically involves: providing test data for classification, classifying the test data using the learned sensor-specific autoencoder (Ai), fusion autoencoder (Af), and classifier weights (M); for the test data corresponding to the i-th sensor, obtaining the latent representation using the learned weights, concatenating all sensor-specific representations to obtain Ztest, and finally obtaining the predicted output label Ytest through the classifier weights M.

[0015] Preferably, in step C, a dynamic weight adjustment mechanism is introduced during the joint learning process. This mechanism assigns dynamic learning weights to the autoencoders of different sensors based on the real-time signal-to-noise ratio (SNR) of each sensor data. Specifically, for sensor data with an SNR higher than a preset threshold, the learning weight of its corresponding autoencoder is increased to enhance the extraction of effective features. For sensor data with a low SNR, the learning weight of its corresponding autoencoder is reduced and an adaptive noise filtering module is enabled. This module calculates the data variance through a sliding window, and automatically triggers wavelet threshold denoising when the variance exceeds a preset fluctuation threshold.

[0016] Preferably, in step D, a hierarchical confidence fusion strategy is used when classifying the test data, specifically including:

[0017] First, the test data is initially classified by the specific autoencoder of each sensor to generate sensor-level prediction labels and corresponding confidence scores.

[0018] Sensor prediction labels with confidence scores higher than the first threshold are directly included in the candidate result set;

[0019] For sensor predicted labels with confidence scores below the second threshold, a secondary feature extraction is initiated, their latent representation is recalculated, and the confidence score is updated by combining the intermediate layer output of the fused autoencoder.

[0020] Finally, the candidate result set is merged through a weighted voting mechanism. The weights are positively correlated with the historical classification accuracy of the sensor, and the final predicted output label Ytest is generated.

[0021] Preferably, a multi-sensor fusion data processing system includes: a memory storing instructions; one or more communication interfaces; and one or more hardware processors connected to the memory via the one or more communication interfaces, wherein these hardware processors are configured by instructions to extract raw sensor data and domain-specific handcrafted features from received signal information from multiple sensors (X1, X2, ..., Xn); jointly learn the weights (M) of a sensor-specific autoencoder (Ai), a fusion autoencoder (Af), and a classifier using knowledge of the output label (Y) to learn discriminative features from the signal information of each sensor signal (X1, X2, ..., Xn); provide test data for classification; and classify the test data using the learned weights (M) of the sensor-specific autoencoder (Ai), the fusion autoencoder (Af), and the classifier.

[0022] Preferably, a computer program product includes a non-volatile computer-readable medium containing a computer-readable program that, when executed on a computing device, enables the computing device to receive signal information from multiple sensors (X1, X2, ..., Xn) for training a sensor-specific autoencoder (Ai) (i = 1, ..., n) and a fusion autoencoder (Af) for multi-label classification; extract raw sensor data and domain-specific manual features from the received signal information from the multiple sensors (X1, X2, ..., Xn); jointly learn the weights (M) of the sensor-specific autoencoder (Ai), the fusion autoencoder (Af), and the classifier using knowledge of the output label (Y) to learn discriminative features from the signal information of each sensor signal (X1, X2, ..., Xn); provide test data for classification; and classify the test data using the learned weights (M) of the sensor-specific autoencoder (Ai), the fusion autoencoder (Af), and the classifier.

[0023] Beneficial effects:

[0024] (1) The present invention adopts an autoencoder-based architecture and learns the weights of the autoencoder and classifier simultaneously through joint optimization techniques, which can learn discriminative features from multiple sensors and shows better performance than existing technologies.

[0025] (2) The technical solution of the present invention reduces complexity, eliminates the need for complex manual feature design, and can automatically learn representations from data, thereby improving the efficiency and accuracy of multi-sensor fusion. In noisy environments, the technical solution of the present invention has better robustness and can maintain a high classification accuracy at different signal-to-noise ratio levels.

[0026] (3) The technical solution of the present invention has a relatively simple structure and has better performance in some application scenarios, such as bearing fault detection, compared with complex methods based on deep learning.

[0027] The above description is merely an overview of the technical solutions of the embodiments of this application. In order to better understand the technical means of the embodiments of this application and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of this application more apparent and understandable, specific implementation methods of this application are described below. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the multi-sensor fusion system based on an autoencoder according to the present invention;

[0029] Figure 2 This is a flowchart of the multi-sensor fusion method based on an autoencoder of the present invention;

[0030] Figure 3 A schematic diagram of a discriminative autoencoder framework used in some embodiments of the present invention is shown below;

[0031] Figure 4 This is a schematic diagram of the deep FDAE framework of some embodiments disclosed in this invention;

[0032] Figure 5 This is a functional block diagram of a specific sensor autoencoder training phase of the present invention;

[0033] Figure 6 This is a functional block diagram of the testing phase of the automatic encoder of the present invention;

[0034] Figure 7 The following are performance graphs of FDAE in bearing fault detection according to some embodiments of the present invention;

[0035] Figure 8 The following are performance graphs of FDAE in bearing fault detection according to some embodiments of the present invention;

[0036] Figure 9 The diagram illustrates the performance of FDAE under additive white Gaussian noise using different signal-to-noise ratios.

[0037] In the picture: Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims and drawings of this application are intended to cover non-exclusive inclusion.

[0040] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0041] Please see Figures 1-9 This invention discloses a data processing method for multi-sensor fusion, comprising the following steps:

[0042] A. Signal information reception and training preparation;

[0043] B. Data Feature Extraction:

[0044] C. Collaborative learning process;

[0045] D. Classification of test data.

[0046] Specifically, step A involves receiving signal information from multiple sensors (X1, X2, ..., Xn), which will be used to train sensor-specific autoencoders (Ai) and fusion autoencoders (Af) for multi-label classification.

[0047] Step B specifically involves extracting raw sensor data and domain-specific manual features from the received signal information from multiple sensors.

[0048] Step C specifically involves: using one or more hardware processors and knowledge of the output label (Y), jointly learning the weights (M) of a sensor-specific autoencoder (Ai), a fusion autoencoder (Af), and a classifier; for the i-th sensor (i = 1, ..., n) and j-th layer depth (j = 1, ..., K), learning sensor-specific autoencoder weights, including encoder weights and decoder weights, as well as hidden representations; during the learning process, applying a nonlinearity of the modified linear unit type to the latent representations by forcing the representations of each layer to be non-negative, stacking autoencoders to learn deep representations from multiple sensors, and fusing them using a fusion autoencoder; the latent representations of the fusion autoencoder are mapped to a linear classifier to perform classification.

[0049] Step D specifically involves: providing test data for classification, using the learned sensor-specific autoencoder (Ai), fusion autoencoder (Af), and classifier weights (M) to classify the test data; for the test data corresponding to the i-th sensor, using the learned weights to obtain the latent representation, concatenating all sensor-specific representations to obtain Ztest, and finally obtaining the predicted output label Ytest through the classifier weights M.

[0050] In step C of this invention, a dynamic weight adjustment mechanism is introduced during the joint learning process. This mechanism assigns dynamic learning weights to the autoencoders of different sensors based on the real-time signal-to-noise ratio (SNR) of each sensor's data. Specifically, for sensor data with an SNR higher than a preset threshold, the learning weight of its corresponding autoencoder is increased to enhance the extraction of effective features. For sensor data with a low SNR, the learning weight of its corresponding autoencoder is reduced, and an adaptive noise filtering module is enabled. This module calculates the data variance through a sliding window, and automatically triggers wavelet thresholding denoising when the variance exceeds a preset fluctuation threshold. Through dynamic weight adjustment and adaptive noise filtering, the problem of uneven data quality from multiple sensors is solved, further improving the targeting and robustness of feature learning in complex noise environments, especially suitable for sensor aging or fluctuating operating conditions in industrial automation.

[0051] In step D of this invention, a hierarchical confidence fusion strategy is used when classifying test data, specifically including:

[0052] First, the test data is initially classified by the specific autoencoder of each sensor to generate sensor-level prediction labels and corresponding confidence scores.

[0053] Sensor prediction labels with confidence scores higher than the first threshold are directly included in the candidate result set;

[0054] For sensor predicted labels with confidence scores below the second threshold, a secondary feature extraction is initiated, their latent representation is recalculated, and the confidence score is updated by combining the intermediate layer output of the fused autoencoder.

[0055] Finally, the candidate result set is merged through a weighted voting mechanism. The weights are positively correlated with the historical classification accuracy of the sensor, and the final predicted output label Ytest is generated.

[0056] The hierarchical confidence fusion strategy combines the advantages of individual sensor reliability with global fusion, reducing the interference of low-quality predictions on the final result. In scenarios requiring high-precision inference, such as remote sensing image classification, it can significantly improve classification stability.

[0057] The present invention also discloses a multi-sensor fusion data processing system, comprising: a memory for storing instructions; one or more communication interfaces; and one or more hardware processors connected to the memory via the one or more communication interfaces, wherein the hardware processors are configured by instructions to extract raw sensor data and domain-specific handcrafted features from received signal information of multiple sensors (X1, X2, ..., Xn); jointly learn sensor-specific autoencoders (Ai), fusion autoencoders (Af), and classifier weights (M) using knowledge of output labels (Y) to learn discriminative features from the signal information of each sensor signal (X1, X2, ..., Xn); provide test data for classification; and classify the test data using the learned sensor-specific autoencoders (Ai), fusion autoencoders (Af), and classifier weights (M).

[0058] like Figure 1 As shown, system 10 is configured for multi-sensor fusion based on an autoencoder. A single-level network is employed here, which incorporates representation learning. This technique performs feature-level fusion, with inputs that can be raw sensor data or domain-specific handcrafted features extracted from different sensors. Unlike two-stage networks, this single-level network allows for the learning of more discriminative features from a single sensor, resulting in a more efficient approach that outperforms existing techniques.

[0059] The system 10 may include one or more computing devices 11, such as a laptop, a desktop computer, a workstation, a cloud-based computing environment, etc. It is understood that the system 10 can communicate via one or more input / output interfaces 12-1,...,12-N, collectively referred to as I / O interfaces 12. Examples of I / O interfaces 12 may include, but are not limited to, user interfaces, portable computers, personal digital assistants, handheld devices, smartphones, tablets, workstations, etc. The I / O interfaces 12 are communicatively coupled to the system 10 via a network 13.

[0060] In one embodiment, network 13 can be a wireless or wired network, or a combination of both. For example, network 13 can be implemented as a computer network, as one of different types of networks such as a Virtual Private Network (PN), Intranet, Local Area Network (LAN), Wide Area Network (WAN), the Internet, etc. Network 13 can be a private network or a shared network, representing a combination of different types of networks using various protocols, such as Hypertext Transfer Protocol (HTTP), Transmission Control Protocol / Internet Protocol (TCP / IP), and Wireless Application Protocol (WAP), to communicate with each other. Furthermore, network 13 may include various network devices, including routers, bridges, servers, computing devices, and storage devices. Network devices within network 13 can interact with system 10 via communication links.

[0061] System 10 supports various connectivity options, such as Bluetooth, USB, ZigBee, and other cellular services. The network environment enables the various components of System 10 to connect via any communication link, including the Internet, wide area network (WAN), local area network (LAN), etc. In one exemplary implementation, System 10 is implemented to operate as a standalone device. In another embodiment, System 10 can work as a loosely coupled device in conjunction with an intelligent computing environment. The components and functions of System 10 are described in further detail.

[0062] In this invention, an autoencoder is used to learn a recognition function so that the output closely follows the input, in which process it learns abstract representations from the data. The framework utilizes a discriminative autoencoder to fuse information from multiple sensors (X1, X2, ..., Xn). Here, a dedicated autoencoder is used to learn a high-level abstract representation from each sensor. Subsequently, for classification, the latent representations from multiple sensors (X1, X2, ..., Xn) are stacked together and fed as input to the fusion autoencoder, whose latent representations are then fed into a label-consistent classifier. Therefore, this framework uses discriminative autoencoder representations for multi-sensor fusion, and is thus referred to as fusion using discriminative autoencoders, as... Figure 3 As shown, joint optimization is used to simultaneously learn sensor-specific representations and classifier weights. By stacking autoencoders on the sensor side, richer and more accurate sensor data representations can be learned to learn a K-layer deep autoencoder.

[0063] Furthermore, the FDAE framework employs a joint learning approach, simultaneously learning the discriminative autoencoder and classifier during the training phase, and then using them for fusion during the testing phase. During training, the autoencoder, fused autoencoder, and classification weights (M) for a specific sensor are jointly learned by utilizing knowledge of the output label (Y). This structure allows discriminative features to be learned from each sensor, thereby leveraging their complementary information to derive the final inference result.

[0064] See Figure 4 For multiple sensors (X1, X2, ..., X...), n A network composed of K layers of depth autoencoders is used on the sensor side to enhance the Lagrangian formula. The joint optimization framework of FDAE is expressed as follows:

[0065]

[0066] Make all H's≥0, H f ≥0 and

[0067] In this invention, a modified linear unit type nonlinearity is applied to the latent representation by forcing the representation of each layer to be nonnegative. Stacked autoencoders ([W e1n W d1n ] to [W eKn W dkn This is used to learn deep representations from multiple sensors (X1, X2, ..., Xn). These representations are treated as features and are processed using a fusion autoencoder (W). ef W df The fusion is performed. The latent representation of the fused autoencoder (Af) is mapped to a linear classifier to perform classification. The first few terms relate to learning a K-layer deep sensor-specific autoencoder. The last three terms relate to learning the fused autoencoder using stacked input representations from multiple sensors (X1, X2, ..., Xn) and classifier weights (M). In this framework, the encoder and decoder weights of the autoencoder are given equal importance, so the hyperparameters associated with these terms are 1 and therefore not explicitly shown.

[0068] For the i-th sensor (i = 1, ..., n), at a depth of j (j = 1, ..., K), learn sensor-specific autoencoder weights (51) & (52). The following is a subproblem of learning encoder and decoder weights and hidden representations (53): The sensor-specific encoder weights are denoted as (54):

[0069] Where, for j=1, H oi =X i ;

[0070] W eji =H ji (H (j-1)i ) -1

[0071] The weights of the sensor-specific decoder are denoted as (55):

[0072]

[0073] W dji =H (j-1)i (H ji ) -1

[0074] For j = K-1, the potential representation of a specific sensor in layers K = 1 to K-1 is as follows:

[0075]

[0076] The latent representations of the last layer (K) are computed differently when used as input to the fusion autoencoder, and therefore they are learned using the following formula (56):

[0077]

[0078] For the second stage, the learned public or fusion encoder W is updated using the least squares method by solving the following problem. ef and decoder W dp :

[0079]

[0080] W ef =H f (Z) -1

[0081]

[0082] W df =Z(H f ) -1

[0083]

[0084] let

[0085] The closed-form update of the fusion representation, Hf, is computed as follows:

[0086]

[0087] For updating the classification weights, M(57):

[0088]

[0089] M = Y(H) f ) -1 .

[0090] An approximation method was adopted, where negative values ​​were set to 0 after each latent representation was resolved. The network parameters of the FDAE framework were updated iteratively until a termination condition was met, at which point the model was considered to have completed learning and training stopped (58). The learning termination condition included: (i) completing a predefined number of iterations, and (ii) the difference between the current iteration's fusion autoencoder (A) and the previous iteration's fusion autoencoder (A) was less than an empirically determined threshold.

[0091] In this invention, for the test data corresponding to the i-th sensor Using the learned weights W e1i W e2i , ..., W eKi Obtaining latent representations (all j = {1, 2, ..., k}) and These can be represented as:

[0092] and

[0093]

[0094] therefore, These representations were learned for all sensors i = {1, 2, ..., n}. All sensor-specific representations were concatenated to obtain Z. test

[0095]

[0096] Finally, Y test The methods to obtain it are as follows:

[0097]

[0098] In one example, an industrial machine for bearing fault detection and classification is provided. Here, FDAE uses a Max Poling architecture to determine the category labels in the output. The hyperparameter λ and hidden layer dimension of the FDAE autoencoder are tuned using a grid search method, and the results are given using the optimal values.

[0099] Bearing data provided by the Case Western Reserve University Bearing Data Center was used for the classification of normal and faulty bearings. The experimental setup consisted of a motor whose shaft was supported by bearings mounted on the drive end and fan end of the motor. Faults of different diameters (7, 14, and 21 mils) were presented for the balls, inner ring, and outer ring of the bearings. Two accelerometers, mounted on the drive end of the motor and the fan end respectively, were used to capture vibration data at a sampling frequency of 12 kHz under four different load conditions. Four levels of classification were performed to detect (i) initial faults (when the fault occurs) and (ii) faults of all severity. The sample duration for both classification problems was considered to be 500 milliseconds. For initial fault detection, only fault data with a diameter of 7 mm were considered, totaling 320 samples for all four categories. For fault detection of all severity, fault data corresponding to all fault diameters were considered, resulting in a total of 640 samples for classification.

[0100] Different time, frequency, and time-frequency domain features were extracted from each data sample and input into the FDAE framework for classification. Time-domain features included squared mean, variance, peak value, and peak-to-peak value. Frequency-domain features included spectral skewness, kurtosis, and energy; wavelet energy was used for time-frequency features. Table 1 lists the performance of single-layer deep (K=1) FDAE and compares it with state-of-the-art methods using 5x cross-validation averaging.

[0101]

[0102] Table 1

[0103] Among them, SVM (Support Vector Machine)

[0104] KNN (K-Nearest Neighbors) algorithm

[0105] MLP (Multi-Layer Perceptron)

[0106] SAE-DNN (Stacked Autoencoders + Deep Neural Networks)

[0107] RNN (Recurrent Neural Networks)

[0108] GRU (Gated Recurrent Unit)

[0109] BiGRU (Bidirectional Gated Recurrent Unit)

[0110] ICDSVM (Improved Combined Decomposition and Support Vector Machine)

[0111] [20,40], [20,20], [20,80], and [10,20] represent the number of neurons in two hidden layers, respectively. For example, [20,40] means that there are 20 neurons in the first layer and 40 neurons in the second layer.

[0112] Table 1 also mentions the network structures used for the deep learning models. It can be seen that FDAE exhibits superior performance for initial faults, despite its relatively simple structure; in contrast, deep learning-based methods such as BiGRU are inferior. To compare fault detection performance at different severity levels, FDAE demonstrates higher accuracy using simple features compared to methods employing permutation entropy and intrinsic mode function-based features combined with optimized SVM classification. Figure 7 , Figure 8 The performance characteristics of the FDAE optimization framework are shown, including convergence and accuracy comparison plots. The algorithm converges within a few iterations. It can be seen that different λ values ​​have little impact on accuracy.

[0113] Furthermore, the performance of FDAE in a simulated real-world noisy environment was tested. Additive white Gaussian noise with different signal-to-noise ratios was introduced, and the corresponding classification accuracy was calculated. Similar to the work described above, four types of faults were identified in bearing signals with noise under zero load conditions. Figure 9 The average accuracy of the proposed algorithm in five cross-validations is shown in the presence of noise, and compared with other methods. These methods include SDA (Stacked Denoising Autoencoder), SAE (Stacked Autoencoder), RF (Random Forest), SVM (Support Vector Machine), and AE (Autoencoder). The results show that FDAE performs well at all signal-to-noise ratios, but its performance is comparable to other methods at 20 dB. This highlights the robustness and effectiveness of the proposed framework, which achieves classification by fusing information from multiple sensors.

[0114] The embodiments of this disclosure address unresolved issues in multi-sensor fusion. Specifically, embodiments of this disclosure provide a multi-sensor fusion method and system based on a discriminative autoencoder architecture. Joint learning of the autoencoder and classifier weights enables the learning of more reliable discriminative and robust features from sensor signals, thereby improving the reliability of inference.

[0115] It is important to clarify that the scope of protection extends to such programs, and in addition to computer-readable storage devices, it includes devices containing program code for implementing one or more steps of the method when the program runs on a server, mobile device, or any suitable programmable device. The hardware device can be any kind of device, can be programmed, including any kind of computer, such as a server or personal computer, or similar, or any combination thereof. The device may also include means, such as hardware means, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or a combination of hardware and software means, such as ASICs and FPGAs, or a memory with at least one microprocessor and at least one built-in software module. Therefore, means can include both hardware and software means. The method embodiments described herein can be implemented in hardware and software. The device may also include software means. Alternatively, embodiments can be implemented on different hardware devices, such as using multiple CPUs.

[0116] The embodiments described herein may include hardware and software elements. Embodiments implemented in software form include, but are not limited to, firmware, resident software, microcode, etc. The functions performed by the various modules described herein may be implemented in other modules or combinations of other modules. For the purposes of this description, a computer-usable or computer-readable medium may be any means capable of containing, storing, communicating, propagating, or transmitting a program for use in or in conjunction with an instruction execution system, apparatus, or device.

[0117] Furthermore, one or more computer-readable storage media may be used when implementing embodiments conforming to this disclosure. A computer-readable storage medium refers to any type of physical memory capable of storing processor-readable information or data. Therefore, a computer-readable storage medium can store instructions for execution by one or more processors, including instructions for causing the processor to perform steps or stages consistent with the embodiments described herein. The term “computer-readable medium” should be understood to include tangible articles but exclude carrier waves and transient signals, i.e., non-volatile signals. Examples include random access memory (RAM), read-only memory (ROM), volatile memory, non-volatile memory, hard disks, CD-ROMs, DVDs, flash drives, magnetic disks, and any other known physical storage media.

[0118] In summary, this invention employs an autoencoder-based architecture and simultaneously learns the weights of the autoencoder and classifier through joint optimization techniques. This enables the learning of discriminative features from multiple sensors, demonstrating superior performance compared to existing technologies. The technical solution of this invention reduces complexity, eliminating the need for complex manual feature design and automatically learning representations from data, thus improving the efficiency and accuracy of multi-sensor fusion. In noisy environments, the technical solution exhibits better robustness, maintaining high classification accuracy across different signal-to-noise ratios. The technical solution of this invention has a relatively simple structure and, in some application scenarios such as bearing fault detection, demonstrates superior performance compared to more complex deep learning-based methods.

[0119] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A multi-sensor fusion data processing method, characterized in that: The method comprises the following steps: A. Signal information receiving and training preparation; B. Data feature extraction; C. Joint learning process; D. Test data classification.

2. The data processing method of claim 1, wherein: The step A specifically comprises: receiving signal information from multiple sensors (X1, X2,..., Xn), which will be used to train sensor-specific autoencoders (Ai) and fusion autoencoders (Af) for multi-label classification.

3. The data processing method of claim 1, wherein: The step B specifically comprises: extracting original sensor data and domain-specific manual features from the received signal information of multiple sensors.

4. The data processing method of claim 1, wherein: The step C specifically comprises: using one or more hardware processors to jointly learn the weights (M) of the sensor-specific autoencoders (Ai), the fusion autoencoders (Af) and the classifier using the knowledge of the output labels (Y); for the i-th sensor (i=1,...,n) and the j-th layer (j=1,...,K), learning the sensor-specific autoencoder weights, including the encoder weights and the decoder weights, and the hidden representation; During the learning process, by forcing the representation of each layer to be non-negative, applying a rectified linear unit type of nonlinearity to the latent representation, the stacked autoencoders are used to learn deep representations from multiple sensors, and the fusion autoencoders are used for fusion, and the latent representation of the fusion autoencoders is mapped to a linear classifier to perform classification.

5. The data processing method of claim 1, wherein: The step D specifically comprises: providing test data for classification, using the learned weights of the sensor-specific autoencoders (Ai), the fusion autoencoders (Af) and the classifier to classify the test data; for the test data corresponding to the i-th sensor, using the learned weights to obtain the latent representation, connecting all sensor-specific representations to obtain Ztest, and finally obtaining the predicted output label Ytest through the classifier weight M.

6. The data processing method of claim 1, wherein: In the step C, a dynamic weight adjustment mechanism is introduced in the joint learning process, which allocates dynamic learning weights to the autoencoders of different sensors according to the real-time signal-to-noise ratio of each sensor data; specifically, for sensor data with a signal-to-noise ratio higher than a preset threshold, the learning weight of the corresponding autoencoder is increased to enhance the extraction of effective features; for sensor data with a lower signal-to-noise ratio, the learning weight of the corresponding autoencoder is reduced and an adaptive noise filtering module is enabled, which calculates the data variance through a sliding window and automatically triggers wavelet threshold denoising when the variance exceeds a preset fluctuation threshold.

7. The data processing method of claim 1, wherein: In the step D, a hierarchical confidence fusion strategy is adopted for test data classification, specifically including: First, the test data is preliminarily classified by each sensor-specific autoencoder to generate sensor-level predicted labels and corresponding confidence scores; The sensor predicted labels with confidence scores higher than a first threshold are directly included in the candidate result set; For sensor predicted labels with confidence scores lower than a second threshold, secondary feature extraction is started, the latent representation is recalculated and the confidence is updated combined with the intermediate layer output of the fusion autoencoder; Finally, the candidate result set is fused through a weighted voting mechanism, the weight is positively correlated with the historical classification accuracy of the sensor, and the final predicted output label Ytest is generated.

8. A multi-sensor fusion data processing system characterized by: comprises: a memory storing instructions; one or more communication interfaces; one or more hardware processors connected to the memory through the one or more communication interfaces, wherein the hardware processors are configured by the instructions to extract raw sensor data and domain-specific handcrafted features from received signal information of a plurality of sensors (X1, X2,..., Xn); jointly learn weights (M) of sensor-specific autoencoders (Ai), a fusion autoencoder (Af), and a classifier using knowledge of output labels (Y) to learn discriminative features from signal information of each sensor signal (X1, X2,..., Xn); provide test data for classification; and use the learned weights (M) of the sensor-specific autoencoders (Ai), the fusion autoencoder (Af), and the classifier to classify the test data.

9. A computer program product, characterised in that: comprises a non-volatile computer-readable medium containing computer-readable program, when the computer-readable program is executed on a computing device, the computing device is enabled to receive signal information from a plurality of sensors (X1, X2,..., Xn) for training sensor-specific autoencoders (Ai) (i = 1,..., n) and a fusion autoencoder (Af) for multi-label classification; extract raw sensor data and domain-specific handcrafted features from the received signal information of the plurality of sensors (X1, X2,..., Xn); jointly learn weights (M) of the sensor-specific autoencoders (Ai), the fusion autoencoder (Af), and a classifier using knowledge of output labels (Y) to learn discriminative features from signal information of each sensor signal (X1, X2,..., Xn); provide test data for classification; and use the learned weights (M) of the sensor-specific autoencoders (Ai), the fusion autoencoder (Af), and the classifier to classify the test data.

Images