A cross-modal feature space mapping and missing modal transfer method and device based on contrast learning, an electronic device, and a storage medium

Abstract

A cross-modal feature space mapping and missing modal transfer method and device based on contrast learning, an electronic device, and a storage medium. The present application classifies the vibration and thermal imaging modalities in the source domain after CNN feature extraction according to the label, maps the features to the same feature space, uses the contrast learning method to minimize the distance between different modalities of the same category and maximize the distance between different categories, forms a feature space, and then classifies through the feature space. In the target domain, after the vibration signal is extracted by 1D-CNN, it is mapped to the same feature space, and then classified, so that the feature expression has semantic consistency. Through cross-modal feature mapping and modal completion technology, efficient knowledge transfer and classification under the condition of missing modal in the target domain are realized. By using the class-aware contrast learning method, the semantic consistency of the cross-modal features in the shared space is ensured, and the classification accuracy is improved. By using the class-weighted contrast learning method, the accuracy and robustness of the minority class fault classification are improved.

Description

Technical Field

[0001] This invention relates to the field of component fault diagnosis technology, specifically to a method and apparatus for cross-modal feature space mapping and missing mode transfer based on contrastive learning. Background Technology

[0002] In industrial settings, a wealth of data exists across multiple modalities, including vibration signals, acoustic emission signals, thermal imaging, and RGB data. A single sensor provides limited information, making it difficult to accurately assess the health status of equipment. Multimodal data reflects the equipment's condition from different perspectives. Therefore, when one or more modal monitoring data are missing, the original classification model lacks information, leading to decreased classification accuracy and impacting the operation, maintenance, and management of the enterprise's production line.

[0003] With the continuous development of science and technology, multimodal technology has become an important field of artificial intelligence. This technology can leverage the powerful generalization ability of deep learning to discover complex patterns and relationships that may be undetectable by single-modal systems, thus advancing work in multiple fields. However, real-world multimodal systems often face the challenge of handling situations where certain data modalities may be missing or incomplete, severely impacting model performance. Therefore, we refer to the challenge of handling missing modalities in multimodal learning as the "missing modality problem," and the solution to this problem as multimodal learning with missing modalities. Multimodal learning with missing modalities often focuses on information retrieval, remote sensing, robot vision, medical diagnosis, sentiment analysis, and multi-view clustering. Existing technologies have the following drawbacks: 1) The problem of missing modes has not been effectively addressed. Existing fault diagnosis methods in smart manufacturing plants typically assume that all sensor data or modal information is complete. However, in practical applications, due to factors such as sensor failure, incomplete signal acquisition, or limitations in funding and space, some modes may be missing, leading to a decline in the performance of existing fault diagnosis models, poor generalization ability, and inability to adapt to incomplete modal conditions.

[0004] 2) Cross-modal features are difficult to align and share. Existing methods lack effective mechanisms for aligning and sharing cross-modal features when processing multimodal data.

[0005] 3) The problems of insufficient data and class imbalance are difficult to solve. Existing methods lack effective cross-modal feature alignment and sharing mechanisms when processing multimodal data. Summary of the Invention

[0006] This invention aims to solve the following problems: (1) The problem of missing modes in smart manufacturing plants. Due to problems such as missing signals, sensor failures, and insufficient funds and space, some modes are missing in smart manufacturing plants, resulting in a decrease in the accuracy and efficiency of fault diagnosis models.

[0007] (2) Problems of cross-modal feature alignment and sharing. In fault diagnosis of smart manufacturing plants, data often comes from different modes (such as vibration signals and thermal images). However, the data from different modes differ in their expression and distribution, making it difficult to align features and directly fuse and share them.

[0008] (3) The problem of insufficient training data in the construction of smart manufacturing plants for SMEs. Fault diagnosis model training requires mature datasets, has a long collection cycle, and suffers from serious class imbalance. SMEs lack the conditions to build datasets for their newly built smart factories. This paper addresses the problem of insufficient data by using existing labeled multimodal datasets to complete fault diagnosis tasks using unlabeled data.

[0009] In addressing the issue of missing modes during multimodal fault diagnosis due to sensor malfunctions, insufficient funding, site constraints, and signal loss, this invention efficiently and lightweightly performs classification tasks to determine equipment health status, thus providing protection against production safety accidents, particularly in cases of single or partial mode loss.

[0010] This invention provides a cross-modal feature space mapping and missing mode transfer method based on contrastive learning, which is applied to fault detection and includes the following steps: Acquire raw vibration data and raw thermal imaging data; The original vibration data and original thermal imaging data are used to construct a source domain dataset and set labels; Construct a target domain dataset using the raw vibration data; Extract multimodal features from the source domain dataset: Features of vibration data and thermal imaging data are extracted using 1D-CNN and 2D-CNN respectively, and then the self-attention mechanism of 1D-CNN and 2D-CNN is applied. Comparative learning: Based on the labels, a contrastive loss function is used to bring together multimodal features of the same category and push away features of different categories, ultimately constructing a shared feature space to ensure semantic consistency. Classifier training: The classifier is trained using the label information of the source domain dataset. By maximizing the matching probability between shared features and their corresponding labels, the classifier receives the shared features after feature extraction and projection as input, and optimizes the loss function of the classification task through backpropagation, thereby learning the discrimination boundary that can effectively distinguish different categories. Target domain feature mapping and classification: After the vibration data in the target domain dataset is processed by 1D-CNN to extract features, it is directly mapped to the shared feature space, and the target domain features are classified using a classifier trained in the source domain. Evaluation: Output fault diagnosis results and calculate evaluation indicators.

[0011] Specifically, the method for constructing the source domain dataset is as follows: Time window processing, label setting, data normalization, and building multi-source heterogeneous datasets; The method for constructing the target domain dataset is as follows: Time window processing and data standardization.

[0012] Specifically, for the original vibration data, feature extraction is performed using 1D-CNN, and the method is as follows: ; ; in, It is the first The output features of the layer and It is the first The convolutional kernel weights and biases of the layer, where K is the kernel size. Represents the convolution operation. These are the output features of a 1D-CNN; Indicates the number of floors. It is the first Layer output features, initial input features ;

[0013] For the original thermal imaging data, feature extraction is performed using 2D-CNN, as follows: ; ; in, Indicates the number of floors. and It is the first The kernel weights and biases of the layers. and Represents the height and width of the convolution kernel. Indicates a 2D convolution operation; It is the first Layer output features, initial input features .

[0014] Specifically, the self-attention mechanism of 1D-CNN is as follows: First, a mapping of query (Q), key (K), and value (V) is established, where... It is a learnable projection matrix. The dimensions after projection; ; Calculate the weights of attention, where Indicate the correlation between features: ; Weighted output: ; Integrating original features, among which These are weighting coefficients: ; The self-attention mechanism of 2D-CNN is as follows: First, the features are reshaped into a matrix form. ,in The feature number after flattening; Then, the mapping of query (Q), key (K), and value (V) is obtained through projection, where It is a learnable projection matrix. The dimensions after projection; ; Calculate the weights of attention, where Indicates the correlation between pixels: ; Weighted output: ; Reshape and integrate the original features, in which These are weighting coefficients: .

[0015] Specifically, the comparative learning refers to: The features of the two modalities are mapped to a shared feature space through a shared mapping module. and ;

[0016] Through projection mapping Mapping the features of the two modalities to the same feature space : ; Positive sample pairs are samples from the same class, while negative sample pairs are samples from different classes. Cosine similarity or Euclidean distance is used to measure the similarity between two features. The goal is to maximize the similarity between positive sample pairs and minimize the similarity between negative sample pairs.

[0017] Specifically, the contrastive learning loss function is as follows: ; in, It is the set of indices of all samples in the current batch. Is with the sample A set of positive samples belonging to the same category. In addition to the batch The set of all samples other than itself, that is, the set of positive and negative samples; express and The similarity is measured using cosine similarity. It is a temperature parameter that adjusts the sensitivity of similarity. in, ; and For positive sample pairs, calculate similarity. By maximizing the molecular Close the distance between samples of the same category; and Features belonging to negative sample pairs or other positive sample pairs are obtained by minimizing the features in the denominator. Push them further away; denominator Indicates target sample The sum of similarities with all samples in the batch.

[0018] Furthermore, the contrastive learning employs a dynamic weighting mechanism: For positive sample pairs Similarity of sample pairs The lower the value, the greater the weight, and the more attention is paid to less similar positive sample pairs; For negative sample pairs Similarity of sample pairs The higher the value, the greater the weight, thus focusing more on negative sample pairs that are easily confused; Define dynamic weights For sample pairs The loss contribution is weighted and adjusted, and the weights of positive sample pairs are initially defined as follows: ; The lower the weight The larger the value, the more the model is encouraged to focus on less similar positive sample pairs; Define the weights of negative sample pairs: ; The higher the weight The larger the value, the further away the negative sample pairs are; To prevent the weight distribution from being too extreme, the weights are normalized, and the final weight formula is defined as follows: ; In the supervised comparison loss formula, a dynamic weighting mechanism is added so that the contribution of each sample pair changes according to its weight. The new loss function is finally defined as follows: .

[0019] Specifically, the loss function of the classifier is the cross-entropy loss: in The number of samples in the batch. For the number of categories, It is a sample The true category label, The class probability predicted by the classifier: ; A classifier is learned in a shared feature space, and the feature representation is further optimized so that it has direct discriminative ability for classification tasks; Combining supervised contrast loss and classification loss, the final complete total loss function is expressed as: 。

[0020] Furthermore, to further enhance the effectiveness of contrastive learning, a projection head is added after CNN feature extraction; the projection head is a small neural network that maps features into a shared feature space. It contains two fully connected layers; projected features and Used to calculate contrastive loss: ; To ensure the stability and numerical performance of contrastive learning, the features are normalized.

[0021] Based on the same inventive concept, this invention provides a cross-modal feature space mapping and missing mode transfer device based on contrastive learning. The device is applied to fault detection and includes the following modules: Data acquisition module: configured to acquire raw vibration data and raw thermal imaging data; Dataset construction module: configured to construct a source domain dataset and set labels using the raw vibration data and raw thermal imaging data; Construct a target domain dataset using the raw vibration data; Source domain model processing module: configured as Extract multimodal features from the source domain dataset: Features of vibration data and thermal imaging data are extracted using 1D-CNN and 2D-CNN respectively, and then the self-attention mechanism of 1D-CNN and 2D-CNN is applied. Comparative learning: Based on the labels, a contrastive loss function is used to bring together multimodal features of the same category and push away features of different categories, ultimately constructing a shared feature space to ensure semantic consistency. Classifier training: The classifier is trained using the label information of the source domain dataset. By maximizing the matching probability between shared features and their corresponding labels, the classifier receives the shared features after feature extraction and projection as input, and optimizes the loss function of the classification task through backpropagation, thereby learning the discrimination boundary that can effectively distinguish different categories. Target domain transfer module: configured for target domain feature mapping and classification. After the vibration data in the target domain dataset is processed by 1D-CNN to extract features, it is directly mapped to the shared feature space, and the target domain features are classified using a classifier trained in the source domain. Evaluation module: configured to output fault diagnosis results and calculate evaluation indicators based on the target domain feature mapping and classification results.

[0022] Based on the same inventive concept, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described above.

[0023] Based on the same inventive concept, the present invention provides a non-transitory computer-readable storage medium that stores computer instructions for causing a computer to perform the method described above.

[0024] The beneficial effects of this invention are: (1) Efficiently solve the problem of fault diagnosis of modality missing: Through cross-modal feature mapping and modality completion technology, achieve efficient knowledge transfer and classification in the case of modality missing in the target domain.

[0025] (2) Improved cross-modal feature alignment performance: By using a category-aware contrastive learning method, the semantic consistency of cross-modal features in the shared space is guaranteed, thereby improving classification accuracy.

[0026] (3) Effective mitigation of class imbalance problem: Improve the accuracy and robustness of minority class fault classification by using class-weighted contrastive learning method. Attached Figure Description

[0027] The present invention includes the following figures: Figure 1 Flowchart of an embodiment of the present invention; Figure 2 A schematic diagram illustrating the principle of self-attention mechanism; Figure 3 A schematic diagram of a cross-modal feature space mapping and missing modality transfer method based on contrastive learning. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0029] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by those skilled in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word covers the element or object listed after the word and its equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly. First, the terms in this invention are explained as follows: 1) Fault diagnosis: This refers to the process of identifying, locating, and assessing the occurrence and impact of faults by analyzing the operating status of a system or equipment. It is a crucial link in industrial automation, equipment maintenance, and quality management, aiming to ensure the normal operation of equipment and reduce economic losses and safety risks caused by faults.

[0030] 2) Multimodal: This refers to a method or research field that utilizes data or information from multiple modalities to accomplish a task. A modality refers to different forms of data representation or perception, such as text, images, audio, video, and time series. The goal of multimodal technology is to achieve more accurate and comprehensive analysis and understanding by integrating information from multiple modalities.

[0031] 3) Missing modalities: This is a key problem in multimodal learning, referring to situations where all or part of the data for certain modalities is missing in a multimodal task. This can be caused by equipment failure, limitations in data acquisition conditions, privacy protection, etc. In practical applications, missing modalities can lead to information loss and degraded model performance, thus requiring specialized technical methods to address them.

[0032] 4) Domain alignment: In multimodal or cross-domain learning, this refers to methods that eliminate the distributional differences between data from different modalities or domains in order to enable effective learning and inference in a common feature space.

[0033] In view of this, such as Figure 1 As shown, this invention proposes a cross-modal feature space mapping and missing modality transfer method based on contrastive learning. The source domain has two modalities: vibration signal and thermal imaging, both labeled. The target domain has only one modality: vibration signal. After feature extraction using CNN, the two modalities in the source domain are classified according to their labels, mapping the features to the same feature space. A contrastive learning method is used to minimize the distance between modalities of the same category and maximize the distance between different categories, forming a single feature space for classification. In the target domain, the vibration signal is extracted using 1D-CNN, mapped to the same feature space, and then classified.

[0034] In the dataset construction section, the source domain contains data from two modalities: vibration signals and thermal images, both labeled. The target domain contains only vibration signal data, requiring a classification task to be performed without labels.

[0035] The core issue is how to map two modalities into the same feature space to ensure semantic consistency in feature representation. It also involves learning from the source domain to transfer relationships between categories to the target domain, enabling the target domain, which only contains vibration signals, to perform fault classification tasks using knowledge derived from existing knowledge.

[0036] Based on this, such as Figure 1 and Figure 3 As shown, the specific implementation steps of this embodiment are as follows: S1: Acquire raw vibration data and raw thermal imaging data.

[0037] S2: Construct a target domain dataset using the original vibration data.

[0038] The method for constructing the source domain dataset is as follows: Time window processing, label setting, data normalization, and building multi-source heterogeneous datasets; The method for constructing the target domain dataset is as follows: Time window processing and data standardization.

[0039] S3: Source domain multimodal feature extraction.

[0040] This invention proposes a multimodal feature alignment and transfer learning approach. First, features of vibration signals and thermal images are extracted using 1D-CNN and 2D-CNN, respectively. Then, a self-attention mechanism is used.

[0041] S3.1: Feature extraction.

[0042] (1) 1D-CNN For vibration signal data, 1D-CNN is used for feature extraction: where, It is the first The output features of the layer and It is the first The convolutional kernel weights and biases of the layer, where K is the kernel size. Represents the convolution operation. These are the output features of a 1D-CNN. Indicates the number of floors. It is the first Layer output features, initial input features .

[0043] ; ; (2) 2D-CNN For 2D-CNN models that process image data: where, Indicates the number of floors. and It is the first The kernel weights and biases of the layers. and Represents the height and width of the convolution kernel. This represents a 2D convolution operation. It is the first Layer output features, initial input features .

[0044] ; ; S3.2: Through self-attention mechanisms. (e.g.) Figure 2 (As shown) (1) Self-attention mechanism of 1D-CNN For the output features of 1D-CNN By calculating the attention weight matrix between features, self-attention enhancement is achieved. Alignment of the feature space for self-attention enhancement better captures semantic consistency between modalities, providing more robust transfer capabilities for target domain tasks. First, a mapping of query (Q), key (K), and value (V) is established, where... It is a learnable projection matrix. The dimension is the projected dimension.

[0045] ; Calculate the weights of attention, where Indicate the correlation between features: ; Weighted output: ; Integrating original features, among which These are weighting coefficients: ; (2) Self-attention mechanism of 2D-CNN For the output features of 2D-CNN Self-attention enhancement is achieved by calculating the attention weight matrix between features. First, the features are reshaped into matrix form. ,in The number of features is the number after flattening. Then, the mapping of query (Q), key (K), and value (V) is obtained through projection, where... , is a learnable projection matrix. The dimension is the projected dimension.

[0046] ; Calculate the weights of attention, where Indicates the correlation between pixels: ; Weighted output: ; Reshape and integrate the original features, in which These are weighting coefficients: ; S4: Comparative learning.

[0047] Through contrastive learning, based on labels, a contrastive loss function is used to bring together multimodal features of the same category, namely the features of vibration signals and thermal images of the same category, while pushing away features of different categories, ultimately constructing a shared feature space to ensure semantic consistency.

[0048] Contrastive learning optimizes the feature space by constructing positive and negative pairs. The goal is to make features of samples from the same class more similar and features of samples from different classes more dissimilar, thereby establishing semantic consistency in the feature space. Features from both modalities are mapped to a shared feature space through a shared mapping module. and Through projection mapping Mapping the features of the two modalities to the same feature space : ; Positive sample pairs consist of features from the same category (such as vibration signal features and thermal imaging features of different modalities but the same category), while negative sample pairs consist of features from different categories. Cosine similarity or Euclidean distance is typically used to measure the similarity between two features. The goal is to maximize the similarity between positive sample pairs and minimize the similarity between negative sample pairs.

[0049] There are two common loss functions for contrastive learning: Contrastive Loss and Supervised Contrastive Loss (SCL). In the source domain, the data is labeled, and the label information can be used to construct more explicit positive and negative sample pairs. The preliminary definition of the loss function formula is as follows: ; in, It is the set of indices of all samples in the current batch. Is with the sample A set of positive samples belonging to the same category. In addition to the batch The set of all samples other than itself, that is, the set of positive and negative samples. express and The similarity is measured using cosine similarity. It's a temperature parameter that adjusts the sensitivity of similarity. Smaller... It amplifies the similarity difference between positive and negative samples, making the model pay more attention to bringing the features of positive samples closer together; a larger A smooth distribution is suitable for more complex feature distributions.

[0050] ; and For positive sample pairs, calculate similarity. By maximizing the molecular It narrows the distance between samples of the same category. and Features belonging to negative sample pairs or other positive sample pairs are obtained by minimizing the features in the denominator. Push them further apart. Denominator Indicates target sample The sum of similarities with all samples in the batch. The logarithm helps reduce the gradient of large similarity values, improving the numerical stability of the optimization. The logarithmic function can transform the similarity ratio into an additive form (easier to optimize).

[0051] Positive and negative sample pairs are constructed using labels from the source domain, explicitly guiding feature space alignment. Ultimately, supervised contrastive loss optimizes the shared feature space, bringing similar features closer together and dissimilar features further apart.

[0052] In another embodiment, the contrastive learning dynamic weight mechanism is as follows: In standard contrastive learning, the contributions of all positive and negative sample pairs are treated uniformly. However, in real-world industrial scenarios, the similarity between different sample pairs can vary significantly. In this case, the differences between the vibration signal and the thermal imaging images across multiple modes may affect the effectiveness of contrastive learning. Therefore, this invention introduces a dynamic weighting mechanism, assigning different weights to each sample pair and dynamically adjusting their contribution to the loss.

[0053] For positive sample pairs Similarity of sample pairs The lower the value, the greater the weight, and the more attention is paid to less similar positive sample pairs. For negative sample pairs... Similarity of sample pairs The higher the value, the greater the weight, thus focusing more on negative sample pairs that are easily confused.

[0054] Define dynamic weights For sample pairs The loss contribution is weighted and adjusted, and the weights of positive sample pairs are initially defined as follows: ; The lower the weight The larger the value, the more the model is encouraged to focus on less similar positive sample pairs. The initial definition of the weights for negative sample pairs is as follows: ; The higher the weight The larger the value, the more important it is to push the negative sample pairs further away. To prevent the weight distribution from becoming too extreme, the weights are normalized, and the final weight formula is defined as follows: ; In the supervised comparison loss formula, a dynamic weighting mechanism is incorporated, causing the contribution of each sample pair to vary according to its weight. The new loss function is ultimately defined as follows: ; S5: Classifier Training Classifier construction: In the shared feature space, we train the classifier using the label information of the source domain data. By maximizing the matching probability between shared features and their corresponding labels, the classifier can accurately capture the category features of the source domain data. Specifically, the classifier receives the shared features after feature extraction and projection as input, and optimizes the loss function of the classification task through backpropagation, thereby learning the discrimination boundary that can effectively distinguish different categories. In this way, the classifier not only understands the feature distribution of the source domain, but also provides a foundation for subsequent data classification in the target domain.

[0055] Loss function: The classification loss is the cross-entropy loss, where The number of samples in the batch. For the number of categories, It is a sample The true category label, It is the class probability predicted by the classifier.

[0056] ; A classifier is learned in a shared feature space, and the feature representation is further optimized to give it direct discriminative ability for classification tasks. Combining supervised contrastive loss and classification loss, the final complete total loss function can be expressed as: ; S6: Target Domain Feature Mapping and Classification.

[0057] After the vibration signal in the target domain is processed by 1D-CNN to extract features, it is directly mapped into the shared feature space, and the target domain features are classified using a classifier trained in the source domain.

[0058] In another embodiment, to further enhance the effect of contrastive learning, a projection head is typically added after CNN feature extraction.

[0059] The projection head is a small neural network that maps features to a shared feature space. It contains two fully connected layers. The shared features have a lower dimensionality than the original features, reducing computational complexity. Projected features and Used to calculate contrast loss.

[0060] ; To ensure the stability and numerical performance of contrastive learning, the features are normalized. After normalization, all features are mapped onto a unit sphere, which facilitates the calculation of cosine similarity.

[0061] S7: Evaluation: Output fault diagnosis results and calculate evaluation indicators.

[0062] The key point of this invention is: Cross-modal contrastive learning optimization. Improvements and innovations: Building upon existing contrastive learning, a category-aware contrastive loss is added, making samples of the same class more aggregated in the cross-modal feature space and further distancing samples of different classes. Simultaneously, for cases where the target domain modality is missing, the target domain classification ability is optimized through the characteristics of single-modal features. Technical highlights: Ensures semantic consistency between different modalities, improves feature alignment, and enhances transfer capabilities.

[0063] Knowledge transfer in the absence of modalities. In a target domain containing only a single vibration signal mode without labels, efficient transfer of knowledge between modalities is achieved through learned feature space relationships.

[0064] Imbalanced Class Handling Mechanism: To address the imbalanced class of fault data between the source and target domains, a weighted contrastive learning loss function is used to improve the classification performance of minority class faults.

[0065] Based on the same inventive concept, and corresponding to any of the above embodiments, this invention proposes a cross-modal feature space mapping and missing modality transfer device based on contrastive learning, comprising the following modules: Data acquisition module: configured to acquire raw vibration data and raw thermal imaging data; Dataset construction module: configured to construct a source domain dataset and set labels using the raw vibration data and raw thermal imaging data; Construct a target domain dataset using the raw vibration data; Source domain model processing module: configured as Extract multimodal features from the source domain dataset: Features of vibration data and thermal imaging data are extracted using 1D-CNN and 2D-CNN respectively, and then the self-attention mechanism of 1D-CNN and 2D-CNN is applied. Comparative learning: Based on the labels, a contrastive loss function is used to bring together multimodal features of the same category and push away features of different categories, ultimately constructing a shared feature space to ensure semantic consistency. Classifier training: The classifier is trained using the label information of the source domain dataset. By maximizing the matching probability between shared features and their corresponding labels, the classifier receives the shared features after feature extraction and projection as input, and optimizes the loss function of the classification task through backpropagation, thereby learning the discrimination boundary that can effectively distinguish different categories. Target domain transfer module: configured for target domain feature mapping and classification. After the vibration data in the target domain dataset is processed by 1D-CNN to extract features, it is directly mapped to the shared feature space, and the target domain features are classified using a classifier trained in the source domain. Evaluation module: configured to output fault diagnosis results and calculate evaluation indicators.

[0066] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware.

[0067] The apparatus described above is used to implement the deployment method of the corresponding server in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0068] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the server deployment method described in any of the above embodiments.

[0069] This embodiment also provides a more specific electronic device hardware structure, which may include: a processor, a memory, an input / output interface, a communication interface, and a bus. The processor, memory, input / output interface, and communication interface are interconnected within the device via the bus. The processor may be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0070] The memory can be implemented as ROM (Read Only Memory), RAM (Random Access Memory), static storage device, or dynamic storage device. The memory can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory and called and executed by the processor.

[0071] Input / output interfaces are used to connect input / output modules to enable information input and output. Input / output modules can be configured as components within a device or connected externally to the device to provide corresponding functions. Input devices can include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices can include displays, speakers, vibrators, indicator lights, etc.

[0072] The communication interface is used to connect the communication module to enable communication and interaction between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0073] A bus is a pathway that transmits information between various components of a device, such as processors, memory, input / output interfaces, and communication interfaces.

[0074] It should be noted that although the above-described device only shows the processor, memory, input / output interface, communication interface, and bus, in actual implementation, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0075] The electronic devices described above are used to implement the deployment method of the corresponding server in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0076] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a non-transitory computer-readable storage medium that stores computer instructions for causing the computer to execute the server deployment method as described in any of the above embodiments.

[0077] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0078] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the server deployment method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0079] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and intended to imply that the scope of this application (including the claims) is limited to these examples; within the scope of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in the details for the sake of brevity. Furthermore, for the sake of simplicity and discussion, and to avoid obscuring the embodiments of this application, well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Additionally, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0080] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0081] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. A method for cross-modal feature space mapping and missing mode transfer based on contrastive learning, the method being applied to fault detection, characterized in that, Includes the following steps: Acquire raw vibration data and raw thermal imaging data; The original vibration data and original thermal imaging data are used to construct a source domain dataset and set labels; Construct a target domain dataset using the raw vibration data; Extract multimodal features from the source domain dataset: Features of vibration data and thermal imaging data are extracted using 1D-CNN and 2D-CNN respectively, and then the self-attention mechanism of 1D-CNN and 2D-CNN is applied. Comparative learning: Based on the labels, a contrastive loss function is used to bring together multimodal features of the same category and push away features of different categories, ultimately constructing a shared feature space to ensure semantic consistency. Classifier training: The classifier is trained using the label information of the source domain dataset. By maximizing the matching probability between shared features and their corresponding labels, the classifier receives the shared features after feature extraction and projection as input, and optimizes the loss function of the classification task through backpropagation, thereby learning the discrimination boundary that can effectively distinguish different categories. Target domain feature mapping and classification: After the vibration data in the target domain dataset is processed by 1D-CNN to extract features, it is directly mapped to the shared feature space, and the target domain features are classified using a classifier trained in the source domain. Evaluation: Based on the target domain feature mapping and classification results, output the fault diagnosis results and calculate the evaluation index; The comparative learning specifically refers to: The features of the two modalities are mapped to a shared feature space through a shared mapping module. and ; Through projection mapping Mapping the features of the two modalities to the same feature space : ； Positive sample pairs are sample features from the same class, while negative sample pairs are sample features from different classes. Cosine similarity or Euclidean distance is used to measure the similarity between two features. The goal is to maximize the similarity between positive sample pairs and minimize the similarity between negative sample pairs. The contrastive learning employs a dynamic weighting mechanism: For positive sample pairs Similarity of sample pairs The lower the value, the greater the weight, and the more attention is paid to less similar positive sample pairs; For negative sample pairs Similarity of sample pairs The higher the value, the greater the weight, thus focusing more on negative sample pairs that are easily confused; The specific method is as follows: Define dynamic weights For sample pairs The loss contribution is weighted and adjusted, and the weights of positive sample pairs are initially defined as follows: ； The lower the weight The larger the value, the more the model is encouraged to focus on less similar positive sample pairs; Define the weights of negative sample pairs: ； The higher the weight The larger the value, the further away the negative sample pairs are; To prevent the weight distribution from being too extreme, the weights are normalized, and the final weight formula is defined as follows: ； In the supervised comparison loss formula, a dynamic weighting mechanism is added so that the contribution of each sample pair changes according to its weight. The new loss function is finally defined as follows: 。 2. The method according to claim 1, characterized in that, The method for constructing the source domain dataset is as follows: Time window processing, label setting, data normalization, and building multi-source heterogeneous datasets; The method for constructing the target domain dataset is as follows: Time window processing and data standardization.

3. A cross-modal feature space mapping and missing mode transfer device based on contrastive learning, the device being applied to fault detection, characterized in that, To implement the method as described in claim 1, the method includes the following modules: Data acquisition module: configured to acquire raw vibration data and raw thermal imaging data; Dataset construction module: configured to construct a source domain dataset and set labels using the raw vibration data and raw thermal imaging data; Construct a target domain dataset using the raw vibration data; Source domain model processing module: configured as Extract multimodal features from the source domain dataset: Features of vibration data and thermal imaging data are extracted using 1D-CNN and 2D-CNN respectively, and then the self-attention mechanism of 1D-CNN and 2D-CNN is applied. Comparative learning: Based on the labels, a contrastive loss function is used to bring together multimodal features of the same category and push away features of different categories, ultimately constructing a shared feature space to ensure semantic consistency. Classifier training: The classifier is trained using the label information of the source domain dataset. By maximizing the matching probability between shared features and their corresponding labels, the classifier receives the shared features after feature extraction and projection as input, and optimizes the loss function of the classification task through backpropagation, thereby learning the discrimination boundary that can effectively distinguish different categories. Target domain transfer module: configured for target domain feature mapping and classification. After the vibration data in the target domain dataset is processed by 1D-CNN to extract features, it is directly mapped to the shared feature space, and the target domain features are classified using a classifier trained in the source domain. Evaluation module: configured to output fault diagnosis results and calculate evaluation indicators based on the target domain feature mapping and classification results.

4. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 2.

5. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method described in any one of claims 1 to 2.

Images