Cross-modal feature space mapping and missing modal migration method and device based on comparative learning, electronic equipment and storage medium

Abstract

The invention discloses a cross-modal feature space mapping and missing modal migration method and device based on comparative learning, electronic equipment and a storage medium. According to the method, vibration and thermal imaging modes in a source domain are classified according to labels after CNN feature extraction, features are mapped to the same feature space, a comparative learning method is used, the distance of the same category of different modes is shortened as much as possible, the distance of different categories is farther as much as possible, a feature space is formed, and the feature space is extracted. And then classification is carried out through the feature space. In a target domain, after 1D-CNN extraction is carried out on vibration signals, the vibration signals are mapped to the same feature space, and then classification is carried out, so that feature expression has semantic consistency. Through cross-modal feature mapping and modal completion technologies, efficient knowledge migration and classification under the condition of target domain modal deficiency are realized. And a class perception contrast learning method is utilized to ensure semantic consistency of the cross-modal features in the shared space, and the classification precision is improved. And through a class weighted contrast learning method, the accuracy and robustness of minority class fault classification are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of parts fault diagnosis, and in particular to a cross-modal feature space mapping and missing mode migration method and device based on contrastive learning. Background Art

[0002] In industrial scenarios, a wealth of data exists across multiple modalities, including vibration signals, acoustic emission signals, thermal images, and RGB images. A single sensor provides limited information, making it difficult to accurately assess the health of equipment. Multimodal data provides insights into equipment status from diverse perspectives. Therefore, when one or more types of modal monitoring data are missing, the existing classification model suffers from inadequate information, resulting in reduced classification accuracy and impacting the operation, maintenance, and management of a company's production line.

[0003] With the continuous development of science and technology, multimodal technology has become an important field of artificial intelligence. This technology can use the powerful generalization ability of deep learning to discover complex patterns and relationships that single-modal systems may not be able to detect, thereby promoting work in multiple fields. However, multimodal systems in the real world often face the challenge of dealing with situations where certain data modalities may be missing or incomplete, which seriously affects model performance. Therefore, we call the challenge of dealing with missing modalities in multimodal learning the "missing modality problem" and the solution to this problem multimodal learning with missing modalities. Multimodal learning with missing modalities often focuses on information retrieval, remote sensing, robotic vision, medical diagnosis, sentiment analysis, and multi-view clustering. The existing technology has the following shortcomings: 1) The problem of missing modalities has not been effectively addressed. Existing fault diagnosis methods for smart manufacturing plants typically assume that all sensor data or modal information is complete. However, in practice, some modalities may be missing due to factors such as sensor failure, incomplete signal acquisition, or funding and site limitations. This results in degraded performance of existing fault diagnosis models, poor generalization, and an inability to adapt to operating conditions with incomplete modalities.

[0004] 2) Cross-modal features are difficult to align and share. Existing methods lack effective cross-modal feature alignment and sharing mechanisms when processing multimodal data.

[0005] 3) Data insufficiency and class imbalance are difficult to solve. Existing methods lack effective cross-modal feature alignment and sharing mechanisms when processing multimodal data. Summary of the Invention

[0006] The present invention aims to solve the following problems: (1) The problem of missing modes in smart manufacturing factories. Due to problems such as signal loss, sensor failure, and insufficient funds and space, some modes are missing in smart manufacturing factories, resulting in a decrease in the accuracy and efficiency of fault diagnosis models.

[0007] (2) Problems with cross-modal feature alignment and sharing. In fault diagnosis in smart manufacturing plants, data often comes from different modalities (such as vibration signals and thermal images). However, data from different modalities differ in expression and distribution, making it difficult to align features and to directly fuse and share them.

[0008] (3) The problem of insufficient training data during the construction of smart manufacturing factories in small and medium-sized enterprises. Fault diagnosis model training requires mature datasets, which have long collection cycles and serious class imbalance problems. Small and medium-sized enterprises lack the conditions to build datasets for newly built smart factories. Using existing labeled multimodal datasets to complete the fault diagnosis task of unlabeled data solves the problem of insufficient data.

[0009] In the process of multimodal fault diagnosis, the problem of missing modes is addressed due to sensor failure, insufficient funds, limited space, missing signals, etc. In the case of missing single modes or partial modes, this invention can efficiently and lightly complete the classification task, determine the health status of the equipment, and provide protection for avoiding production safety accidents.

[0010] The present invention provides a cross-modal feature space mapping and missing mode migration method based on contrastive learning, which is applied to fault detection and includes the following steps: Obtain raw vibration data and raw thermal imaging data; Using the original vibration data and the original thermal imaging data to construct a source domain dataset and set labels; constructing a target domain dataset using the original vibration data; Extract multimodal features of the source domain dataset: The features of vibration data and thermal imaging data are extracted through 1D-CNN and 2D-CNN respectively, and then the self-attention mechanism of 1D-CNN and 2D-CNN is used; Contrastive Learning: Based on the labels, a contrast loss function is used to bring multimodal features of the same category closer together and push features of different categories further apart, ultimately constructing a shared feature space to ensure semantic consistency. Classifier training: The classifier is trained using the label information of the source domain dataset. By maximizing the matching probability between the shared features and their corresponding labels, the classifier receives the extracted and projected shared features as input and optimizes the loss function of the classification task through backpropagation, thereby learning a discriminant boundary that can effectively distinguish different categories. Target domain feature mapping and classification: The vibration data in the target domain dataset is directly mapped to the shared feature space after feature extraction by 1D-CNN, and the target domain features are classified using the classifier trained in the source domain; Evaluation: Output fault diagnosis results and calculate evaluation indicators.

[0011] Specifically, the method for constructing the source domain dataset is: Time window processing, label setting, data normalization, and building multi-source heterogeneous data sets; The target domain dataset is constructed as follows: Time window processing and data normalization.

[0012] Specifically, for the original vibration data, 1D-CNN is used to extract features using the following method: ; ; in, It is The output features of the layer, and It is The convolution kernel weights and biases of the layer, K is the size of the kernel, represents the convolution operation, is the output feature of 1D-CNN; Indicates the number of layers, It is Output features of the layer, initial input features ; For the raw thermal imaging data, 2D-CNN is used to extract features using the following method: ; ; in, Indicates the number of layers, and It is The convolution kernel weights and biases of the layer, and Represents the height and width of the convolution kernel, Represents a 2D convolution operation; It is Output features of the layer, initial input features .

[0013] Specifically, the self-attention mechanism of 1D-CNN is as follows: first, the mapping of query (Q), key (K) and value (V) is performed, where is a learnable projection matrix, is the dimension after projection; ; Calculate the attention weight, where Represents the correlation between features: ; Weighted output: ; Fusion of original features, where is the weight coefficient: ; The self-attention mechanism method of 2D-CNN is: First, reshape the features into matrix form, ,in is the number of features after flattening; Then the mapping of query (Q), key (K) and value (V) is obtained by projection, where is a learnable projection matrix, is the dimension after projection; ; Calculate the attention weight, where Represents the correlation between pixels: ; Weighted output: ; Reshape and integrate the original features, is the weight coefficient: .

[0014] Specifically, the contrastive learning is as follows: The features of the two modalities are mapped to a shared feature space through a shared mapping module and ; Through projection mapping , mapping the features of the two modalities into the same feature space : ; Among them, the positive sample pairs are sample features from the same category, and the negative sample pairs are sample features from different categories. The cosine similarity or Euclidean distance is used to measure the similarity of the two features. The goal is to maximize the similarity between the positive sample pairs and minimize the similarity between the negative sample pairs.

[0015] Specifically, the contrastive learning loss function is: ; in, is the index set of all samples in the current batch, Is with the sample A set of positive samples belonging to the same category, In addition to the batch All samples other than itself, that is, the collection of positive samples and negative samples; express and The similarity is measured using cosine similarity. is the temperature parameter, which adjusts the sensitivity of similarity; in, ; and Belong to the positive sample pair, calculate the similarity , by maximizing the Narrow the distance between samples of the same category; and Belong to the features of negative sample pairs or other positive sample pairs, by minimizing the denominator Push them further away; Denominator Represents the target sample The sum of similarities with all samples in the batch.

[0016] Furthermore, the contrastive learning adopts a dynamic weight mechanism: For the positive sample pair , the similarity of sample pairs The lower the value, the greater the weight, and the more attention is paid to the positive sample pairs that are not similar enough; For negative sample pairs , the similarity of sample pairs The higher it is, the greater the weight, thus paying more attention to negative sample pairs that are easily confused; Defining dynamic weights For sample pairs The loss contribution of is weighted and adjusted, and the weight of the positive sample pair is initially defined as follows: ; The lower the weight The larger the value, the more it encourages the model to focus on positive pairs that are not similar enough; Define the weights of negative sample pairs: ; The higher the weight The larger it is, the further away the negative sample pairs are pushed; In order to prevent the weight distribution from being too extreme, the weights are normalized and the weight formula is finally defined as follows: ; In the supervised contrast loss formula, a dynamic weight mechanism is added so that the contribution of each sample pair changes with its weight. The new loss function is finally defined as follows: 。

[0017] Specifically, the loss function of the classifier is cross entropy loss: in is the number of samples in the batch, is the number of categories, It is a sample The true category label is is the class probability predicted by the classifier: ; Learning classifiers in a shared feature space further optimizes feature representations to enable them to have direct discriminative capabilities for classification tasks; Combining the supervised contrast loss and the classification loss, the final complete total loss function is expressed as: .

[0018] Furthermore, in order to further enhance the effect of contrastive learning, a projection head is added after CNN feature extraction; the projection head is a small neural network that maps features into a shared feature space. Contains two layers of fully connected network; the features after projection and Used to calculate contrast loss: ; In order to ensure the stability and numerical effect of contrastive learning, the features are normalized.

[0019] Based on the same inventive concept, the present invention provides a device for cross-modal feature space mapping and missing mode migration based on contrastive learning. The device is applied to fault detection and includes the following modules: Data acquisition module: configured to acquire raw vibration data and raw thermal imaging data; A data set construction module is configured to construct a source domain data set using the original vibration data and the original thermal imaging data and set labels; constructing a target domain dataset using the original vibration data; Source domain model processing module: configured as Extract multimodal features of the source domain dataset: The features of vibration data and thermal imaging data are extracted through 1D-CNN and 2D-CNN respectively, and then the self-attention mechanism of 1D-CNN and 2D-CNN is used; Contrastive Learning: Based on the labels, a contrast loss function is used to bring multimodal features of the same category closer together and push features of different categories further apart, ultimately constructing a shared feature space to ensure semantic consistency. Classifier training: The classifier is trained using the label information of the source domain dataset. By maximizing the matching probability between the shared features and their corresponding labels, the classifier receives the extracted and projected shared features as input and optimizes the loss function of the classification task through backpropagation, thereby learning a discriminant boundary that can effectively distinguish different categories. Target domain transfer module: configured for target domain feature mapping and classification: The vibration data in the target domain dataset is directly mapped to the shared feature space after feature extraction by 1D-CNN, and the target domain features are classified using the classifier trained in the source domain; Evaluation module: configured to output fault diagnosis results and calculate evaluation indicators based on the target domain feature mapping and classification results.

[0020] Based on the same inventive concept, the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described above when executing the program.

[0021] Based on the same inventive concept, the present invention provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method described above.

[0022] Beneficial effects of the present invention: (1) Efficiently solve the fault diagnosis problem of missing modalities: Through cross-modal feature mapping and modal completion technology, efficient knowledge transfer and classification are achieved when the target domain modality is missing.

[0023] (2) Improvement of cross-modal feature alignment performance: Using category-aware contrastive learning methods to ensure the semantic consistency of cross-modal features in the shared space and improve classification accuracy.

[0024] (3) Effective mitigation of the class imbalance problem: Improve the accuracy and robustness of minority class fault classification through class-weighted contrastive learning methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The present invention has the following accompanying drawings: Figure 1Flowchart of an embodiment of the present invention; Figure 2 Schematic diagram of the self-attention mechanism principle; Figure 3 Schematic diagram of the model of cross-modal feature space mapping and missing modality transfer method based on contrastive learning. DETAILED DESCRIPTION

[0026] In order to make the objectives, technical solutions and advantages of this application more clear, this application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0027] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should have the usual meanings understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in the embodiments of the present application do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly. First, the terms in the present invention are explained as follows: 1) Fault diagnosis: This refers to the process of identifying, locating, and assessing the occurrence and impact of faults by analyzing the operating status of a system or device. It is a crucial component of industrial automation, equipment maintenance, and quality management, aiming to ensure the proper operation of equipment and reduce the economic losses and safety risks associated with faults.

[0028] 2) Multimodality: This refers to a method or research field that utilizes data or information from multiple modalities to accomplish tasks. Modality refers to the different ways data can be represented or perceived, such as text, images, audio, video, and time series. The goal of multimodal technology is to achieve more accurate and comprehensive analysis and understanding by integrating information from multiple modalities.

[0029] 3) Missing modality: This is a key issue in multimodal learning. It refers to the situation where all or part of the data for certain modalities is missing in a multimodal task. This situation may be caused by equipment failure, limited collection conditions, privacy protection, and other reasons. In practical applications, missing modality can lead to information loss and degraded model performance, necessitating specialized technical approaches to address it.

[0030] 4) Domain alignment: In multimodal learning or cross-domain learning, it refers to a method that eliminates the distribution differences between data from different modalities or domains so that effective learning and reasoning can be performed in a common feature space.

[0031] In view of this, if Figure 1 As shown, the embodiment of the present invention application proposes a cross-modal feature space mapping and missing modality migration method based on contrastive learning. The source domain has two modalities, vibration signals and thermal imaging images, and has labels, and the target domain has only one modality, vibration signals. After the two modalities of the source domain are extracted by CNN features, they are classified according to the labels, and the features are mapped to the same feature space. Using the contrastive learning method, the distance between the same category of different modalities is shortened as much as possible, and the distance between different categories is pushed as far as possible to form a feature space, and then classification is performed through this feature space. In the target domain, the vibration signal is extracted by 1D-CNN, mapped to the same feature space, and then classified.

[0032] In the dataset construction part, the source domain contains data in two modalities: vibration signals and thermal images, and both are labeled. The target domain only has vibration signal data, and the classification task needs to be completed without labels.

[0033] The core of the problem lies in how to map the two modalities into the same feature space so that the feature expressions are semantically consistent. Furthermore, how to transfer the relationship between categories to the target domain through learning in the source domain, so that the target domain, which only has vibration signals, can also use the source knowledge to complete the fault classification task.

[0034] Based on this, Figure 1 and Figure 3 As shown, the specific implementation steps of this embodiment are as follows: S1: Obtain raw vibration data and raw thermal imaging data.

[0035] S2: Construct a target domain dataset using the original vibration data.

[0036] The method for constructing the source domain dataset is as follows: Time window processing, label setting, data normalization, and building multi-source heterogeneous data sets; The target domain dataset is constructed as follows: Time window processing and data normalization.

[0037] S3: Source domain multimodal feature extraction.

[0038] This paper proposes a multimodal feature alignment and transfer learning approach. First, 1D-CNN and 2D-CNN are used to extract the features of vibration signals and thermal images respectively, and then a self-attention mechanism is used.

[0039] S3.1: Feature extraction.

[0040] (1) 1D-CNN For vibration signal data, 1D-CNN is used for feature extraction: It is The output features of the layer, and It is The convolution kernel weights and biases of the layer, K is the size of the kernel, represents the convolution operation, is the output feature of 1D-CNN. Indicates the number of layers, It is Output features of the layer, initial input features .

[0041] ; ; (2) 2D-CNN For the 2D-CNN model that processes image data: Indicates the number of layers, and It is The convolution kernel weights and biases of the layer, and Represents the height and width of the convolution kernel, Represents a 2D convolution operation. It is Output features of the layer, initial input features .

[0042] ; ; S3.2: Through the self-attention mechanism. (e.g. Figure 2 shown) (1) 1D-CNN’s self-attention mechanism For the output features of 1D-CNN By calculating the attention weight matrix between features, self-attention enhancement is achieved. By aligning the feature space enhanced by self-attention, the semantic consistency between modalities is better captured, providing more robust migration capabilities for target domain tasks. First, the mapping of query (Q), key (K) and value (V) is is a learnable projection matrix, is the dimension after projection.

[0043] ; Calculate the attention weight, where Represents the correlation between features: ; Weighted output: ; Fusion of original features, where is the weight coefficient: ; (2) Self-attention mechanism of 2D-CNN For the output features of 2D-CNN , by calculating the attention weight matrix between features, self-attention enhancement is achieved. First, the features are reshaped into a matrix form, ,in is the number of features after flattening. Then the mapping of query (Q), key (K) and value (V) is obtained by projection, where is a learnable projection matrix, is the dimension after projection.

[0044] ; Calculate the attention weight, where Represents the correlation between pixels: ; Weighted output: ; Reshape and integrate the original features, is the weight coefficient: ; S4: Contrastive learning.

[0045] Through contrastive learning, based on labels and using a contrast loss function, we bring multimodal features of the same category closer together, that is, the features of vibration signals and thermal images of the same category, and push features of different categories further apart. Ultimately, we construct a shared feature space to ensure semantic consistency.

[0046] Contrastive learning optimizes the feature space by constructing positive pairs and negative pairs. The goal is to make the features of samples of the same type closer and the features of samples of different types farther apart, thereby establishing semantic consistency in the feature space. The features of the two modalities are mapped to the shared feature space through a shared mapping module. and Through projection mapping , mapping the features of the two modalities into the same feature space : ; Positive pairs are sample features from the same category (e.g., vibration signal features and thermal imaging features from different modalities but the same category), while negative pairs are sample features from different categories. The similarity between the two features is typically measured using cosine similarity or Euclidean distance. The goal is to maximize the similarity between positive pairs and minimize the similarity between negative pairs.

[0047] There are two common contrastive learning loss functions: contrastive loss and supervised contrastive loss (SCL). In the source domain, the data is labeled, and label information can be used to construct clearer positive and negative sample pairs. The loss function formula is initially defined as follows: ; in, is the index set of all samples in the current batch, Is with the sample A set of positive samples belonging to the same category, In addition to the batch All samples other than itself, that is, the collection of positive samples and negative samples. express and The similarity is measured using cosine similarity. is the temperature parameter that adjusts the sensitivity of the similarity. It will amplify the similarity difference between positive and negative samples, making the model pay more attention to the characteristics of positive samples; larger The distribution is smooth and suitable for more complex feature distributions.

[0048] ; and Belong to the positive sample pair, calculate the similarity , by maximizing the Close the distance between samples of the same category. and Belong to the features of negative sample pairs or other positive sample pairs, by minimizing the denominator Push them further away. Represents the target sample The sum of similarities with all examples in the batch. Logarithms help reduce the gradient for large similarity values, improving the numerical stability of the optimization. The logarithmic function can convert the ratio of similarities into an additive form (which is easier to optimize).

[0049] Using the source domain’s labels to construct positive and negative sample pairs, we explicitly guide feature space alignment. Ultimately, the supervised contrastive loss optimizes the shared feature space, keeping similar features close together and dissimilar features apart.

[0050] In another embodiment, the contrastive learning dynamic weighting mechanism is as follows: In standard contrastive learning, the contributions of all positive and negative sample pairs are treated uniformly. However, in real industrial scenarios, the similarity between different sample pairs can vary significantly. In this case, the multimodal differences between the vibration signal and the thermal image may affect the effectiveness of contrastive learning. Therefore, this paper introduces a dynamic weighting mechanism that assigns different weights to each sample pair, dynamically adjusting its contribution to the loss.

[0051] For the positive sample pair , the similarity of sample pairs The lower the value, the greater the weight, and the more attention is paid to the positive sample pairs that are not similar enough. , the similarity of sample pairs The higher it is, the greater the weight, thus paying more attention to negative sample pairs that are easily confused.

[0052] Defining dynamic weights For sample pairs The loss contribution of is weighted and adjusted, and the weight of the positive sample pair is initially defined as follows: ; The lower the weight The larger the value, the more it encourages the model to focus on positive sample pairs that are not similar enough. The weights of negative sample pairs are preliminarily defined as follows: ; The higher the weight The larger it is, the more it emphasizes the need to push the negative sample pairs further. In order to prevent the weight distribution from being too extreme, the weights are normalized, and the weight formula is finally defined as follows: ; In the supervised contrast loss formula, a dynamic weight mechanism is added so that the contribution of each sample pair changes with its weight. The new loss function is finally defined as follows: ; S5: Classifier training Classifier construction: In the shared feature space, we leverage the label information of the source domain data to train a classifier. By maximizing the matching probability between shared features and their corresponding labels, the classifier accurately captures the categorical characteristics of the source domain data. Specifically, the classifier receives the extracted and projected shared features as input and optimizes the classification loss function through backpropagation, thereby learning a discriminant boundary that effectively distinguishes different categories. In this way, the classifier not only understands the feature distribution of the source domain but also provides a foundation for subsequent data classification in the target domain.

[0053] Loss function: The classification loss is the cross entropy loss, where is the number of samples in the batch, is the number of categories, It is a sample The true category label is is the class probability predicted by the classifier.

[0054] ; The classifier is learned in the shared feature space, and the feature representation is further optimized to make it have direct discriminative ability for classification tasks. Combining the supervised contrast loss and the classification loss, the final complete total loss function can be expressed as: ; S6: Target domain feature mapping and classification.

[0055] After the target domain vibration signal is extracted by 1D-CNN, it is directly mapped to the shared feature space, and the target domain features are classified using the classifier trained in the source domain.

[0056] In another embodiment, in order to further enhance the effect of contrastive learning, a projection head is usually added after CNN feature extraction.

[0057] The projection head is a small neural network that maps features into a shared feature space. Contains two layers of fully connected layers. The dimension of shared features is lower than the original features, which reduces the computational complexity. The projected features and Used to calculate contrastive loss.

[0058] ; In order to ensure the stability and numerical effect of contrastive learning, the features are normalized. After normalization, all features are mapped to the unit sphere to facilitate the calculation of cosine similarity.

[0059] S7: Evaluation: Output fault diagnosis results and calculate evaluation indicators.

[0060] The key points of the present invention are: Cross-modal contrastive learning optimization. Improvements and innovations: Building on existing contrastive learning, we add a category-aware contrastive loss, which allows samples of the same category to be more aggregated in the cross-modal feature space, further increasing the distance between samples of different categories. Furthermore, in the absence of a target domain modality, we leverage the characteristics of unimodal features to optimize target domain classification capabilities. Technical highlights: This ensures semantic consistency across modalities, improves feature alignment, and enhances transfer capabilities.

[0061] Knowledge transfer in the absence of modalities: When the target domain only has a single modality of vibration signals and no labels, efficient knowledge transfer between modalities is achieved through the learned feature space relationship.

[0062] Class imbalance processing mechanism: To address the problem of class imbalance in source and target domain fault data, a weighted contrastive learning loss function is used to improve the classification performance of minority class faults.

[0063] Based on the same inventive concept, corresponding to any of the above embodiments, the present invention proposes a cross-modal feature space mapping and missing modality migration device based on contrastive learning, comprising the following modules: Data acquisition module: configured to acquire raw vibration data and raw thermal imaging data; A data set construction module is configured to construct a source domain data set using the original vibration data and the original thermal imaging data and set labels; constructing a target domain dataset using the original vibration data; Source domain model processing module: configured as Extract multimodal features of the source domain dataset: The features of vibration data and thermal imaging data are extracted through 1D-CNN and 2D-CNN respectively, and then the self-attention mechanism of 1D-CNN and 2D-CNN is used; Contrastive Learning: Based on the labels, a contrast loss function is used to bring multimodal features of the same category closer together and push features of different categories further apart, ultimately constructing a shared feature space to ensure semantic consistency. Classifier training: The classifier is trained using the label information of the source domain dataset. By maximizing the matching probability between the shared features and their corresponding labels, the classifier receives the extracted and projected shared features as input and optimizes the loss function of the classification task through backpropagation, thereby learning a discriminant boundary that can effectively distinguish different categories. Target domain transfer module: configured for target domain feature mapping and classification: The vibration data in the target domain dataset is directly mapped to the shared feature space after feature extraction by 1D-CNN, and the target domain features are classified using the classifier trained in the source domain; Evaluation module: configured to output fault diagnosis results and calculate evaluation indicators.

[0064] For the convenience of description, the above devices are described as being divided into various modules according to their functions. Of course, when implementing this application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0065] The apparatus of the above embodiment is used to implement the corresponding server deployment method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0066] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the program, the server deployment method described in any of the above embodiments is implemented.

[0067] This embodiment also provides a more specific electronic device hardware structure, which may include: a processor, a memory, an input / output interface, a communication interface, and a bus. The processor, the memory, the input / output interface, and the communication interface are connected to each other within the device via a bus. The processor can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute related programs to implement the technical solutions provided in the embodiments of this specification.

[0068] The memory can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, or dynamic storage devices. The memory can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented using software or firmware, the relevant program code is stored in the memory and called and executed by the processor.

[0069] Input / output interfaces connect to input / output modules to enable information input and output. Input / output modules can be built into a device or connected externally to provide corresponding functionality. Input devices include keyboards, mice, touch screens, microphones, and various sensors. Output devices include displays, speakers, vibrators, and indicator lights.

[0070] The communication interface is used to connect the communication module to enable communication between the device and other devices. The communication module can achieve communication through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.).

[0071] The bus comprises a pathway that transmits information between various components of a device, such as a processor, memory, input / output interfaces, and communication interfaces.

[0072] It should be noted that although the above device only illustrates a processor, memory, input / output interface, communication interface, and bus, in a specific implementation, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will appreciate that the above device may only include components necessary to implement the embodiments of this specification, and need not include all of the components shown in the figures.

[0073] The electronic device of the above embodiment is used to implement the corresponding server deployment method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0074] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the server deployment method described in any of the above embodiments.

[0075] The computer-readable media of this embodiment include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0076] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the server deployment method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0077] Those skilled in the art will understand that the discussion of any of the above embodiments is merely exemplary and is intended to imply that the scope of the present application (including the claims) is limited to these examples. Within the context of the present application, the technical features of the above embodiments or different embodiments may also be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity. In addition, to simplify the description and discussion, and to avoid obscuring the embodiments of the present application, known power / ground connections to the integrated circuit (IC) chip and other components may or may not be shown in the provided figures. In addition, the devices may be shown in the form of block diagrams to avoid obscuring the embodiments of the present application, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present application will be implemented (i.e., these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present application, it will be apparent to those skilled in the art that the embodiments of the present application may be implemented without these specific details or with variations in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0078] Although the present invention has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art in light of the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may utilize the embodiments discussed.

[0079] The embodiments of the present application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of this application.

Claims

1. A cross-modal feature space mapping and missing mode migration method based on contrastive learning, the method is applied to fault detection, characterized in that: The steps include: Obtain raw vibration data and raw thermal imaging data; Using the original vibration data and the original thermal imaging data to construct a source domain dataset and set labels; constructing a target domain dataset using the original vibration data; Extract multimodal features of the source domain dataset: The features of vibration data and thermal imaging data are extracted through 1D-CNN and 2D-CNN respectively, and then the self-attention mechanism of 1D-CNN and 2D-CNN is used; Contrastive Learning: Based on the labels, a contrast loss function is used to bring multimodal features of the same category closer together and push features of different categories further apart, ultimately constructing a shared feature space to ensure semantic consistency. Classifier training: The classifier is trained using the label information of the source domain dataset. By maximizing the matching probability between the shared features and their corresponding labels, the classifier receives the extracted and projected shared features as input and optimizes the loss function of the classification task through backpropagation, thereby learning a discriminant boundary that can effectively distinguish different categories. Target domain feature mapping and classification: The vibration data in the target domain dataset is directly mapped to the shared feature space after feature extraction by 1D-CNN, and the target domain features are classified using the classifier trained in the source domain; Evaluation: Output fault diagnosis results and calculate evaluation indicators based on the target domain feature mapping and classification results.

2. The method according to claim 1, characterized in that The method for constructing the source domain dataset is as follows: Time window processing, label setting, data normalization, and building multi-source heterogeneous data sets; The target domain dataset is constructed as follows: Time window processing and data normalization.

3. The method according to claim 1, characterized in that For the original vibration data, 1D-CNN is used to extract features using the following method: ； ； in, It is The output features of the layer, and It is The convolution kernel weights and biases of the layer, is the size of the kernel, represents the convolution operation, is the output feature of 1D-CNN; Indicates the number of layers, It is Output features of the layer, initial input features ； For the raw thermal imaging data, 2D-CNN is used to extract features using the following method: ; ; in, Indicates the number of layers, and It is The convolution kernel weights and biases of the layer, and Represents the height and width of the convolution kernel, Represents a 2D convolution operation; It is Output features of the layer, initial input features .

4. The method according to claim 3, characterized in that The self-attention mechanism of 1D-CNN is as follows: first, the mapping of query (Q), key (K) and value (V) is performed, where is a learnable projection matrix, is the dimension after projection; ； Calculate the attention weight, where Represents the correlation between features: ； Weighted output: ； Fusion of original features, where is the weight coefficient: ； The self-attention mechanism method of 2D-CNN is: First, reshape the features into matrix form, ,in is the number of features after flattening; Then the mapping of query (Q), key (K) and value (V) is obtained by projection, where is a learnable projection matrix, is the dimension after projection; ； Calculate the attention weight, where Represents the correlation between pixels: ； Weighted output: ； Reshape and integrate the original features, is the weight coefficient: 。 5. The method according to claim 1, wherein The contrastive learning is specifically as follows: The features of the two modalities are mapped to a shared feature space through a shared mapping module and ; Through projection mapping , mapping the features of the two modalities into the same feature space : ； Among them, the positive sample pairs are sample features from the same category, and the negative sample pairs are sample features from different categories. The cosine similarity or Euclidean distance is used to measure the similarity of the two features. The goal is to maximize the similarity between the positive sample pairs and minimize the similarity between the negative sample pairs.

6. The method according to claim 5, characterized in that The contrastive learning loss function is specifically: ； in, is the index set of all samples in the current batch, Is with the sample A set of positive samples belonging to the same category, In addition to the batch All samples other than itself, that is, the collection of positive samples and negative samples; express and The similarity is measured using cosine similarity. is the temperature parameter, which adjusts the sensitivity of similarity; in, ; and Belong to the positive sample pair, calculate the similarity , by maximizing the Narrow the distance between samples of the same category; and Belong to the features of negative sample pairs or other positive sample pairs, by minimizing the denominator Push them further away; Denominator Represents the target sample The sum of similarities with all samples in the batch.

7. The method according to claim 5 or 6, characterized in that The contrastive learning adopts a dynamic weight mechanism: For the positive sample pair , the similarity of sample pairs The lower the value, the greater the weight, and the more attention is paid to the positive sample pairs that are not similar enough; For negative sample pairs , the similarity of sample pairs The higher it is, the greater the weight, thus paying more attention to negative sample pairs that are easily confused; The specific method is: Defining dynamic weights For sample pairs The loss contribution of is weighted and adjusted, and the weight of the positive sample pair is initially defined as follows: ； The lower the weight The larger the value, the more it encourages the model to focus on positive pairs that are not similar enough; Define the weights of negative sample pairs: ； The higher the weight The larger it is, the further away the negative sample pairs are pushed; In order to prevent the weight distribution from being too extreme, the weights are normalized and the weight formula is finally defined as follows: ； In the supervised contrast loss formula, a dynamic weight mechanism is added so that the contribution of each sample pair changes with its weight. The new loss function is finally defined as follows: 。 8. The method according to claim 1, characterized in that The loss function of the classifier is the cross entropy loss: in is the number of samples in the batch, is the number of categories, It is a sample The true category label is is the class probability predicted by the classifier: ； Learning classifiers in a shared feature space further optimizes feature representations to enable them to have direct discriminative capabilities for classification tasks; Combining the supervised contrast loss and the classification loss, the final complete total loss function is expressed as: 。 9. The method according to claim 1, characterized in that In the contrastive learning, a projection head is added after the CNN feature extraction; The projection head is a small neural network that maps features into a shared feature space. Contains two layers of fully connected layer network; the features after projection and Used to calculate contrast loss: ； In order to ensure the stability and numerical effect of contrastive learning, the features are normalized.

10. A device for cross-modal feature space mapping and missing mode migration based on contrastive learning, the device being applied to fault detection, characterized in that: Includes the following modules: Data acquisition module: configured to acquire raw vibration data and raw thermal imaging data; A data set construction module is configured to construct a source domain data set using the original vibration data and the original thermal imaging data and set labels; constructing a target domain dataset using the original vibration data; Source domain model processing module: configured as Extract multimodal features of the source domain dataset: The features of vibration data and thermal imaging data are extracted through 1D-CNN and 2D-CNN respectively, and then the self-attention mechanism of 1D-CNN and 2D-CNN is used; Contrastive Learning: Based on the labels, a contrast loss function is used to bring multimodal features of the same category closer together and push features of different categories further apart, ultimately constructing a shared feature space to ensure semantic consistency. Classifier training: The classifier is trained using the label information of the source domain dataset. By maximizing the matching probability between the shared features and their corresponding labels, the classifier receives the extracted and projected shared features as input and optimizes the loss function of the classification task through backpropagation, thereby learning a discriminant boundary that can effectively distinguish different categories. Target domain transfer module: configured for target domain feature mapping and classification: The vibration data in the target domain dataset is directly mapped to the shared feature space after feature extraction by 1D-CNN, and the target domain features are classified using the classifier trained in the source domain; Evaluation module: configured to output fault diagnosis results and calculate evaluation indicators based on the target domain feature mapping and classification results.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 9 is implemented.

12. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 9.

Images