Method and device for training state monitoring image classification model, medium and equipment

Abstract

The invention discloses a state monitoring image classification model training method and device, a medium and equipment, and the method comprises the steps: carrying out the pre-training to obtain a backbone network of a state monitoring image classification model, extracting a first image feature of a source domain and a second image feature of a target domain, and constructing a similarity graph; according to the constructed similarity graph, obtaining a pseudo label of the target domain; and obtaining a total loss function according to the cross entropy loss of the source domain, the cross entropy loss of the target domain and the graph embedding loss, and carrying out back propagation on the backbone network according to the total loss function to obtain the state monitoring image classification model. On the basis of an optimal graph self-learning strategy, two different types of similarity graphs are obtained through learning so as to fully mine structural information of respective fields, and the problem that a label propagation algorithm is sensitive to distribution deviation is solved; and meanwhile, nuclear norm maximization constraint is introduced into a label propagation algorithm, so that the problem of data category imbalance is effectively solved.

Description

Technical Field

[0001] The present invention relates to the technical field of image classification, and in particular to a training method, device, medium and equipment for a state monitoring image classification model. Background Art

[0002] In the industrial field, deep learning technology has been widely used. The application of deep learning technology in the industrial field includes but is not limited to the following aspects: quality control and product inspection, fault diagnosis and predictive maintenance, production process optimization, supply chain management, intelligent manufacturing and automation, etc.

[0003] However, in actual industrial applications, the quality of the data source is a key factor that restricts the performance of deep learning technology. For example, in the condition monitoring process of industrial equipment such as belt conveyors, due to the influence of time, environment and human factors, there may be serious distribution deviation problems between the data used for model training and the test data. For example, there is a significant difference in light intensity in the condition monitoring pictures.

[0004] In addition, industrial data is usually highly unevenly distributed. The amount of data in certain categories or time periods may far exceed that in other categories or time periods, which can lead to category imbalance problems in the data set. For example, the amount of abnormal condition monitoring data actually collected is often very small.

[0005] Both of the above problems will lead to a serious degradation in the performance of the deep state monitoring algorithm. On the one hand, the distribution bias problem causes the model to overfit to the specific distribution of the training set, and seriously fail on the test set. On the other hand, the class imbalance problem causes the model to tend to identify most states as states corresponding to large categories, and fail to monitor states corresponding to small categories.

[0006] Therefore, there is an urgent need for a class-imbalanced deep cross-domain image classification model to overcome the above problems. Summary of the invention

[0007] In view of the above problems, the present invention is proposed to provide a training method, device, medium and equipment for a condition monitoring image classification model that overcomes the above problems or at least partially solves the above problems.

[0008] Other features and advantages of the present invention will become apparent from the following detailed description, or may be learned in part by practice of the present invention.

[0009] According to a first aspect of an embodiment of the present invention, a training method for a condition monitoring image classification model is provided. The condition monitoring image classification model is used for classifying condition monitoring images of industrial equipment. The training method for the condition monitoring image classification model includes:

[0010] Based on the collected state image data of industrial equipment, we construct the source domain dataset with real labels and the target domain dataset that are not labeled, which are required for the state monitoring image classification model.

[0011] Pre-training to obtain a backbone network of the condition monitoring image classification model, and respectively extracting first image features of the source domain and second image features of the target domain through the backbone network, and constructing a similarity graph based on the extracted first image features and second image features;

[0012] According to the constructed similarity graph, the real labels annotated in the source domain are transferred to the target domain through a label propagation algorithm based on nuclear norm maximization to obtain pseudo labels of the target domain;

[0013] Determine the cross entropy loss of the source domain according to the real label of the source domain and the label of the source domain predicted by the backbone network;

[0014] Determine the cross entropy loss of the target domain according to the pseudo label of the target domain and the target domain label predicted by the backbone network;

[0015] Using the annotated real labels in the source domain and the pseudo labels in the target domain, combined with the first image feature and the second image feature, the similarity graph is updated, and the graph embedding loss is calculated;

[0016] A total loss function is obtained according to the cross entropy loss of the source domain, the cross entropy loss of the target domain and the graph embedding loss, and the backbone network is back-propagated according to the total loss function to obtain the state monitoring image classification model.

[0017] In some embodiments of the present invention, the similarity graph includes a cross-domain similarity graph, a source domain similarity graph, and a target domain similarity graph, and constructing the similarity graph based on the extracted first image features and second image features includes:

[0018] Based on the extracted first image features and second image features, construct a cross-domain similarity graph from the source domain to the target domain;

[0019] constructing a source domain similarity graph based on any two of the first image features extracted in the source domain;

[0020] A target domain similarity graph is constructed based on any two of the second image features extracted in the target domain.

[0021] In some embodiments of the present invention, the similarity graph constructed is used to transfer the real labels annotated in the source domain to the target domain through a label propagation algorithm based on nuclear norm maximization to obtain pseudo labels of the target domain, and the pseudo labels of the target domain are obtained based on the following formula:

[0022]

[0023]

[0024] In the formula, and They are the cross-domain similarity graph and the target domain similarity graph respectively; and Represent the predicted labels of the i-th and j-th samples respectively; and The predicted label matrices representing the source domain and the target domain respectively; Y s The true label matrix representing the source domain; δ and σ are the hyperparameters of the corresponding items.

[0025] In some embodiments of the present invention, determining the cross entropy loss of the source domain according to the real label of the source domain and the source domain label predicted by the backbone network includes:

[0026] According to the real label of the source domain and the source domain label predicted by the backbone network, respectively determine the first real probability and the first predicted probability that the i-th real label of the source domain is classified into the c-th category, where i and c are positive integers;

[0027] The cross entropy loss of the source domain is determined according to the first true probability and the first predicted probability.

[0028] In some embodiments of the present invention, determining the cross entropy loss of the target domain according to the pseudo label of the target domain and the target domain label predicted by the backbone network includes:

[0029] According to the pseudo label of the target domain and the target domain label predicted by the backbone network, respectively determine the second true probability and the second predicted probability that the i-th true label of the target domain is classified into the c-th category, where i and c are positive integers;

[0030] The cross entropy loss of the target domain is determined according to the second true probability and the second predicted probability.

[0031] In some embodiments of the present invention, the method calculates the graph embedding loss by the following formula:

[0032]

[0033]

[0034]

[0035] In the formula, is the updated source domain similarity graph; is the target domain similarity graph; is a cross-domain similarity graph; m is the number of state image data in the source domain dataset; n is the number of state image data in the target domain dataset; Embedding loss function for source domain graph; Embedding loss function for target domain graph; Embedding loss function for cross-domain graphs; and are the image features of the i-th sample in the source domain and the j-th sample in the source domain respectively; and are the image features of the i-th sample in the target domain and the j-th sample in the target domain respectively; z i and z j It means that after considering the source domain and the target domain as a whole, the image features of the i-th sample and the j-th sample without distinguishing the domain labels.

[0036] In some embodiments of the present invention, the method further includes: updating the similarity graph based on the extracted first image features and second image features at intervals of a complete training cycle, and updating the pseudo-labels of the target domain using the updated similarity graph.

[0037] According to a second aspect of an embodiment of the present invention, a training device for a condition monitoring image classification model is provided, and the training device for the condition monitoring image classification model comprises:

[0038] A data acquisition module is used to construct a data set in the source field that has been annotated with real labels and a data set in the target field that has not been annotated, which are required for the condition monitoring image classification model, based on the collected state image data of the industrial equipment;

[0039] A feature processing module, used for pre-training a backbone network of the condition monitoring image classification model, extracting first image features of the source domain and second image features of the target domain through the backbone network, and constructing a similarity graph based on the extracted first image features and second image features;

[0040] A similarity graph construction module is used to transfer the annotated real labels in the source domain to the target domain through a label propagation algorithm based on nuclear norm maximization according to the constructed similarity graph, so as to obtain pseudo labels of the target domain;

[0041] A source domain loss calculation module, used to determine the cross entropy loss of the source domain according to the real label of the source domain and the source domain label predicted by the backbone network;

[0042] A target domain loss calculation module, used to determine the cross entropy loss of the target domain according to the pseudo label of the target domain and the target domain label predicted by the backbone network;

[0043] A graph embedding loss calculation module, used to update the similarity graph by using the annotated real labels in the source domain and the pseudo labels in the target domain, combined with the first image feature and the second image feature, and calculate the graph embedding loss;

[0044] The model parameter adjustment module is used to obtain a total loss function according to the cross entropy loss of the source domain, the cross entropy loss of the target domain and the graph embedding loss, and to back-propagate the backbone network according to the total loss function to obtain the state monitoring image classification model.

[0045] According to a third aspect of an embodiment of the present invention, a computer-readable storage medium is provided, in which computer program instructions are stored. The computer program instructions are loaded and executed by a processor to implement the operations performed by any of the methods described above.

[0046] According to a fourth aspect of an embodiment of the present invention, there is provided an electronic device, comprising a processor and a memory, wherein the memory stores computer program instructions that can be executed by the processor, and when the processor executes the computer program instructions, it implements instructions of any of the methods described above.

[0047] The technical solution provided in the embodiments of the present invention has at least the following technical effects or advantages:

[0048] The embodiments of the present invention provide a training method, device, medium and equipment for a condition monitoring image classification model. The method described in the embodiments of the present invention is based on an optimal graph self-learning strategy to learn and obtain two different types of similarity graphs to fully mine the structural information of their respective fields, thereby solving the problem that the label propagation algorithm is sensitive to distribution deviations; at the same time, a nuclear norm maximization constraint is introduced into the label propagation algorithm to increase the number of samples predicted into small classes and the classification accuracy of small class samples, thereby improving the diversity of prediction results and effectively solving the problem of data category imbalance; and by using cross entropy loss to supervise the network classification layer, the label propagation algorithm is introduced into a deep convolutional neural network to achieve the migration of the excellent image classification ability of the label propagation algorithm to the deep network model.

[0049] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 A schematic diagram of a flow chart of a training method for a condition monitoring image classification model provided by an embodiment of the present invention;

[0052] Figure 2 This is a schematic diagram of the backbone network architecture reference;

[0053] Figure 3(a) shows the prediction results of the standard label propagation algorithm LP;

[0054] Figure 3(b) shows the prediction results of the label propagation algorithm OGL for LP that introduces optimal graph learning;

[0055] Figure 3(c) shows the prediction results of the label propagation algorithms OGL and NNM forLP that simultaneously introduce optimal graph learning and nuclear norm maximization;

[0056] Figure 4 A schematic diagram of the principle structure of a training device for a condition monitoring image classification model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings.

[0058] Various structural schematic diagrams according to embodiments of the present disclosure are shown in the accompanying drawings. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may further design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0059] In the context of the present disclosure, when a layer / element is referred to as being "on" another layer / element, the layer / element may be directly on the other layer / element, or there may be an intervening layer / element therebetween. Additionally, if a layer / element is "on" another layer / element in one orientation, the layer / element may be "below" the other layer / element when the orientation is reversed. In the context of the present disclosure, similar or identical components may be denoted by the same or similar reference numerals.

[0060] In order to better understand the above-mentioned technical scheme, the above-mentioned technical scheme will be described in detail below in combination with specific implementation methods. It should be understood that the embodiments of the present disclosure and the specific features in the embodiments are detailed descriptions of the technical scheme of the present invention, rather than limitations on the technical scheme of the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0061] Figure 1 is a flow chart of a training method of a condition monitoring image classification model provided by an embodiment of the present invention, wherein the condition monitoring image classification model is used for classifying condition monitoring images of industrial equipment, such as Figure 1 As shown, the training method of the state monitoring image classification model includes the following steps:

[0062] S1. Based on the collected state image data of industrial equipment, a dataset with real labels in the source field and an unlabeled dataset in the target field required for the state monitoring image classification model are constructed respectively;

[0063] In the embodiment of the present invention, the industrial equipment may be a belt conveyor or other equipment. The state image data of the industrial equipment may be raw data collected in real time or directly obtained public training set data. A suitable data set may be selected according to actual needs.

[0064] In the embodiment of the present invention, the source domain dataset that has been annotated with real labels is defined as The unlabeled dataset of the target domain is defined as and Wherein, m is the number of state image data in the data set of the source domain; n is the number of state image data in the data set of the target domain.

[0065] S2, pre-training to obtain a backbone network of the state monitoring image classification model, and extracting the first image features of the source domain and the second image features of the target domain through the backbone network, and constructing a similarity graph based on the extracted first image features and second image features;

[0066] Combination Figure 2As shown in FIG. 1 , it is a schematic diagram for reference of the architecture of the backbone network. In an embodiment of the present invention, the backbone network is, for example, a ResNet-50 model or a ResNet-101 model; the backbone network includes at least a feature extractor. Fully connected layer Fc and network classification layer For example, the feature extractor can be used to extract the first image feature of the source domain from a data set of the source domain that has been annotated with real labels, and the feature extractor can be used to extract the second image feature of the target domain from an unlabeled data set of the target domain; for example, the network classification layer can be used for classification processing of the state image data.

[0067] Specifically, the source domain dataset that has been annotated with real labels and the target domain dataset that has not been annotated are processed by a feature extractor. Perform feature extraction processing to obtain the initialization image features and in Represents k-dimensional image features; then, initialize the image features and After the fully connected layer Fc, the image embedding features are obtained respectively and in represents the d-dimensional image embedding feature; finally, the image embedding feature and After the network classification layer Get the initialization network classification results respectively and in Represents the c-dimensional classification label; That is the first image feature, That is the second image feature.

[0068] The similarity graph includes a cross-domain similarity graph, a source domain similarity graph, and a target domain similarity graph. In the embodiment of the present invention, based on the extracted first image features and second image features, constructing the similarity graph includes:

[0069] Based on the extracted first image features and second image features, construct a cross-domain similarity graph from the source domain to the target domain;

[0070] constructing a source domain similarity graph based on any two of the first image features extracted in the source domain;

[0071] A target domain similarity graph is constructed based on any two of the second image features extracted in the target domain.

[0072] Specifically, the embodiment of the present invention constructs the similarity graph through the following formula:

[0073]

[0074]

[0075] In formula (1), z i and Represent image features from two different fields, y i and Represent image labels from two different fields, w ij ∈W represents the similarity between the i-th and j-th samples.

[0076] In formula (2), z i and z j Represent two image features from the same field, y i and j Represent two image labels from the same domain.

[0077] Represents the calculation of the Euclidean distance between two vectors. It represents the F norm of the calculation matrix W, and the 'st' is represented by the non-negative constraint and the normalization constraint. α1 and α2 are the hyperparameters of the corresponding items.

[0078] In the embodiment of the present invention, when only image features are considered, formula (1) makes the similarity between two samples with a close distance in the feature space larger, and the two samples are from different fields; formula (2) makes the similarity between two samples with a close distance in the feature space larger, and the two samples are from the same field. When features and labels are considered at the same time, formula (1) makes the similarity between two samples with a close distance in both the feature space and the label space larger, and the two samples are from different fields; formula (2) makes the similarity between two samples with a close distance in both the feature space and the label space larger, and the two samples are from the same field. Based on this, the embodiment of the present invention can obtain a cross-domain similarity graph W through formula (1) cro , and the source domain similarity graph W can be obtained through formula (2) ss and the target domain similarity graph W tt .

[0079] S3, according to the constructed similarity graph, transferring the real labels annotated in the source domain to the target domain through a label propagation algorithm based on nuclear norm maximization to obtain pseudo labels of the target domain;

[0080] The embodiment of the present invention utilizes the constructed similarity graph to transfer the real labels annotated in the source domain to the target domain through a label propagation algorithm based on nuclear norm maximization, and obtains the pseudo labels of the target domain, and converts the cross-domain similarity graph W into a pseudo-label. cro , target domain similarity graph W tt And the source domain true label Y s Input into formula (3), and obtain the pseudo label of the target domain based on the following formula (3):

[0081]

[0082]

[0083] In formula (3), and They are the cross-domain similarity graph and the target domain similarity graph respectively; and Represent the predicted labels of the i-th and j-th samples respectively; and The predicted label matrices representing the source domain and the target domain respectively; Y s The true label matrix representing the source domain; ||·|| * Characterizes the nuclear norm of the matrix; 'st' is followed by non-negative constraints, normalization constraints, and constraints that keep the source domain labels consistent with the true labels; δ and σ are the hyperparameters of the corresponding items.

[0084] In formula (3), the first term propagates the labels of the source domain to the target domain through the cross-domain similarity graph; the second term enables the predicted labels of the target domain to maintain the clustering structure of the target domain itself through the target domain similarity graph; the third term solves the class imbalance problem in the state detection process by improving the diversity of the prediction results in the target domain; and finally realizes the transfer of the annotated real labels in the source domain to the target domain to obtain the pseudo labels of the target domain.

[0085] S4, determining the cross entropy loss of the source domain according to the real label of the source domain and the source domain label predicted by the backbone network;

[0086] According to an embodiment of the present invention, determining the cross entropy loss of the source domain based on the real label of the source domain and the source domain label predicted by the backbone network includes: determining a first real probability and a first predicted probability that the i-th real label in the source domain is classified into the c-th category based on the real label of the source domain and the source domain label predicted by the backbone network, respectively, where i and c are positive integers; and determining the cross entropy loss of the source domain based on the first real probability and the first predicted probability.

[0087] Specifically, the prediction results of the network classification layer on the source domain and the true label Y in the source domain s Input into the following formula (4), the embodiment of the present invention calculates the cross entropy loss of the source domain through the following loss function:

[0088]

[0089] In formula (4), is the cross entropy loss of the source domain; represents the first true probability that the i-th sample in the source domain is classified into the c-th category, and Represents the first predicted probability that the i-th sample in the source domain is classified into the c-th category.

[0090] S5. Determine the cross entropy loss of the target domain according to the pseudo label of the target domain and the target domain label predicted by the backbone network;

[0091] In an embodiment of the present invention, determining the cross entropy loss of the target field according to the pseudo-label of the target field and the target field label predicted by the backbone network includes: determining the second true probability and the second predicted probability of the i-th true label of the target field being classified into the c-th category according to the pseudo-label of the target field and the target field label predicted by the backbone network, respectively, where i and c are positive integers; and determining the cross entropy loss of the target field according to the second true probability and the second predicted probability.

[0092] Specifically, the prediction results of the network classification layer on the target field and pseudo labels for the target domain Input into the following formula (5), the embodiment of the present invention calculates the cross entropy loss of the target domain through the following loss function:

[0093]

[0094] In formula (5), is the cross entropy loss for the target domain; represents the second true probability of classifying the i-th sample in the target domain into the c-th class; and It represents the second predicted probability that the network classification layer classifies the i-th sample in the target field into the c-th category.

[0095] S6, using the annotated real labels in the source domain and the pseudo labels in the target domain, combined with the first image feature and the second image feature, updating the similarity graph, and calculating the graph embedding loss;

[0096] The embodiment of the present invention updates the similarity graph specifically by the following formula:

[0097]

[0098]

[0099] By transforming the image feature x (i.e., the initialized image feature) and the source domain true label Y s , and the target domain pseudo-label Input into the above optimal graph self-learning mathematical model to obtain the updated similarity graph. Compared with formula (1) and formula (2), labels are introduced in formula (6) and formula (7), so that the two types of similarity graphs have local discriminant structures.

[0100] The updated similarity graph is input into the following formula. The graph embedding loss calculated in the embodiment of the present invention is obtained by the following formula:

[0101]

[0102]

[0103]

[0104] In the formula, is the updated source domain similarity graph; is the target domain similarity graph; is a cross-domain similarity graph; m is the number of state image data in the source domain dataset; n is the number of state image data in the target domain dataset; Embedding loss function for source domain graph; Embedding loss function for target domain graph; Embedding loss function for cross-domain graphs; and are the image features of the i-th sample in the source domain and the j-th sample in the source domain respectively; and are the image features of the i-th sample in the target domain and the j-th sample in the target domain respectively; z i and z j It means that after considering the source domain and the target domain as a whole, the image features of the i-th sample and the j-th sample without distinguishing the domain labels.

[0105] The embodiment of the present invention is for the above and Features and labels are used for optimization respectively. By minimizing the above three graph embedding loss functions, samples belonging to the same category and close to each other in the source domain are clustered together, samples belonging to the same category and close to each other in the target domain are clustered together, and samples belonging to the same category and close to each other between the source domain and the target domain are clustered together. Finally, the features learned by the backbone network have good local discriminant structure and domain invariance, which helps to propagate labels more accurately.

[0106] Specifically, formula (8) forces the source domain sample point pairs with larger similarity weights in feature space x to be closer in feature space z; formula (9) forces the target domain sample point pairs with larger similarity weights in feature space x to be closer in feature space z; formula (10) forces the sample point pairs with larger similarity weights in feature space x (from the source domain and the target domain, respectively) to be closer in feature space z. Through the above graph embedding process, the model can be forced to extract local discriminative and domain-invariant image representations, thereby improving the reliability of the pseudo-labels obtained in step S3.

[0107] S7. Obtain a total loss function according to the cross entropy loss of the source domain, the cross entropy loss of the target domain, and the graph embedding loss, and perform backpropagation on the backbone network according to the total loss function to obtain the state monitoring image classification model.

[0108] The total loss function is calculated by the following formula:

[0109]

[0110] Where α, β, and λ are the hyperparameters of the corresponding items.

[0111] After obtaining the condition monitoring image classification model, the embodiment of the present invention updates the similarity graph based on the extracted first image features and second image features at every complete training cycle epoch, and uses the updated similarity graph to update the pseudo-label of the target domain.

[0112] Table 1 Experimental results on VisDA-2017

[0113]

[0114] Refer to Table 1. The Methods column in Table 1 shows the comparison results of the Ours method of the present invention and other mainstream methods; the Venue column shows the names of international journals and conferences where the above-mentioned comparison methods were published, as well as the corresponding years. The columns corresponding to airplane, bicycle, bus, car, horse, knife, motorcycle, person, plant, skateboard, train, and truck in the table are the accuracies of different methods in the category; the Avg. column in the table is the average accuracies of different methods in all categories. It can be determined from Table 1 that on the mainstream dataset VisDA-2017, the present invention achieved the best results in terms of the average accuracy of all categories.

[0115] Figure 3(a)-3(c) This is an ablation experiment of an embodiment of the present invention on the mainstream dataset VisDA-2017: "gtlabel" represents the true label, and the height of the corresponding black dotted column depicts the number of true samples in the corresponding category; "pseudo-label" represents the pseudo-label or model prediction label, and the height of the corresponding colored column depicts the number of predicted samples in the corresponding category; the red line corresponding to "acc per class (%)" depicts the accuracy of the model prediction on the corresponding category (the number corresponding to the inflection point of the line marks the specific accuracy value); the horizontal axis corresponds to each category, see Table 1; the vertical axis value is the number of samples divided by 103, which depicts the number of samples. From Figure 3(a)-3(c) It can be observed that the VisDA-2017 dataset exhibits a serious class imbalance distribution.

[0116] Figure 3(a) shows the prediction results of the standard label propagation algorithm LP (the average accuracy of all classes is 73.1%). A large number of samples are predicted to be in the large class (exceeding the number of real samples), and a small number of samples are predicted to be in the small class (lower than the number of real samples). Since optimal graph learning is robust to distribution deviation, the addition of training set samples will alleviate the class imbalance problem to a certain extent. Figure 3(b) shows the prediction results of the label propagation algorithm OGL for LP with the introduction of optimal graph learning (the average accuracy of all classes is 75.9%). The number of samples classified into small classes begins to increase. Figure 3(c) shows the prediction results of the label propagation algorithm OGL and NNM for LP with the introduction of optimal graph learning and nuclear norm maximization (the average accuracy of all classes is 83.4%). The introduction of nuclear norm maximization constraint will further increase the number of small class samples. This ablation experiment verifies the effectiveness of the label propagation algorithm based on optimal graph learning and nuclear norm maximization.

[0117] Compared with the prior art, the training method of the condition monitoring image classification model described in the embodiment of the present invention has the following advantages:

[0118] 1. Based on the optimal graph self-learning strategy, two different types of similarity graphs are learned. One is to build a cross-domain graph between two different domains to prevent the distribution bias from causing the similarity between two samples from different domains to be too small. Two domain-specific similarity graphs are built in the source domain and the target domain respectively to fully explore the structural information of each domain, solving the problem that the label propagation algorithm is sensitive to distribution bias.

[0119] 2. In terms of regularization constraints, the nuclear norm maximization constraint is introduced into the label propagation algorithm to increase the number of samples predicted into small classes and the classification accuracy of small class samples, improve the diversity of prediction results, and effectively solve the problem of data category imbalance;

[0120] 3. By using cross entropy loss to supervise the network classification layer, the label propagation algorithm is introduced into a deep convolutional neural network, and the excellent image classification ability of the label propagation algorithm is transferred to the deep network model;

[0121] 4. Introduce graph embedding technology and apply the above two types of similarity graphs to the deep image representation learning process, promote local discrimination and domain invariance of representation, further improve the performance of the model, and improve the image classification performance of the deep network model.

[0122] Based on the above embodiment, as the above Figure 1 The present invention provides an embodiment of a training device for a state monitoring image classification model. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices, see Figure 4 As shown, the training device of the state monitoring image classification model includes:

[0123] The data acquisition module 100 is used to construct a data set in the source field that has been annotated with real labels and a data set in the target field that has not been annotated, which are required for the condition monitoring image classification model, based on the collected state image data of the industrial equipment;

[0124] A feature processing module 200, used for pre-training a backbone network of the condition monitoring image classification model, and extracting first image features of the source domain and second image features of the target domain through the backbone network, and constructing a similarity graph based on the extracted first image features and second image features;

[0125] A similarity graph construction module 300 is used to transfer the annotated real labels in the source domain to the target domain through a label propagation algorithm based on nuclear norm maximization according to the constructed similarity graph, so as to obtain pseudo labels of the target domain;

[0126] A source domain loss calculation module 400 is used to determine the cross entropy loss of the source domain according to the real label of the source domain and the source domain label predicted by the backbone network;

[0127] A target domain loss calculation module 500 is used to determine the cross entropy loss of the target domain according to the pseudo label of the target domain and the target domain label predicted by the backbone network;

[0128] A graph embedding loss calculation module 600 is used to update the similarity graph by using the real labels annotated in the source domain and the pseudo labels in the target domain, combined with the first image feature and the second image feature, and calculate the graph embedding loss;

[0129] The model parameter adjustment module 700 is used to obtain a total loss function according to the cross entropy loss of the source domain, the cross entropy loss of the target domain and the graph embedding loss, and to back-propagate the backbone network according to the total loss function to obtain the state monitoring image classification model.

[0130] The training device for the status monitoring image classification model described in the embodiment of the present invention can execute the training method for the status monitoring image classification model provided in the above embodiment. The training device for the status monitoring image classification model has the corresponding functional steps and beneficial effects of the training method for the status monitoring image classification model described in the above embodiment. Please refer to the embodiment of the training method for the status monitoring image classification model mentioned above for details. The embodiment of the present invention will not be repeated here.

[0131] The embodiment of the present invention also provides an electronic device, which may include a processor and a memory, wherein the processor and the memory may be connected via a bus or otherwise. The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components and other chips, or a combination of the above-mentioned various types of chips. The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs, non-transient computer executable programs and modules, such as the program instructions / modules corresponding to the training method of the state monitoring image classification model in the embodiment of the present invention. The processor executes various functional applications and data processing of the processor by running the non-transient software programs, instructions and modules stored in the memory, that is, the training method of the state monitoring image classification model in the above method embodiment is realized.

[0132] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. The one or more modules are stored in the memory, and when executed by the processor, the training method of the state monitoring image classification model in the above-mentioned method embodiment is executed. The specific details of the above-mentioned electronic device can be understood by corresponding to the corresponding descriptions and effects in the above-mentioned method embodiment, and will not be repeated here. Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and the program, when executed, may include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk drive (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above types of memory.

[0133] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0134] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the intention that the claimed invention requires more features than those expressly recited in each claim. Rather, as reflected in the claims, inventive aspects lie in less than all of the features of the individual embodiments previously disclosed. Therefore, the claims that follow the detailed description are hereby expressly incorporated into the detailed description, with each claim itself serving as a separate embodiment of the present invention.

[0135] It should be noted that the above-mentioned embodiments illustrate rather than limit the invention and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims.

Claims

1. A training method for a condition monitoring image classification model, wherein the condition monitoring image classification model is used for classifying condition monitoring images of industrial equipment, characterized in that: The training method of the condition monitoring image classification model includes: Based on the collected state image data of industrial equipment, we construct the source domain dataset with real labels and the target domain dataset that are not labeled, which are required for the state monitoring image classification model. Pre-training to obtain a backbone network of the condition monitoring image classification model, and respectively extracting first image features of the source domain and second image features of the target domain through the backbone network, and constructing a similarity graph based on the extracted first image features and second image features; According to the constructed similarity graph, the real labels annotated in the source domain are transferred to the target domain through a label propagation algorithm based on nuclear norm maximization to obtain pseudo labels of the target domain; Determine the cross entropy loss of the source domain according to the real label of the source domain and the label of the source domain predicted by the backbone network; Determine the cross entropy loss of the target domain according to the pseudo label of the target domain and the target domain label predicted by the backbone network; Using the annotated real labels in the source domain and the pseudo labels in the target domain, combined with the first image feature and the second image feature, the similarity graph is updated, and the graph embedding loss is calculated; A total loss function is obtained according to the cross entropy loss of the source domain, the cross entropy loss of the target domain and the graph embedding loss, and the backbone network is back-propagated according to the total loss function to obtain the state monitoring image classification model.

2. The training method of the condition monitoring image classification model according to claim 1 is characterized in that: The similarity graph includes a cross-domain similarity graph, a source domain similarity graph, and a target domain similarity graph. The constructing of the similarity graph based on the extracted first image features and second image features includes: Based on the extracted first image features and second image features, construct a cross-domain similarity graph from the source domain to the target domain; constructing a source domain similarity graph based on any two of the first image features extracted in the source domain; A target domain similarity graph is constructed based on any two of the second image features extracted in the target domain.

3. The training method of the condition monitoring image classification model according to claim 2 is characterized in that: The similarity graph constructed by using the label propagation algorithm based on nuclear norm maximization is used to transfer the real labels marked in the source domain to the target domain to obtain the pseudo labels of the target domain. The pseudo labels of the target domain are obtained based on the following formula: In the formula, and They are the cross-domain similarity graph and the target domain similarity graph respectively; and Represent the predicted labels of the i-th and j-th samples respectively; and The predicted label matrices representing the source domain and the target domain respectively; Y s The true label matrix representing the source domain; δ and σ are the hyperparameters of the corresponding items.

4. The training method of the condition monitoring image classification model according to claim 1, characterized in that: Determining the cross entropy loss of the source domain according to the real label of the source domain and the source domain label predicted by the backbone network includes: According to the real label of the source domain and the source domain label predicted by the backbone network, respectively determine the first real probability and the first predicted probability that the i-th real label of the source domain is classified into the c-th category, where i and c are positive integers; The cross entropy loss of the source domain is determined according to the first true probability and the first predicted probability.

5. The training method of the condition monitoring image classification model according to claim 1, characterized in that: Determining the cross entropy loss of the target domain according to the pseudo label of the target domain and the target domain label predicted by the backbone network includes: According to the pseudo label of the target domain and the target domain label predicted by the backbone network, respectively determine the second true probability and the second predicted probability that the i-th true label of the target domain is classified into the c-th category, where i and c are positive integers; The cross entropy loss of the target domain is determined according to the second true probability and the second predicted probability.

6. The training method of the condition monitoring image classification model according to claim 3 is characterized in that: The method calculates the graph embedding loss through the following formula: In the formula, is the updated source domain similarity graph; is the target domain similarity graph; is a cross-domain similarity graph; m is the number of state image data in the source domain dataset; n is the number of state image data in the target domain dataset; Embedding loss function for source domain graph; Embedding loss function for target domain graph; Embedding loss function for cross-domain graphs; and are the image features of the i-th sample in the source domain and the j-th sample in the source domain respectively; and are the image features of the i-th sample in the target domain and the j-th sample in the target domain respectively; z i and z j It means that after considering the source domain and the target domain as a whole, the image features of the i-th sample and the j-th sample without distinguishing the domain labels.

7. The training method of the condition monitoring image classification model according to claim 1, characterized in that: The method further includes: updating the similarity graph based on the extracted first image features and second image features at intervals of a complete training cycle, and updating the pseudo-label of the target domain using the updated similarity graph.

8. A training device for a condition monitoring image classification model, applied to the method according to any one of claims 1 to 7, characterized in that: The training device of the condition monitoring image classification model comprises: A data acquisition module is used to construct a data set in the source field that has been annotated with real labels and a data set in the target field that has not been annotated, which are required for the condition monitoring image classification model, based on the collected state image data of the industrial equipment; A feature processing module, used for pre-training a backbone network of the condition monitoring image classification model, extracting first image features of the source domain and second image features of the target domain through the backbone network, and constructing a similarity graph based on the extracted first image features and second image features; A similarity graph construction module is used to transfer the annotated real labels in the source domain to the target domain through a label propagation algorithm based on nuclear norm maximization according to the constructed similarity graph, so as to obtain pseudo labels of the target domain; A source domain loss calculation module, used to determine the cross entropy loss of the source domain according to the real label of the source domain and the source domain label predicted by the backbone network; A target domain loss calculation module, used to determine the cross entropy loss of the target domain according to the pseudo label of the target domain and the target domain label predicted by the backbone network; A graph embedding loss calculation module, used to update the similarity graph by using the annotated real labels in the source domain and the pseudo labels in the target domain, combined with the first image feature and the second image feature, and calculate the graph embedding loss; The model parameter adjustment module is used to obtain a total loss function according to the cross entropy loss of the source domain, the cross entropy loss of the target domain and the graph embedding loss, and to back-propagate the backbone network according to the total loss function to obtain the state monitoring image classification model.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, which are loaded and executed by a processor to implement the operations performed by the method according to any one of claims 1 to 7.

10. An electronic device comprising a processor and a memory, characterized in that: The memory stores computer program instructions that can be executed by the processor, and when the processor executes the computer program instructions, the processor implements instructions of any one of the methods described in claims 1-7.

Images