Small sample rotating equipment damage identification method and system based on multi-modal large model

Abstract

The invention discloses a small sample rotating equipment damage identification method and system based on a multi-modal large model, and the method comprises the steps: extracting a damage image feature vector and a damage description text feature vector, and quantitatively representing the distance between a damage image feature and a text feature through a similarity function; constructing a small sample rotating equipment damage knowledge extraction module by combining an image encoder and damage hot independent encoding, and extracting small sample image-label knowledge containing damage categories; and a fusion identification module of large model image-semantic priori knowledge and small sample knowledge in the damage feature buffer is constructed, and high-precision identification of the damage type of the rotating equipment under the small sample condition is realized. According to the method, the problem of domain deviation between the rotating equipment damage image and a large pre-training natural image data set is effectively solved, and the identification precision of the rotating equipment damage image under the small sample is improved by introducing small sample rotating equipment damage knowledge and utilizing priori knowledge of a large model at the same time.

Description

Background Art

[0002] Rotating equipment is the core component of the transmission system of mechanical equipment, and its working performance directly determines the service performance and reliability of the whole machine. The tribological failure behavior of rotating equipment during operation may limit the service performance of key components and even the whole machine, which has become a bottleneck restricting the service reliability of mechanical equipment. Rotating equipment damage morphology is an intuitive presentation of rotating equipment failure. Accurately identifying the various types of damage distributed on the surface of rotating equipment is the key to exploring the causes of rotating equipment failures and formulating rotating equipment operation and maintenance measures.

[0003] The existing recognition models include the combination of surface morphology features and machine learning models, the recognition model based on deep learning CNN network, and the efficient fine-tuning recognition model based on image-semantic large model. The recognition model combining surface morphology features with machine learning models realizes the automatic recognition of the type of surface damage of rotating equipment by extracting the morphology characterization parameters of the detected surface damage area and using the linear or nonlinear distinction ability of the machine learning model. However, this method is highly dependent on the fine feature selection of damage and expert experience knowledge. How to select the best parameters from a large number of morphology characterization parameters for damage with randomness and diversity of morphology is the main problem faced in practical applications. In order to avoid complex and tedious feature engineering, scholars introduced the recognition model based on deep learning CNN network. This model can effectively overcome the limitations of the weak feature representation power extracted by shallow neural networks, but due to the scarcity of rotating equipment fault data, the trained rotating equipment damage recognition model has the problems of low recognition accuracy and poor generalization ability in practical applications. Aiming at the classification and recognition problem of small sample images, an efficient fine-tuning model based on the image-semantic large model is introduced. This model uses the small sample data of the target domain to fine-tune the image-semantic model by freezing all or part of the parameters of the original image-semantic model and adding a small number of training parameters on the basis of the image-semantic multimodal large model. It can learn transferable visual features from paired natural language supervision using contrastive learning, and then show strong small sample image classification capabilities without retraining, while reducing computing resource requirements and high training costs. However, the uniqueness of the morphology of rotating equipment surface damage and the abstractness of semantic description lead to a large domain deviation between the image-text information space of rotating equipment damage and the text-information space of natural images; this situation limits the application accuracy of the fine-tuned large model in the identification of rotating equipment surface damage.

[0004] In summary, the combination of surface morphological features and machine learning models can realize the identification of typical damage types on the surface of rotating equipment, but it faces the problems of cumbersome feature parameter design and selection and low intelligence. The recognition model based on deep learning CNN network effectively avoids the cumbersome and time-consuming feature parameter engineering, but in practical applications, it still needs to solve the problem of low generalization ability under small sample conditions. Although efficient fine-tuning based on image-semantic large model can provide a new solution for the identification of rotating equipment damage under small sample conditions, there is a large domain deviation between the image-text feature space of rotating equipment damage and the feature space of the large model training set, which limits the generalization ability of the large model application. Therefore, how to effectively utilize the excellent image and text feature extraction capabilities of the large model to improve the recognition accuracy of rotating equipment damage images under small samples still needs further research. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a small sample rotating equipment damage identification method and system based on a multimodal large model in response to the deficiencies in the above-mentioned prior art, so as to solve the technical problem of domain deviation between rotating equipment damage images and large pre-trained natural image data sets, and realize accurate identification of rotating equipment damage types.

[0006] The present invention adopts the following technical solutions:

[0007] A small sample rotating equipment damage identification method based on a multi-modal large model comprises the following steps:

[0008] S1. Extract the feature vector of the damaged image and the feature vector of the damaged description text based on the pre-trained image-semantic large model, and quantify the distance between the damaged image features and the text features using a similarity function to achieve the prior knowledge representation of the damage type and obtain the large model image-semantic prior knowledge;

[0009] S2, combining the image encoder with the damage hot-single encoding to extract the small sample image-semantic knowledge containing the damage category; combining the feature vector matrix converted by the image encoder with the label matrix obtained by the hot-single encoding to jointly construct a feature cache library containing the small sample image-semantic knowledge of the damage category;

[0010] S3. The large model image-semantic prior knowledge obtained in step S1 and the feature cache library of the small sample image-semantic knowledge containing the damage category obtained in step S2 are merged to achieve high-precision identification of the damage type of the rotating equipment under small sample conditions.

[0011] Preferably, the large model image-semantic prior knowledge is obtained as follows:

[0012] S101, setting the visual encoder and text encoder to the pre-trained ViT-B / 16 architecture and the pre-trained text Transformer architecture respectively; using a large amount of paired data of natural images and texts to train the image-semantic large model;

[0013] S102, using the pre-trained image-semantic large model to extract the damaged image feature vector and the damaged text feature vector respectively; constructing a text database of the damage type to be identified, using a text encoder to convert the damaged text into a C-dimensional text feature vector library, using an image encoder to convert the damaged image to be tested into a C-dimensional image feature vector, and realizing the extraction of damaged image and text features;

[0014] S103, using similarity to quantify the distance between the damaged image features and the text features; combining the image and text features in pairs, calculating the similarity of the combined pairs, and obtaining an image-text similarity matrix, where the elements on the diagonal of the matrix belong to positive samples, representing correctly paired images and texts, and the remaining elements are negative samples; maximizing the similarity of the positive samples and minimizing the similarity of the negative samples;

[0015] S104: Calculate the similarity between the feature vector and the text feature, and generate a picture-text feature matching result vector As a large model image-semantic prior knowledge, Represented as the feature matrix of all damage description texts extracted by the text encoder.

[0016] Preferably, extracting the semantic knowledge of a small sample image containing a damage category is specifically as follows:

[0017] S201, construct a training set of small sample images with N categories and labels;

[0018] S202, construct a small sample damage knowledge extraction module including a two-layer perceptron, and use the pre-trained image encoder to convert the small sample image training set into a feature vector matrix As the initial weight of the two-layer linear network structure f(·);

[0019] S203, perform one-hot encoding on the small sample image training set to obtain a label matrix The small sample damaged image data is used as the input of the two-layer linear network structure f(·), and the cross entropy is used as the loss function. The pre-trained knowledge extracted by the image encoder and text encoder in step S1 is combined. Jointly train a two-layer linear network structure f(·);

[0020] S204: weight W of the double-layer linear network structure f(·) trained in step S203 f(·) After L2 regularization, we get As the final feature knowledge of the small sample damaged image, the joint label matrix L train Build a feature cache library containing small sample images-semantic knowledge of damage categories.

[0021] Preferably, in step S201, in the small sample image training set, there are K images of each type of damage images, constituting the small sample image training set.

[0022] Preferably, in step S202, the input layer parameter of the two-layer linear network structure f(·) is C, and the output layer parameter is NK, where C represents the number of categories, and NK represents the number of damaged images of each category multiplied by the total number of categories. The eigenvector matrix is ​​as follows:

[0023]

[0024] Where V(·) is the pre-trained image encoder; It is a vector concatenation operation.

[0025] Preferably, in step S203, the label matrix is:

[0026]

[0027] Among them, O(·) is the one-hot encoding operation.

[0028] Preferably, the large model image-semantic prior knowledge obtained in step S1 and the small sample image-semantic knowledge containing damage categories extracted in step S2 are fused to achieve high-precision identification of the damage type of the rotating equipment under small sample conditions. Specifically,

[0029] S301, pass the damaged image to be tested through the image encoder to obtain the image prior knowledge feature f after L2 norm normalization test ; The image prior knowledge feature f test As the query vector, match the small sample damage feature matrix F in the cache train , calculate the membership

[0030] S302: The membership degree A obtained in step S301 is compared with the label matrix L in the feature buffer. train Perform linear combination to obtain the damage type identification result AL train ;

[0031] S303, weighted combination of the small sample feature buffer obtained in step S2 and the damage type identification result of the large model prior knowledge obtained in step S1, to obtain the final rotating equipment damage image type to be identified.

[0032] Preferably, the degree of membership A is:

[0033]

[0034] Among them, β is the adjustment hyperparameter; is equivalent to the large model prior knowledge f test With the small sample feature matrix F train The similarity between .

[0035] Preferably, the type of the rotating equipment damage image to be finally identified is

[0036]

[0037] Among them, α is the weight coefficient, and logits is the membership probability of each damage type.

[0038] In a second aspect, an embodiment of the present invention provides a small sample rotating equipment damage identification system based on a multi-modal large model, comprising:

[0039] The quantization module extracts the feature vector of the damage image and the feature vector of the damage description text based on the pre-trained image-semantic large model, and quantifies the distance between the damage image features and the text features using a similarity function to achieve the prior knowledge representation of the damage type and obtain the large model image-semantic prior knowledge;

[0040] An extraction module combines an image encoder with a damage hot-single encoding to extract small sample image-semantic knowledge containing damage categories, and combines a feature vector matrix converted by the image encoder with a label matrix obtained by hot-single encoding to construct a feature cache library containing small sample image-semantic knowledge containing damage categories;

[0041] The identification module fuses the large model image-semantic prior knowledge and the feature cache library of small sample image-semantic knowledge containing damage categories to achieve high-precision identification of rotating equipment damage types under small sample conditions.

[0042] In a third aspect, a computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned small sample rotating equipment damage identification method based on a multi-modal large model when executing the computer program.

[0043] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned small sample rotating equipment damage identification method based on a multi-modal large model.

[0044] In a fifth aspect, a chip comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method for damage identification of small sample rotating equipment based on a multi-modal large model when executing the computer program.

[0045] In a sixth aspect, an embodiment of the present invention provides an electronic device, including a computer program, which, when executed by the electronic device, implements the steps of the above-mentioned small sample rotating equipment damage identification method based on a multi-modal large model.

[0046] Compared with the prior art, the present invention has at least the following beneficial effects:

[0047] A small sample rotating equipment damage identification method based on a multimodal large model is proposed. By combining a pre-trained image-semantic large model and a constructed small sample rotating equipment damage knowledge extraction module, the recognition accuracy of rotating equipment damage images under small sample conditions is effectively improved, solving the problems of poor generalization ability and low recognition accuracy of traditional deep learning models on small sample data.

[0048] Furthermore, by using a pre-trained image-semantic large model, feature vectors can be extracted from both image and text modalities, and the combination of the two can provide richer data representation. Providing rich feature representation for subsequent small sample learning helps to achieve high-precision damage identification under small sample conditions.

[0049] Furthermore, by introducing small sample rotating equipment damage knowledge, the domain deviation between the large model pre-training dataset and the rotating equipment dataset can be effectively compensated. Even when the domain distribution of the rotating equipment damage image and the large model training set is quite different, a high recognition accuracy can be maintained.

[0050] Furthermore, the introduction of a two-layer structure helps optimize the feature space, so that different categories of damage images have better separation in the feature space. By adjusting the network weights, the intra-class differences can be reduced while the inter-class differences can be expanded, thereby improving the classification performance and generalization ability of the model, and effectively solving the overfitting problem that is prone to occur in small sample learning.

[0051] Furthermore, by integrating the prior knowledge of the large model and the small sample data, the generalization ability of the large model and the specificity of the small sample data are effectively utilized to improve the accuracy of damage identification of rotating equipment.

[0052] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0053] In summary, the present invention provides a small sample rotation equipment damage identification method based on a multimodal large model. The method effectively improves the identification accuracy of the rotation equipment damage image under the condition of small samples by combining the pre-trained image-semantic large model and the constructed small sample rotation equipment damage knowledge extraction module, and solves the problem of poor generalization ability and low identification accuracy of the traditional deep learning model on small sample data; the present invention uses the pre-trained image-semantic large model to extract feature vectors from the two modalities of image and text, provides rich data representation, and introduces the small sample rotation equipment damage knowledge to make up for the domain deviation between the large model pre-trained data set and the rotation equipment data set, and maintains a high recognition accuracy rate; in addition, the present invention optimizes the feature space through a double-layer structure, reduces the intra-class difference and expands the inter-class difference, improves the classification performance and generalization ability of the model, and effectively solves the overfitting problem that is easy to occur in small sample learning. By integrating the prior knowledge of the large model and the small sample data, the present invention not only improves the accuracy of rotation equipment damage identification, but also effectively solves the domain deviation problem between the rotation equipment damage image and the large pre-trained natural image data set, and provides an innovative solution for high-precision identification of rotation equipment damage types under small sample conditions.

[0054] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0056] Figure 1 It is the overall flow chart of the present invention;

[0057] Figure 2 is a framework diagram of image-semantic model pre-training and feature extraction, where (a) is a comparative pre-training based on a natural image-text pair dataset, and (b) is image-text feature extraction and matching;

[0058] Figure 3 Schematic diagram of the training set and test set of typical fault damage of rotating equipment;

[0059] FIG4 is a graph showing the change in loss value and accuracy of the recognition model, wherein (a) is a curve showing the change in loss value during the model training process, and (b) is a curve showing the change in accuracy during the model training process;

[0060] Figure 5 Comparison of the accuracy of the models with and without damage knowledge guidance in the test set;

[0061] Figure 6 A schematic diagram of a computer device provided by an embodiment of the present invention;

[0062] Figure 7 The present invention is a block diagram of an electronic device provided according to an embodiment. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] In the description of the present invention, it should be understood that the terms “include” and “comprises” indicate the presence of described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0065] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0066] It should be further understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects are in an "or" relationship.

[0067] It should be understood that, although the terms first, second, third, etc. may be used to describe preset ranges, etc. in the embodiments of the present invention, these preset ranges should not be limited to these terms. These terms are only used to distinguish preset ranges from each other. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0068] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0069] Various structural schematic diagrams of the embodiments disclosed in the present invention are shown in the accompanying drawings. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0070] The present invention provides a small sample rotating equipment damage identification method based on a multimodal large model. Based on a pre-trained image-semantic large model, a damage image feature vector and a damage description text feature vector are extracted, and the distance between the damage image feature and the text feature is quantified by a similarity function to achieve a priori knowledge representation of the damage type; a small sample rotating equipment damage knowledge extraction module is constructed by combining an image encoder with a damage hot-single encoding to extract small sample image-semantic knowledge containing damage categories, laying a foundation for correcting the domain deviation between a pre-trained data set of an image-semantic model and a damage image data set to be identified; a fusion identification module of large model image-semantic prior knowledge and small sample knowledge in a damage feature buffer is constructed to achieve high-precision identification of rotating equipment damage types under small sample conditions.

[0071] See also Figure 1 The present invention provides a small sample rotating equipment damage identification method based on a multi-modal large model, comprising the following steps:

[0072] S1. Based on the pre-trained image-semantic model, the feature vector of the damage image and the feature vector of the damage description text are extracted, and the distance between the damage image features and the text features is quantified using a similarity function to achieve a priori knowledge representation of the damage type;

[0073] S101, the visual encoder and the text encoder are set to the pre-trained ViT-B / 16 architecture and the pre-trained text Transformer architecture respectively; a large amount of paired data of natural images and texts is used to train the image-semantic large model;

[0074] Please refer to Figure 2, which is a framework for model pre-training and feature extraction. A training batch containing image-text pairs is constructed, and image and text encoders are used to extract image and text features, laying the foundation for subsequent calculation of the similarity between the damaged image features and all damaged text features to achieve preliminary classification and recognition of damaged images.

[0075] S102, in order to learn the correct alignment relationship between the image and the text, the distance between the damaged image features and the text features is quantified by similarity; the image and text features are combined in pairs, and the similarity of the combined pairs is calculated to obtain an image-text similarity matrix; wherein the elements on the diagonal of the matrix are positive samples, representing correctly paired images and texts, and the remaining elements are negative samples;

[0076] Use the text encoder to convert the damaged text into a C-dimensional text feature vector library Use the image encoder to convert the damaged image to be tested into a C-dimensional image feature vector Realize the extraction of damaged image and text features.

[0077] S103, maximizing the similarity of positive samples and minimizing the similarity of negative samples;

[0078]

[0079] S104: To extract the features of the damage image and text, a text database of the damage type to be identified is constructed, and a text encoder is used to convert the damage text into a C-dimensional text feature vector library. Use the image encoder to convert the damaged image to be tested into a C-dimensional image feature vector

[0080] S105: Calculate the similarity between the feature vector and the text feature to generate a picture-text feature matching result vector

[0081] S2, combining image encoder and damage hot-single encoding to construct a small sample rotating equipment damage knowledge extraction module to extract small sample image-semantic knowledge containing damage categories;

[0082] S201. In order to construct a small sample rotating equipment damage knowledge extraction module, it is first necessary to construct a training set of N types of labeled damage images. In this training set, each type of damage image has K images, and these images together constitute the basis of the small sample image training set;

[0083] S202, the small sample damage knowledge extraction module is composed of a two-layer perceptron, and uses a pre-trained image encoder to convert the above small sample image training set into a feature vector matrix

[0084] This process involves encoding the image data through an image encoder to obtain the corresponding feature vectors; these feature vectors are subsequently used as the initial weights of a two-layer linear network structure f(·);

[0085] The input layer parameter of the network structure is C, and the output layer parameter is NK, where C represents the number of categories, and NK represents the number of damaged images of each category multiplied by the total number of categories, as follows:

[0086]

[0087] Where V(·) is the pre-trained image encoder; It is a vector concatenation operation.

[0088] S203, perform one-hot encoding on the damaged image training set to obtain a label matrix The small sample damaged image data is used as the network f(·) input, the cross entropy is used as the loss function, and the image-text feature matching results extracted by the image encoder and the text encoder are combined. Jointly train the network structure f(·);

[0089]

[0090] Among them, O(·) is the one-hot encoding operation.

[0091] S204, weight W of the trained network f(·) f(·) After L2 regularization, we get That is, the final feature knowledge of the small sample damaged image, the joint label matrix L train Build the feature cache library.

[0092] S3. Construct a fusion recognition module of large model image-semantic prior knowledge and small sample knowledge to achieve high-precision identification of rotating equipment damage types under small sample conditions.

[0093] S301, the damaged image to be tested is obtained through an image encoder with L2 norm normalized image prior knowledge features f test These features are used as query vectors to match the small sample damage feature matrix F in the cache train , and calculate the membership for:

[0094]

[0095] Among them, β is the adjustment hyperparameter; is equivalent to the large model prior knowledge f test With the small sample feature matrix F train The exponential function is used to convert the similarity into a non-negative value, and the β parameter adjusts the sharpness of the function.

[0096] S302: Based on the calculated membership degree A, compare it with the label matrix L in the feature buffer. train Perform linear combination to obtain the damage type identification result AL train ;

[0097] This step uses membership as a weight to fuse large models and small sample knowledge to improve the accuracy of recognition.

[0098] S303, the damage type identification result calculated from the large model image-semantic prior knowledge is expressed as in Represented as the feature matrix of all injury description texts extracted by the text encoder;

[0099] S304, obtaining the final rotating equipment damage image type to be identified by weighted combination of damage type identification results from the small sample feature buffer and the large model prior knowledge;

[0100]

[0101] Among them, α is the weight coefficient.

[0102] When the domain distribution of the damaged image data of the rotating equipment to be tested is significantly different from that in the pre-training set, the α weight can be increased to increase the guided correction ratio of small sample knowledge.

[0103] S305. During the training process, the weight parameters of the image-semantic large model module are frozen and the weight parameters of the double-layer perceptron are optimized for training.

[0104] The loss function for training the multimodal large model is the cross entropy loss. The training set and test set of typical fault damage of some rotating equipment are as follows: Figure 3 As shown, random sampling and horizontal flipping operations are used to increase the diversity of the image.

[0105] The change results of the loss value and accuracy of the recognition model under the condition of 50 training times are shown in Figure 4. It can be found that the loss value and accuracy of the constructed multimodal large model can converge quickly, and the highest accuracy of the training set is 94.79% when the iteration number is 19, and the accuracy is 93.75% when the iteration number is 50.

[0106] In the test set, the model identification accuracy is compared with and without using small sample damage knowledge and with and without performing hyperparameter optimization. The specific identification accuracy results are as follows: Figure 5As shown, the results show that the introduction of small sample rotating equipment damage knowledge can effectively make up for the domain deviation between the large model pre-training dataset and the rotating equipment dataset, and the introduction of image-semantic large model prior knowledge can also help improve the accuracy of rotating equipment damage identification under small sample conditions.

[0107] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "platforms".

[0108] In another embodiment of the present invention, a small sample rotating equipment damage identification system based on a multimodal large model is provided. The system can be used to implement the above-mentioned small sample rotating equipment damage identification method based on a multimodal large model. Specifically, the small sample rotating equipment damage identification system based on a multimodal large model includes a quantification module, an extraction module and an identification module.

[0109] Among them, the quantization module extracts the feature vector of the damage image and the feature vector of the damage description text based on the pre-trained image-semantic large model, and quantifies the distance between the damage image features and the text features using a similarity function, realizes the prior knowledge representation of the damage type, and obtains the large model image-semantic prior knowledge;

[0110] An extraction module combines an image encoder with a damage hot-single encoding to extract small sample image-semantic knowledge containing damage categories, and combines a feature vector matrix converted by the image encoder with a label matrix obtained by hot-single encoding to construct a feature cache library containing small sample image-semantic knowledge containing damage categories;

[0111] The identification module fuses the large model image-semantic prior knowledge and the feature cache library of small sample image-semantic knowledge containing damage categories to achieve high-precision identification of rotating equipment damage types under small sample conditions.

[0112] In another embodiment of the present invention, a terminal device is provided, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, graphics processing units (GPU), tensor processing units (TPU), digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of a small sample rotating equipment damage identification method based on a multi-modal large model, including:

[0113] Based on the pre-trained image-semantic large model, the damage image feature vector and the damage description text feature vector are extracted, and the distance between the damage image features and the text features is quantified by the similarity function to realize the prior knowledge representation of the damage type and obtain the large model image-semantic prior knowledge; the image encoder and the damage hot-semantic coding are combined to extract the small sample image-semantic knowledge containing the damage category; the feature vector matrix converted by the image encoder is combined with the label matrix obtained by the hot-semantic coding to jointly construct a feature cache library of small sample image-semantic knowledge containing the damage category; the obtained large model image-semantic prior knowledge and the obtained feature cache library of small sample image-semantic knowledge containing the damage category are fused to achieve high-precision identification of the damage type of rotating equipment under small sample conditions.

[0114] See also Figure 6, the terminal device is a computer device. The computer device 60 of this embodiment includes: a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When the computer program 63 is executed by the processor 61, the small sample rotating equipment damage identification method based on a multi-modal large model in the embodiment is implemented. To avoid repetition, it is not described here one by one. Alternatively, when the computer program 63 is executed by the processor 61, the functions of each model / unit in the small sample rotating equipment damage identification system based on a multi-modal large model in the embodiment are implemented. To avoid repetition, it is not described here one by one.

[0115] The computer device 60 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The computer device 60 may include, but is not limited to, a processor 61 and a memory 62. Those skilled in the art will appreciate that Figure 6 This is only an example of the computer device 60 and does not constitute a limitation of the computer device 60. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer device may also include input and output devices, network access devices, buses, etc.

[0116] The processor 61 may be a central processing unit (CPU), or other general-purpose processors, graphics processing units (GPU), tensor processing units (TPU), digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0117] The memory 62 may be an internal storage unit of the computer device 60, such as a hard disk or memory of the computer device 60. The memory 62 may also be an external storage device of the computer device 60, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the computer device 60.

[0118] Furthermore, the memory 62 may include both an internal storage unit of the computer device 60 and an external storage device. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 may also be used to temporarily store data that has been output or is to be output.

[0119] See also Figure 7 The terminal device 600 is an electronic device, and the electronic device is in the form of a general computing device. The components of the electronic device may include but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0120] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present invention described in the above method section of this specification. For example, the processing unit 610 can perform the following steps: Figure 1 Follow the steps shown in .

[0121] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0122] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0123] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0124] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, etc.

[0125] In another embodiment of the present invention, the present invention also provides a storage medium, specifically a computer-readable storage medium, which is a memory device in a terminal device for storing programs and data. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and the extended storage medium supported by the terminal device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs. It should be noted that the computer-readable storage medium here can be a high-speed RAM memory, or a non-volatile memory (Non-VolatileMemory), such as at least one disk memory.

[0126] The processor may load and execute one or more instructions stored in a computer-readable storage medium to implement the corresponding steps of the small sample rotating equipment damage identification method based on a multi-modal large model in the above embodiment; the processor may load and execute the following steps of one or more instructions in the computer-readable storage medium:

[0127] Based on the pre-trained image-semantic large model, the damage image feature vector and the damage description text feature vector are extracted, and the distance between the damage image features and the text features is quantified by the similarity function to realize the prior knowledge representation of the damage type and obtain the large model image-semantic prior knowledge; the image encoder and the damage hot-semantic coding are combined to extract the small sample image-semantic knowledge containing the damage category; the feature vector matrix converted by the image encoder is combined with the label matrix obtained by the hot-semantic coding to jointly construct a feature cache library of small sample image-semantic knowledge containing the damage category; the obtained large model image-semantic prior knowledge and the obtained feature cache library of small sample image-semantic knowledge containing the damage category are fused to achieve high-precision identification of the damage type of rotating equipment under small sample conditions.

[0128] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention described and shown in the drawings here can usually be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0129] See also Figure 5 , using only the CLIP model without training, the damage identification accuracy can reach 32.29%, which reflects the potential of multimodal large model CLIP in the application of rotating equipment damage identification, but at the same time it is limited by the domain difference between the general data set in the large model and the rotating equipment-specific data set. After introducing the rotating equipment damage knowledge, the damage identification accuracy of the model can reach 93.75%, which is significantly higher than the 84.38% accuracy without integrating the CLIP model prior knowledge. At the same time, the model test accuracy after hyperparameter optimization has achieved the highest value of 94.79%. The results further show that the introduction of small sample rotating equipment damage knowledge can effectively make up for the domain deviation between the large model pre-training data set and the rotating equipment data set, and the introduction of large model CLIP prior knowledge is also helpful to improve the accuracy of rotating equipment damage identification under small sample conditions.

[0130] In summary, the present invention provides a method and system for small sample rotating equipment damage identification based on a multimodal large model. The present invention significantly improves the accuracy of rotating equipment damage identification by innovatively integrating the CLIP multimodal large model that does not require training and the proprietary knowledge of rotating equipment damage. In the initial state, the damage identification accuracy is 32.29% using only the CLIP model, which shows the application potential of CLIP in the field of rotating equipment damage identification, but is limited by the domain difference between the model pre-training data set and the proprietary data set of rotating equipment. By introducing the knowledge of rotating equipment damage, the damage identification accuracy of the model is improved to 93.75%, which is not only significantly higher than the 84.38% accuracy rate without the integration of CLIP prior knowledge, but also through hyperparameter optimization, the model test accuracy can reach up to 94.79%. These results further confirm that the introduction of small sample rotating equipment damage knowledge can effectively make up for the domain deviation between the large model pre-training data set and the rotating equipment data set, and the prior knowledge of the large model CLIP also has a positive effect on improving the accuracy of rotating equipment damage identification under small sample conditions.

[0131] Therefore, the present invention not only improves the accuracy of rotating equipment damage identification, but also enhances the generalization ability of the model under small sample conditions, and provides an efficient and accurate identification method for the field of rotating equipment damage detection.

[0132] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0133] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0134] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0135] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0136] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0137] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0138] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0139] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0140] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0141] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0142] The above contents are only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A small sample rotating equipment damage identification method based on a multi-modal large model, characterized in that: The following steps are involved: S1. Extract the feature vector of the damaged image and the feature vector of the damaged description text based on the pre-trained image-semantic large model, and quantify the distance between the damaged image features and the text features using a similarity function to achieve the prior knowledge representation of the damage type and obtain the large model image-semantic prior knowledge; S2, combining the image encoder with the damage hot-single encoding to extract the small sample image-semantic knowledge containing the damage category; combining the feature vector matrix converted by the image encoder with the label matrix obtained by the hot-single encoding to jointly construct a feature cache library containing the small sample image-semantic knowledge of the damage category; S3. The large model image-semantic prior knowledge obtained in step S1 and the feature cache library of the small sample image-semantic knowledge containing the damage category obtained in step S2 are merged to achieve high-precision identification of the damage type of the rotating equipment under small sample conditions.

2. The small sample rotating equipment damage identification method based on multi-modal large model according to claim 1 is characterized in that: The specific semantic prior knowledge of the large model image is: S101, setting the visual encoder and text encoder to the pre-trained ViT-B / 16 architecture and the pre-trained text Transformer architecture respectively; using a large amount of paired data of natural images and texts to train the image-semantic large model; S102, using the pre-trained image-semantic large model to extract the damaged image feature vector and the damaged text feature vector respectively; constructing a text database of the damage type to be identified, using a text encoder to convert the damaged text into a C-dimensional text feature vector library, using an image encoder to convert the damaged image to be tested into a C-dimensional image feature vector, and realizing the extraction of damaged image and text features; S103, using similarity to quantify the distance between the damaged image features and the text features; combining the image and text features in pairs, calculating the similarity of the combined pairs, and obtaining an image-text similarity matrix, where the elements on the diagonal of the matrix belong to positive samples, representing correctly paired images and texts, and the remaining elements are negative samples; maximizing the similarity of the positive samples and minimizing the similarity of the negative samples; S104: Calculate the similarity between the feature vector and the text feature, and generate a picture-text feature matching result vector As a large model image-semantic prior knowledge, Represented as the feature matrix of all damage description texts extracted by the text encoder.

3. The small sample rotating equipment damage identification method based on multi-modal large model according to claim 1 is characterized in that: Extracting small sample images containing damage categories - semantic knowledge is as follows: S201, construct a training set of small sample images with N categories and labels; S202, construct a small sample damage knowledge extraction module including a two-layer perceptron, and use the pre-trained image encoder to convert the small sample image training set into a feature vector matrix As the initial weight of the two-layer linear network structure f(·); S203, perform one-hot encoding on the small sample image training set to obtain a label matrix The small sample damaged image data is used as the input of the two-layer linear network structure f(·), and the cross entropy is used as the loss function. The pre-trained knowledge extracted by the image encoder and text encoder in step S1 is combined. Jointly train a two-layer linear network structure f(·); S204: weight W of the double-layer linear network structure f(·) trained in step S203 f(·) After L2 regularization, we get As the final feature knowledge of the small sample damaged image, the joint label matrix L train Build a feature cache library containing small sample images-semantic knowledge of damage categories.

4. The small sample rotating equipment damage identification method based on multi-modal large model according to claim 3 is characterized in that: In step S201, in the small sample image training set, there are K images of each type of damage images, constituting the small sample image training set.

5. The small sample rotating equipment damage identification method based on multi-modal large model according to claim 3 is characterized in that: In step S202, the input layer parameter of the two-layer linear network structure f(·) is C, and the output layer parameter is NK, where C represents the number of categories, and NK represents the number of damaged images of each category multiplied by the total number of categories. The eigenvector matrix is ​​as follows: Where V(·) is the pre-trained image encoder; It is a vector concatenation operation.

6. The small sample rotating equipment damage identification method based on multi-modal large model according to claim 3 is characterized in that: In step S203, the label matrix is: Among them, O(·) is the one-hot encoding operation.

7. The small sample rotating equipment damage identification method based on multi-modal large model according to claim 1 is characterized in that: The large model image-semantic prior knowledge obtained in step S1 and the small sample image-semantic knowledge containing damage categories extracted in step S2 are fused to achieve high-precision identification of the damage type of rotating equipment under small sample conditions. Specifically: S301, pass the damaged image to be tested through the image encoder to obtain the image prior knowledge feature f after L2 norm normalization test ; The image prior knowledge feature f test As the query vector, match the small sample damage feature matrix F in the cache train , calculate the membership S302: The membership degree A obtained in step S301 is compared with the label matrix L in the feature buffer. train Perform linear combination to obtain the damage type identification result AL train ; S303, weighted combination of the small sample feature buffer obtained in step S2 and the damage type identification result of the large model prior knowledge obtained in step S1, to obtain the final rotating equipment damage image type to be identified.

8. The small sample rotating equipment damage identification method based on multi-modal large model according to claim 7 is characterized in that: The membership degree A is: Among them, β is the adjustment hyperparameter; is equivalent to the large model prior knowledge f test With the small sample feature matrix F train The similarity between .

9. The small sample rotating equipment damage identification method based on multi-modal large model according to claim 7 is characterized in that: The final type of rotating equipment damage image to be identified is logits=αAL train +f test W C T Among them, α is the weight coefficient, and logits is the membership probability of each damage type.

10. A small sample rotating equipment damage identification system based on a multi-modal large model, characterized in that: include: The quantization module extracts the feature vector of the damage image and the feature vector of the damage description text based on the pre-trained image-semantic large model, and quantifies the distance between the damage image features and the text features using a similarity function to achieve the prior knowledge representation of the damage type and obtain the large model image-semantic prior knowledge; An extraction module combines an image encoder with a damage hot-single encoding to extract small sample image-semantic knowledge containing damage categories, and combines a feature vector matrix converted by the image encoder with a label matrix obtained by hot-single encoding to construct a feature cache library containing small sample image-semantic knowledge containing damage categories; The identification module fuses the large model image-semantic prior knowledge and the feature cache library of small sample image-semantic knowledge containing damage categories to achieve high-precision identification of rotating equipment damage types under small sample conditions.

Images