Small sample rotating equipment damage identification method and system based on multi-modal large model

Abstract

The application discloses a kind of based on multimodal big model small sample rotating equipment damage identification method and system, extract damage image feature vector and damage description text feature vector, and with similarity function quantification representation distance between damage image feature and text feature, joint image encoder and damage hot unique coding constructs small sample rotating equipment damage knowledge extraction module, extract small sample image-label knowledge containing damage class;Fusion identification module of constructing big model image-semantic priori knowledge and small sample knowledge in damage feature buffer, realize the high-precision identification of rotating equipment damage type under small sample condition.The application effectively solves the domain deviation problem between rotating equipment damage image and large pre-training natural image dataset, by introducing small sample rotating equipment damage knowledge, while using the priori knowledge of big model, improve the identification precision of rotating equipment damage image under small sample.

Description

Technical Field

[0001] This invention belongs to the field of machine wear condition monitoring technology, specifically relating to a method and system for identifying damage to small-sample rotating equipment based on a multimodal large model. Background Technology

[0002] Rotating equipment is a core component of the transmission system of mechanical equipment, and its performance directly determines the service performance and reliability of the entire machine. Tribological failure behavior of rotating equipment during operation can limit the service performance of critical components and even the entire machine, becoming a bottleneck restricting the reliability of mechanical equipment. Damage morphology of rotating equipment serves as a direct representation of its failures; accurately identifying the various damage types distributed on the surface of rotating equipment is crucial for investigating the causes of failures and developing maintenance measures.

[0003] Existing identification models include those combining surface morphology features with machine learning models, those based on deep learning CNN networks, and efficient fine-tuning models based on large image-semantic models. The identification model combining surface morphology features with machine learning models extracts morphological representation parameters of detected surface damage regions and utilizes the linear or nonlinear discrimination ability of machine learning models to automatically identify the type of surface damage on rotating equipment. However, this method heavily relies on the selection of fine-grained damage features and expert knowledge. How to select the optimal parameters from numerous morphological representation parameters for damage with randomness and diversity is a major problem in practical applications. To avoid complex and tedious feature engineering, researchers have introduced identification models based on deep learning CNN networks. This model can effectively overcome the limitations of weak feature representation capabilities extracted by shallow neural networks. However, due to the scarcity of rotating equipment fault data, the trained rotating equipment damage identification model suffers from low identification accuracy and poor generalization ability in practical applications. To address the classification and identification problem of images with few samples, an efficient fine-tuning model based on a large image-semantic model is introduced. This model is built upon a large image-semantic multimodal model by freezing all or some parameters of the original image-semantic model and adding a small number of training parameters. It utilizes small sample data from the target domain to fine-tune the image-semantic model, enabling it to learn transferable visual features from paired natural language supervision through contrastive learning. This eliminates the need for retraining and demonstrates strong small sample image classification capabilities, while also reducing computational resource requirements and high training costs. However, the unique morphology of damage on rotating equipment surfaces and the abstract nature of its semantic description lead to a significant domain deviation between the image-text information space of rotating equipment damage and the text-information space of natural images. This limitation restricts the accuracy of the fine-tuned large model in identifying damage on rotating equipment surfaces.

[0004] In summary, combining surface morphology features with machine learning models can identify typical damage types on the surface of rotating equipment, but it faces challenges such as cumbersome feature parameter design and selection, and low intelligence. Identification models based on deep learning CNN networks effectively avoid the tedious and time-consuming feature parameter engineering, but in practical applications, the problem of low generalization ability under small sample conditions still needs to be addressed. Although efficient fine-tuning based on large image-semantic models can provide a new solution for identifying damage on rotating equipment under small sample conditions, there is a significant domain bias between the image-text feature space of rotating equipment damage and the feature space of the large model training set, limiting the generalization ability of the large model application. Therefore, how to effectively utilize the superior image and text feature extraction capabilities of large models to improve the identification accuracy of rotating equipment damage images under small sample conditions still requires further research. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method and system for identifying small-sample rotating equipment damage based on a multimodal large model, which addresses the shortcomings of the prior art and solves the technical problem of domain bias between rotating equipment damage images and large pre-trained natural image datasets, thereby achieving accurate identification of rotating equipment damage types.

[0006] The present invention adopts the following technical solution:

[0007] A method for identifying damage to rotating equipment using small samples based on a multimodal large model includes the following steps:

[0008] S1. Based on the pre-trained image-semantic large model, extract the feature vector of the damaged image and the feature vector of the damaged text description, and use the similarity function to quantify the distance between the damaged image features and the text features, realize the prior knowledge representation of the damage type, and obtain the image-semantic prior knowledge of the large model.

[0009] S2. Joint image encoder and damage hot-unique encoding extract few sample images containing damage categories - semantic knowledge; the feature vector matrix converted by the image encoder is combined with the label matrix obtained by hot-unique encoding to construct a feature cache library containing few sample images containing damage categories - semantic knowledge.

[0010] S3. The feature cache library of large model image-semantic prior knowledge obtained in step S1 and small sample image-semantic knowledge containing damage categories obtained in step S2 is fused to achieve high-precision identification of damage type of rotating equipment under small sample conditions.

[0011] Preferably, the large model image-semantic prior knowledge is obtained as follows:

[0012] S101. Set the visual encoder and text encoder to a pre-trained ViT-B / 16 architecture and a pre-trained text Transformer architecture, respectively; train the image-semantic large model using a large amount of paired natural image and text data.

[0013] S102. Use a pre-trained image-semantic large model to extract feature vectors of damaged images and damaged text respectively; construct a text database of the damage types to be identified, use a text encoder to convert the damaged text into a C-dimensional text feature vector library, and use an image encoder to convert the damaged image to be tested into a C-dimensional image feature vector, thereby realizing the extraction of damage image and text features.

[0014] S103. Use similarity metric to measure the distance between damaged image features and text features; combine image and text features pairwise, calculate the similarity of the combined pairs, and obtain the image-text similarity matrix. The elements on the diagonal of the matrix are positive samples, representing correctly paired images and texts, and the remaining elements are negative samples; maximize the similarity of positive samples and minimize the similarity of negative samples.

[0015] S104. Calculate the similarity between the feature vector and the text features, and generate the image-text feature matching result vector. As prior knowledge of image-semantics in large models, among which... It is represented as the feature matrix of all damage description texts extracted by the text encoder.

[0016] Preferably, extracting semantic knowledge from small sample images containing damage categories specifically involves:

[0017] S201. Construct a training set of small sample images containing N classes with labels;

[0018] S202. Construct a small-sample damage knowledge extraction module containing a two-layer perceptron, and use a pre-trained image encoder to convert the small-sample image training set into a feature vector matrix. As the initial weights of the two-layer linear network structure f(·);

[0019] S203. Perform one-hot encoding on the small sample image training set to obtain the label matrix. Using small sample damaged image data as input to a two-layer linear network structure f(·), and cross-entropy as the loss function, this network combines the pre-trained knowledge extracted from the image encoder and text encoder in step S1. Jointly train a two-layer linear network structure f(·);

[0020] S204. The weights W of the two-layer linear network structure f(·) trained in step S203 are... f(·) After L2 regularization, the obtained As the final feature knowledge of small sample damaged images, the joint label matrix L train Construct a feature cache library containing few-sample images with semantic knowledge of damage categories.

[0021] Preferably, in step S201, the small sample image training set contains K images of each type of damage, forming a small sample image training set.

[0022] Preferably, in step S202, the input layer parameter of the two-layer linear network structure f(·) is C, and the output layer parameter is NK, where C represents the number of categories, and NK represents the number of damaged images of each category multiplied by the total number of categories. The feature vector matrix is ​​as follows:

[0023]

[0024] Where V(·) is the pre-trained image encoder; This is a vector concatenation operation.

[0025] Preferably, in step S203, the tag matrix is:

[0026]

[0027] Where O(·) represents the one-hot encoding operation.

[0028] Preferably, the large model image-semantic prior knowledge obtained in step S1 and the small sample image-semantic knowledge containing damage categories extracted in step S2 are fused to achieve high-precision identification of damage types of rotating equipment under small sample conditions. Specifically:

[0029] S301. Pass the image of the damage to be tested through an image encoder to obtain the L2 norm normalized image prior knowledge features f. test ; to incorporate prior knowledge features f of the image test As the query vector, match the few-sample damage feature matrix F in the cache. train The membership degree is calculated.

[0030] S302. Combine the membership degree A obtained in step S301 with the label matrix L in the feature cache. train Linear combination is performed to obtain the damage type identification result AL. train ;

[0031] S303. The small sample feature buffer obtained in step S2 is combined with the damage type identification result of the large model prior knowledge obtained in step S1 by weighting to obtain the final rotating equipment damage image type to be identified.

[0032] Preferably, the membership degree A is:

[0033]

[0034] Where β is the adjustment hyperparameter; To be equivalent to the prior knowledge f of the large model test With small sample feature matrix F train The similarity between them.

[0035] Preferably, the final image type of the rotating equipment damage to be identified is:

[0036]

[0037] Where α is the weighting coefficient and logits is the membership probability of each damage type.

[0038] Secondly, embodiments of the present invention provide a small-sample rotating device damage identification system based on a multimodal large model, comprising:

[0039] The quantization module extracts feature vectors of damaged images and feature vectors of damaged text based on a pre-trained image-semantic large model, and uses a similarity function to quantify the distance between the damaged image features and the text features, thereby realizing the prior knowledge representation of the damage type and obtaining the image-semantic prior knowledge of the large model.

[0040] The extraction module extracts semantic knowledge of small sample images containing damage categories by combining the image encoder and the damage hot-unique encoding. The feature vector matrix converted by the image encoder is combined with the label matrix obtained by hot-unique encoding to jointly construct a feature cache library containing semantic knowledge of small sample images containing damage categories.

[0041] The identification module integrates a feature cache library of large model image-semantic prior knowledge and small sample image-semantic knowledge containing damage categories to achieve high-precision identification of damage types of rotating equipment under small sample conditions.

[0042] Thirdly, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method for identifying damage to small-sample rotating equipment based on a multimodal large model.

[0043] Fourthly, embodiments of the present invention provide a computer-readable storage medium including a computer program, which, when executed by a processor, implements the steps of the above-described method for identifying damage to small-sample rotating equipment based on a multimodal large model.

[0044] Fifthly, a chip includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method for identifying damage to small-sample rotating devices based on a multimodal large model.

[0045] In a sixth aspect, embodiments of the present invention provide an electronic device including a computer program, wherein when the computer program is executed by the electronic device, it implements the steps of the above-described method for identifying damage to small-sample rotating devices based on a multimodal large model.

[0046] Compared with the prior art, the present invention has at least the following beneficial effects:

[0047] A method for identifying damage to rotating equipment based on a multimodal large model in small samples is proposed. By combining a pre-trained image-semantic large model with a constructed knowledge extraction module for damage to rotating equipment in small samples, the method effectively improves the identification accuracy of damage images of rotating equipment under small sample conditions and solves the problems of poor generalization ability and low identification accuracy of traditional deep learning models on small sample data.

[0048] Furthermore, by using a pre-trained image-semantic large model, feature vectors can be extracted simultaneously from both image and text modalities. Combining these two approaches provides richer data representations, offering abundant feature representations for subsequent few-shot learning and facilitating high-precision damage identification under limited sample conditions.

[0049] Furthermore, by introducing knowledge of damage to rotating equipment from small samples, the domain bias between the large model pre-training dataset and the rotating equipment dataset can be effectively compensated. Even when the domain distribution of the rotating equipment damage images differs significantly from that of the large model training set, a high recognition accuracy can still be maintained.

[0050] Furthermore, introducing a two-layer structure helps optimize the feature space, resulting in better separation between different categories of damage images. By adjusting the network weights, intra-class variance can be reduced while inter-class variance can be increased, thereby improving the model's classification performance and generalization ability, while effectively addressing the overfitting problem that easily occurs in few-shot learning.

[0051] Furthermore, by integrating the prior knowledge of the large model with small sample data, the generalization ability of the large model and the specificity of the small sample data can be effectively utilized to improve the accuracy of damage identification of rotating equipment.

[0052] It is understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0053] In summary, this invention provides a method for identifying damage to rotating equipment in small samples based on a multimodal large model. This method effectively improves the identification accuracy of damaged rotating equipment images under small sample conditions by combining a pre-trained image-semantic large model with a constructed small sample rotating equipment damage knowledge extraction module. It solves the problems of poor generalization ability and low identification accuracy of traditional deep learning models on small sample data. This invention utilizes a pre-trained image-semantic large model to extract feature vectors from both image and text modalities, providing rich data representation. By introducing small sample rotating equipment damage knowledge, it compensates for the domain bias between the large model's pre-training dataset and the rotating equipment dataset, maintaining high identification accuracy. Furthermore, this invention optimizes the feature space through a two-layer structure, reducing intra-class differences while expanding inter-class differences, improving the model's classification performance and generalization ability, and effectively solving the overfitting problem that easily occurs in small sample learning. By fusing the prior knowledge of the large model with small sample data, this invention not only improves the accuracy of rotating equipment damage identification but also effectively solves the domain bias problem between damaged rotating equipment images and large pre-trained natural image datasets, providing an innovative solution for high-precision identification of rotating equipment damage types under small sample conditions.

[0054] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0055] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is the overall flowchart of the present invention;

[0057] Figure 2 shows the framework diagram of image-semantic model pre-training and feature extraction, where (a) is the comparative pre-training based on natural image-text pair datasets, and (b) is the image-text feature extraction and matching.

[0058] Figure 3 A schematic diagram of the training and test sets for typical faults and damages of rotating equipment;

[0059] Figure 4 shows the results of the loss value and accuracy of the identification model. (a) is the curve of the loss value during the model training process, and (b) is the curve of the accuracy during the model training process.

[0060] Figure 5 To compare the accuracy of models with and without knowledge-guided impairment on the test set;

[0061] Figure 6 A schematic diagram of a computer device provided in an embodiment of the present invention;

[0062] Figure 7 This is a block diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0063] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0064] In the description of this invention, it should be understood that the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0065] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0066] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Additionally, the character " / " in this invention generally indicates that the preceding and following objects have an "or" relationship.

[0067] It should be understood that although terms such as first, second, third, etc., may be used in the embodiments of the present invention to describe the preset range, these preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from one another. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0068] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0069] The accompanying drawings illustrate various structural schematic diagrams according to embodiments disclosed in this invention. These drawings are not to scale, and some details have been enlarged for clarity, and some details may have been omitted. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0070] This invention provides a method for identifying damage to rotating equipment in small samples based on a multimodal large model. Based on a pre-trained image-semantic large model, it extracts feature vectors from damaged images and text descriptions of the damage. A similarity function is used to quantify the distance between the damaged image features and the text features, thus representing prior knowledge of the damage type. A small-sample rotating equipment damage knowledge extraction module is constructed by jointly using an image encoder and a damage hot-coded unique encoder. This module extracts small-sample image-semantic knowledge containing damage categories, laying the foundation for correcting the domain bias between the pre-trained dataset of the image-semantic model and the dataset of damaged images to be identified. Finally, a fusion identification module is constructed, integrating the large model's image-semantic prior knowledge with the small-sample knowledge in the damage feature buffer, achieving high-precision identification of rotating equipment damage types under small-sample conditions.

[0071] Please see Figure 1 The present invention provides a method for identifying damage to rotating equipment using small samples based on a multimodal large model, comprising the following steps:

[0072] S1. Based on the pre-trained image-semantic large model, extract the feature vector of the damaged image and the feature vector of the damaged description text, and use the similarity function to quantify the distance between the damaged image features and the text features, so as to realize the prior knowledge representation of the damage type.

[0073] S101, the visual encoder, and the text encoder are set to a pre-trained ViT-B / 16 architecture and a pre-trained text Transformer architecture, respectively; a large image-semantic model is trained using a large amount of paired natural image and text data.

[0074] Please refer to Figure 2, which shows the framework for model pre-training and feature extraction. A training batch containing image-text pairs is constructed, and image and text features are extracted using an image and text encoder. This lays the foundation for subsequent calculation of the similarity between damaged image features and all damaged text features to achieve preliminary classification and recognition of damaged images.

[0075] S102. To learn the correct alignment relationship between images and text, a similarity metric is used to measure the distance between impaired image features and text features. The image and text features are combined pairwise, and the similarity of the combined pairs is calculated to obtain the image-text similarity matrix. The elements on the diagonal of the matrix are positive samples, representing correctly paired images and text, and the remaining elements are negative samples.

[0076] Use a text encoder to convert damaged text into a C-dimensional text feature vector library. The image of the damage to be tested is converted into a C-dimensional image feature vector using an image encoder. To extract features from damaged images and text.

[0077] S103. Maximize the similarity of positive samples and minimize the similarity of negative samples;

[0078]

[0079] S104. To extract features from damaged images and text, a text database of the types of damage to be identified is constructed, and a text encoder is used to convert the damaged text into a C-dimensional text feature vector library. The image of the damage to be tested is converted into a C-dimensional image feature vector using an image encoder.

[0080] S105. Calculate the similarity between the feature vector and the text features to generate the image-text feature matching result vector.

[0081] S2. A small sample rotating device damage knowledge extraction module is constructed by combining the image encoder and the damage thermal independent encoder to extract small sample images containing damage categories and semantic knowledge.

[0082] S201. In order to construct a small sample rotating equipment damage knowledge extraction module, it is first necessary to construct a training set of damage images containing N classes with labels. In this training set, there are K damage images for each class. These images together constitute the basis of the small sample image training set.

[0083] S202, The small sample damage knowledge extraction module consists of a two-layer perceptron, which uses a pre-trained image encoder to convert the aforementioned small sample image training set into a feature vector matrix.

[0084] This process involves encoding image data using an image encoder to obtain corresponding feature vectors; these feature vectors are subsequently used as initial weights for a two-layer linear network structure f(·).

[0085] The input layer parameter of this network structure is C, and the output layer parameter is NK, where C represents the number of categories, and NK represents the number of damaged images of each category multiplied by the total number of categories, as follows:

[0086]

[0087] Where V(·) is the pre-trained image encoder; This is a vector concatenation operation.

[0088] S203. Perform one-hot encoding on the damaged image training set to obtain the label matrix. Using small sample damaged image data as input to network f(·), and cross-entropy as the loss function, the network combines the image-text feature matching results extracted by the image encoder and the text encoder. The network structure f(·) is trained together.

[0089]

[0090] Where O(·) represents the one-hot encoding operation.

[0091] S204, Set the weights W of the trained network f(·) f(·) After L2 regularization, we get This represents the final feature knowledge of small sample damaged images, and the joint label matrix L train Build a feature cache library.

[0092] S3. Construct a fusion identification module for large model image-semantic prior knowledge and small sample knowledge to achieve high-precision identification of damage types of rotating equipment under small sample conditions.

[0093] S301. The image of the damage to be tested is processed by an image encoder to obtain the L2 norm normalized image prior knowledge features f. test These features are used as query vectors to match the few-sample damage feature matrix F in the cache. train And calculate the membership degree. for:

[0094]

[0095] Where β is the adjustment hyperparameter; To be equivalent to the prior knowledge f of the large model test With small sample feature matrix F train The similarity between them. The exponential function is used to convert the similarity into a non-negative value, and the β parameter adjusts the sharpness of the function.

[0096] S302. Based on the calculated membership degree A, compare it with the label matrix L in the feature buffer. train Linear combination is performed to obtain the damage type identification result AL. train ;

[0097] This step utilizes membership degree as a weight to integrate knowledge from large models and small samples, thereby improving the accuracy of identification.

[0098] S303. Represent the damage type identification results calculated from the large model image-semantic prior knowledge. in This is represented as a matrix of all damage description text features extracted by the text encoder;

[0099] S304. By weighted combination of the damage type identification results from the small sample feature buffer and the prior knowledge of the large model, the final type of rotating equipment damage image to be identified is obtained.

[0100]

[0101] Where α is the weighting coefficient.

[0102] When the domain distribution of the damage image data of the rotating equipment to be tested differs significantly from that in the pre-training set, the α weight can be increased to increase the proportion of guidance and correction based on small sample knowledge.

[0103] S305. During training, freeze the weight parameters of the image-semantic large model module and optimize the weight parameters of the two-layer perceptron.

[0104] The loss function for training the multimodal large model is cross-entropy loss. The training and test sets for typical fault damage of rotating equipment are as follows: Figure 3 As shown, random sampling and horizontal flipping operations are used to increase the diversity of images.

[0105] Figure 4 shows the results of the loss and accuracy changes of the identification model under the condition of 50 training iterations. It can be found that the loss and accuracy of the constructed multimodal large model can converge quickly, and the highest accuracy of 94.79% on the training set is achieved when the number of iterations is 19, and the accuracy of 93.75% is achieved when the number of iterations is 50.

[0106] In the test set, the identification accuracy of the model with and without utilizing small-sample damage knowledge and with and without hyperparameter optimization was compared. Specific identification accuracy results are as follows: Figure 5As shown, the results indicate that the introduction of small-sample rotating equipment damage knowledge can effectively compensate for the domain bias between the large model pre-training dataset and the rotating equipment dataset, while the introduction of prior knowledge from the image-semantic large model also helps to improve the accuracy of rotating equipment damage identification under small-sample conditions.

[0107] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "platform."

[0108] In another embodiment of the present invention, a small-sample rotating equipment damage identification system based on a multimodal large model is provided. This system can be used to implement the above-mentioned small-sample rotating equipment damage identification method based on a multimodal large model. Specifically, the small-sample rotating equipment damage identification system based on a multimodal large model includes a quantization module, an extraction module, and an identification module.

[0109] The quantization module extracts feature vectors of damaged images and feature vectors of damaged text based on a pre-trained image-semantic big model, and uses a similarity function to quantify the distance between the damaged image features and the text features, thereby realizing the prior knowledge representation of the damage type and obtaining the image-semantic prior knowledge of the big model.

[0110] The extraction module extracts semantic knowledge of small sample images containing damage categories by combining the image encoder and the damage hot-unique encoding. The feature vector matrix converted by the image encoder is combined with the label matrix obtained by hot-unique encoding to jointly construct a feature cache library containing semantic knowledge of small sample images containing damage categories.

[0111] The identification module integrates a feature cache library of large model image-semantic prior knowledge and small sample image-semantic knowledge containing damage categories to achieve high-precision identification of damage types of rotating equipment under small sample conditions.

[0112] In another embodiment of the present invention, a terminal device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, graphics processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve a corresponding method flow or corresponding function. The processor described in this embodiment can be used for the operation of a small-sample rotating device damage identification method based on a multimodal large model, including:

[0113] Based on a pre-trained image-semantic large model, feature vectors of damaged images and feature vectors of damaged text are extracted. The distance between the damaged image features and text features is quantified by a similarity function to represent the prior knowledge of the damage type, thus obtaining the large model image-semantic prior knowledge. The image encoder and the damage hot-coded unique code extract small sample image-semantic knowledge containing the damage category. The feature vector matrix converted by the image encoder and the label matrix obtained by the hot-coded unique code are used to jointly construct a feature cache library containing the small sample image-semantic knowledge of the damage category. The obtained large model image-semantic prior knowledge and the obtained feature cache library containing the small sample image-semantic knowledge of the damage category are fused to achieve high-precision identification of the damage type of rotating equipment under small sample conditions.

[0114] Please see Figure 6The terminal device is a computer device. In this embodiment, the computer device 60 includes a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When executed by the processor 61, the computer program 63 implements the small-sample rotating device damage identification method based on a multimodal large model as described in this embodiment. To avoid repetition, these details are not elaborated here. Alternatively, when executed by the processor 61, the computer program 63 implements the functions of each model / unit in the small-sample rotating device damage identification system based on a multimodal large model as described in this embodiment. To avoid repetition, these details are not elaborated here.

[0115] Computer device 60 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. Computer device 60 may include, but is not limited to, a processor 61 and a memory 62. Those skilled in the art will understand that... Figure 6 This is merely an example of computer device 60 and does not constitute a limitation on computer device 60. It may include more or fewer components than shown, or combine certain components, or different components. For example, computer device may also include input / output devices, network access devices, buses, etc.

[0116] The processor 61 may be a Central Processing Unit (CPU), or other general-purpose processors, graphics processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0117] The memory 62 can be an internal storage unit of the computer device 60, such as a hard disk or RAM of the computer device 60. The memory 62 can also be an external storage device of the computer device 60, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on the computer device 60.

[0118] Furthermore, the memory 62 may include both internal storage units of the computer device 60 and external storage devices. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 can also be used to temporarily store data that has been output or will be output.

[0119] Please see Figure 7 The terminal device 600 is an electronic device, which takes the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including storage unit 620 and processing unit 610), a display unit 640, etc.

[0120] The storage unit stores program code, which can be executed by the processing unit 610 to perform the steps described in the method section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 610 can perform actions such as... Figure 1 The steps are shown in the figure.

[0121] Storage unit 620 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 6201 and / or cache memory 6202, and may further include a read-only memory (ROM) 6203.

[0122] Storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0123] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the multiple bus structures.

[0124] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 660. Network adapter 660 can communicate with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0125] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium, which is a memory device in a terminal device for storing programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and extended storage media supported by the terminal device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor; these instructions can be one or more computer programs. It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device.

[0126] One or more instructions stored in a computer-readable storage medium can be loaded and executed by a processor to implement the corresponding steps of the damage identification method for small-sample rotating equipment based on a multimodal large model in the above embodiments; one or more instructions in the computer-readable storage medium are loaded and executed by the processor in the following steps:

[0127] Based on a pre-trained image-semantic large model, feature vectors of damaged images and feature vectors of damaged text are extracted. The distance between the damaged image features and text features is quantified by a similarity function to represent the prior knowledge of the damage type, thus obtaining the large model image-semantic prior knowledge. The image encoder and the damage hot-coded unique code extract small sample image-semantic knowledge containing the damage category. The feature vector matrix converted by the image encoder and the label matrix obtained by the hot-coded unique code are used to jointly construct a feature cache library containing the small sample image-semantic knowledge of the damage category. The obtained large model image-semantic prior knowledge and the obtained feature cache library containing the small sample image-semantic knowledge of the damage category are fused to achieve high-precision identification of the damage type of rotating equipment under small sample conditions.

[0128] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0129] Please see Figure 5 Using only the CLIP model without training, the damage identification accuracy reached 32.29%, reflecting the potential of the multimodal large model CLIP in rotating equipment damage identification. However, it is limited by the domain differences between the general dataset and the rotating equipment-specific dataset in the large model. After incorporating rotating equipment damage knowledge, the model's damage identification accuracy reached 93.75%, significantly higher than the 84.38% accuracy without incorporating CLIP model prior knowledge. Furthermore, the model with optimized hyperparameters achieved a maximum test accuracy of 94.79%. These results further demonstrate that the introduction of small-sample rotating equipment damage knowledge can effectively compensate for the domain bias between the large model's pre-training dataset and the rotating equipment dataset, while the introduction of CLIP prior knowledge from the large model also helps improve the accuracy of rotating equipment damage identification under small-sample conditions.

[0130] In summary, this invention presents a method and system for identifying damage to rotating equipment using a small sample size based on a multimodal large model. This invention significantly improves the accuracy of rotating equipment damage identification by innovatively fusing the untrained CLIP multimodal large model with specialized knowledge of rotating equipment damage. Initially, using only the CLIP model, the damage identification accuracy is 32.29%, demonstrating the application potential of CLIP in the field of rotating equipment damage identification, but limited by the domain difference between the model's pre-training dataset and the specialized rotating equipment dataset. By introducing rotating equipment damage knowledge, the model's damage identification accuracy is improved to 93.75%, a result significantly higher than the 84.38% accuracy without incorporating CLIP prior knowledge. Furthermore, through hyperparameter optimization, the model's test accuracy can reach a maximum of 94.79%. These results further confirm that the introduction of small sample rotating equipment damage knowledge can effectively compensate for the domain bias between the large model's pre-training dataset and the rotating equipment dataset, while the prior knowledge of the large model CLIP also plays a positive role in improving the accuracy of rotating equipment damage identification under small sample conditions.

[0131] Therefore, this invention not only improves the accuracy of damage identification in rotating equipment, but also enhances the generalization ability of the model under small sample conditions, providing an efficient and accurate identification method for the field of rotating equipment damage detection.

[0132] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0133] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0134] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0135] In the embodiments provided by this invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0136] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0137] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0138] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random-access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0139] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0140] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0141] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0142] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A small sample rotating equipment damage identification method based on a multi-modal large model, characterized in that, The method comprises the following steps: S1, based on the pre-trained image-semantics large model, extracting damage image feature vectors and damage description text feature vectors, and quantifying the distance between the damage image features and the text features with a similarity function to represent the prior knowledge of the damage type, obtaining the large model image-semantics prior knowledge, specifically: S101, the visual encoder and the text encoder are set to the pre-trained ViT-B / 16 architecture and the pre-trained text Transformer architecture respectively; a large amount of paired data of natural images and texts is used to train the image-semantics large model; S102, using the pre-trained image-semantic large model to extract the damage image feature vector and the damage text feature vector respectively; constructing a text database of the to-be-identified damage type, using a text encoder to convert the damage text into C dimensional text feature vector library, using an image encoder to convert the to-be-tested damage image into C dimensional image feature vector, realizing the extraction of the damage image and the text feature; S103, the similarity is used to quantify the distance between the damage image features and the text features; the image and text features are combined in pairs, the similarity of the combined pairs is calculated, and an image-text similarity matrix is obtained, the elements on the diagonal of the matrix belong to positive samples, representing correctly paired images and texts, and the remaining elements are negative samples; The similarity of the positive samples is maximized, and the similarity of the negative samples is minimized; S104, calculate the similarity between the feature vector and the text feature, and generate a text-image feature matching result vector , as a large model image-semantic prior knowledge, wherein is expressed as a text feature matrix extracted by a text encoder; S2, jointly extracting small sample image-semantics knowledge containing damage categories by combining image encoders and damage hot encoding; the feature vector matrix converted by the image encoder and the label matrix obtained by hot encoding are jointly constructed to form a feature cache library of small sample image-semantics knowledge containing damage categories, and the small sample image-semantics knowledge containing damage categories is extracted, specifically: S201、constructing a training set containing small sample images with labels N S201、constructing a training set containing small sample images with labels S202, a small sample damage knowledge extraction module containing a double-layer perception machine is constructed, and a pre-trained image encoder is used to convert a small sample image training set into a feature vector matrix , as the initial weights of the double-layer linear network structure . S203, the small sample image training set is one-hot encoded to obtain a label matrix , the small sample damage image data is used as a double-layer linear network structure , cross-entropy is used as a loss function, and pre-training knowledge extracted by combining the image encoder and the text encoder in step S1 is input , the double-layer linear network structure is co-trained ; S204, the double-layer linear network structure trained in step S203 weights After L2 regularization, the obtained as the final feature knowledge of small sample damage image, joint label matrix construct a feature cache library containing small sample image-semantics knowledge of damage category; S3, the large model image-semantics prior knowledge obtained in step S1 and the feature cache library of small sample image-semantics knowledge containing damage categories obtained in step S2 are fused to realize high-precision recognition of damage types of rotating equipment under small sample conditions, specifically: S301. Pass the image of the damage to be tested through an image encoder to obtain the image prior knowledge features normalized to the L2 norm. ; Incorporate prior knowledge features of images As a query vector, it matches the few-sample damage feature matrix in the cache. The membership degree is calculated. ; S302, The membership degree obtained in step S301 is... With the label matrix in the feature buffer Linear combination is performed to obtain the damage type identification results. ; S303, the small sample feature cache obtained in step S2 and the damage type recognition result of the large model prior knowledge obtained in step S1 are combined by weighting to obtain the final rotating equipment damage image type to be recognized.

2. The small sample rotating equipment damage identification method based on a multi-modal large model according to claim 1, characterized in that, In step S201, in the small sample image training set, each type of damage image has K Zhang, to constitute a small sample image training set.

3. The small sample rotating equipment damage identification method based on a multi-modal large model according to claim 1, characterized in that, In step S202, the double-layer linear network structure The input layer parameters are C The output layer parameters are NK , C represent the number of categories, NK represent the number of each category of damage image multiplied by the total number of categories, and the feature vector matrix is as follows: wherein, is a pre-trained image encoder; is a vector concatenation operation.

4. The small sample rotating equipment damage identification method based on a multi-modal large model according to claim 1, characterized in that, In step S203, the label matrix is: wherein, is a one-hot encoding operation.

5. The small sample rotating equipment damage identification method based on a multi-modal large model according to claim 1, characterized in that, The membership A is: wherein, to adjust hyperparameters; to be equivalent to large model prior knowledge similarity between small sample feature matrix similarity between small sample feature matrix 6. The small sample rotating equipment damage identification method based on a multi-modal large model according to claim 1, characterized in that, The final rotating equipment damage image type to be recognized is wherein, are weight coefficients, logits are membership probabilities for each class of damage type.

7. A small sample rotating equipment damage identification system based on a multi-modal large model, characterized in that, It comprises: The quantization module extracts damage image feature vectors and damage description text feature vectors based on a pre-trained image-semantics large model, and quantifies the distance between the damage image features and the text features with a similarity function to represent the prior knowledge of the damage type, obtains the large model image-semantics prior knowledge, specifically: The visual encoder and the text encoder are set to the pre-trained ViT-B / 16 architecture and the pre-trained text Transformer architecture respectively; a large amount of paired data of natural images and texts is used to train the image-semantics large model; The pre-trained image-semantic large model is used to extract the damage image feature vector and the damage text feature vector respectively; a text database of the to-be-identified damage type is constructed, and a text encoder is used to convert the damage text into C dimensional text feature vector library, and an image encoder is used to convert the to-be-tested damage image into C dimensional image feature vector, so as to realize the extraction of the damage image and text features; The similarity is used to quantify the distance between the damage image features and the text features; the image and text features are combined in pairs, the similarity of the combined pairs is calculated, and an image-text similarity matrix is obtained, the elements on the diagonal of the matrix belong to positive samples, representing correctly paired images and texts, and the remaining elements are negative samples; The similarity of the positive samples is maximized, and the similarity of the negative samples is minimized; The similarity between the feature vector and the text feature is calculated to generate a text-image feature matching result vector , as a large model image-semantic prior knowledge, wherein is expressed as a text feature matrix of all damage description texts extracted by the text encoder The extraction module extracts small sample image-semantic knowledge containing damage categories by combining the image encoder with the damage heat one-hot encoding, and jointly heat one-hot encoding of the feature vector matrix converted by the image encoder and the label matrix to construct a feature cache library of small sample image-semantic knowledge containing damage categories. The extraction of small sample image-semantic knowledge containing damage categories is specifically: constructing a training set of small sample images with labels N a training set of small sample images with labels A small sample damage knowledge extraction module including a double-layer perceptron is constructed, and a pre-trained image encoder is used to convert a small sample image training set into a feature vector matrix , as initial weights of a double-layer linear network structure . The small sample image training set is one-hot encoded to obtain a label matrix Using small sample damage image data as a double-layer linear network structure Input, cross-entropy as a loss function, combining pre-training knowledge extracted by the quantization module image encoder and the text encoder Co-training the double-layer linear network structure ; The weights of the trained double-layer linear network structure are subjected to L2 regularization , and the obtained are taken as the final feature knowledge of the small sample damage image, and the joint label matrix is used to construct a feature cache library containing small sample image-semantic knowledge of damage categories; The recognition module fuses the large model image-semantic prior knowledge and the feature cache library of small sample image-semantic knowledge containing damage categories to realize high-precision recognition of the damage type of the rotating equipment under the condition of small samples. Specifically: The image of the damage to be tested is passed through an image encoder to obtain the L2-norm normalized image prior knowledge features. ; Incorporate prior knowledge features of images As a query vector, it matches the few-sample damage feature matrix in the cache. The membership degree is calculated. ; The obtained membership degrees with the tag matrix in the feature cache Performing linear combination to obtain the recognition result of damage type ; The small sample feature cache obtained by the extraction module and the damage type recognition result of the large model prior knowledge obtained by the quantization module are weighted and combined to obtain the final rotating equipment damage image type to be identified.

Images