Multimodal model style embedding method and system for complex power visual scenes

Abstract

The invention discloses a multimodal model style embedding method and system for complex electric power visual scenes, belonging to the field of image processing technology, and solving the problem of how to enhance the style expression ability of visual language pre-training models. The invention utilizes a diffusion model to assist a multimodal model in establishing domain-level style prompt words in a continuous space, and transfers the knowledge of the diffusion model to the multimodal model, so that the method has richer style expression and modeling capabilities when facing complex electric power visual scenes, and can describe more detailed style information. The invention establishes an instance-level style feature extraction model, aligns the instance-level style feature extraction model with the style information of a domain-level style information library, and can generate efficient and accurate style prompt words for a single image input during the inference process, so that the method is suitable for tasks such as defect recognition and target detection in real electric power scenes, and effectively enhances the style generalization performance of visual text pre-training models in downstream tasks.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and relates to a multimodal model style embedding method and system for complex electric power visual scenes. Background Art

[0002] In recent years, visual language pre-training models represented by the CLIP model have demonstrated strong processing capabilities in visual tasks by pre-training on a large amount of visual text pairing data, and have good zero-sample recognition capabilities for conventional natural image processing. However, for specific downstream tasks, especially for samples that are not common in pre-training data, the domain generalization ability of visual language pre-training models is still limited.

[0003] Most of the existing technologies improve the performance of downstream tasks by fine-tuning model parameters. For example, the invention patent with publication number CN118228200A discloses a method for identifying abnormalities of power equipment based on a multimodal model. According to the historical equipment information group, each initial power equipment abnormality identification network is trained to generate each network weight information and each network structure information, and further generate the target network weight information and the target network structure information, construct a multi-source power equipment abnormality identification network, and input the target power equipment information into the multimodal power equipment abnormality identification model to obtain the target power equipment abnormality identification result.

[0004] The invention patent with publication number CN118072104A discloses a method for improving image classification by using a multimodal model to assist. According to the image to be classified or multiple frames of images in the video to be classified, a pre-trained single-modal classification model is used to output multiple possible classification results and the probabilities of each classification result. The image, classification result and the probability of each classification result are input into a given multimodal model, and the multimodal model is driven to conduct multiple rounds of self-questioning and self-answering on the image content, and multiple question and answer results are obtained. The multimodal model is driven to select the best matching classification from the classification results based on the question and answer results.

[0005] For specific downstream tasks, existing technologies only focus on visual text alignment in downstream tasks, but do not pay attention to how to effectively model domain and style information. Since the current visual language pre-training model lacks explicit expression of style information, the generalization performance of the model is limited in complex and changing visual scenes, resulting in the visual text pre-training model's weak ability to depict style. Therefore, how to enhance the style expression ability of the visual language pre-training model is particularly important in visual tasks. Summary of the invention

[0006] The technical problem to be solved by the present invention is how to enhance the style expression ability of the visual language pre-training model.

[0007] The present invention solves the above technical problems through the following technical solutions:

[0008] The multimodal model style embedding method for complex power visual scenes includes the following steps:

[0009] S1. Acquire image data and semantic category text data of electric power visual scenes and construct a training set, wherein the training set includes images and semantic category texts;

[0010] S2. Based on the Pytorch deep learning framework, a domain-level style information model is established, wherein the domain-level style information model includes a CLIP model, a diffusion model, and a learnable style cue vocabulary; the CLIP model and the diffusion model are jointly learned to perform gradient updates on the style cue vocabulary to obtain a domain-level style information library;

[0011] S3, randomly crop the training set images, scale all the training set images to a uniform size, and then perform numerical normalization;

[0012] S4. Based on the Pytorch deep learning framework, an instance-level style feature extraction model is established to extract the style information of a single image, conduct cross-modal knowledge interaction, and perform gradient updates on the instance-level style feature extraction model.

[0013] Furthermore, the image data includes images of power transmission, transformation and distribution scenes taken by inspection robots and fixed cameras, and the semantic category data includes semantic category texts that need to be considered in image recognition tasks.

[0014] Further, the S2 includes:

[0015] S21. Build a learnable style cue vocabulary;

[0016] S22, combining the semantic category dataset and the style hint vocabulary to obtain text with a specific style, inputting the text branch of the CLIP model to obtain text features with a specific style;

[0017] S23, inputting text features of a specific style into a diffusion model, generating an image corresponding to the text features, and inputting the image into a visual branch of a CLIP model to obtain image features of a specific style;

[0018] S24. Based on the style-specific text features and the style-specific image features, the content consistency constraint and the style space consistency loss are calculated, and the style hint vocabulary is gradient updated to obtain a domain-level style information library.

[0019] Furthermore, in S22, the text features of a specific style are obtained using the following logic:

[0020]

[0021]

[0022] In the formula, represents the i-th semantic category in the semantic category dataset, Indicates the semantic category And the style is A text description of Indicates the semantic category is And the style is The text features of represents the jth style hint word in the style hint word library, Represents the text branch of the CLIP model.

[0023] Furthermore, in S23, the image features of a specific style are obtained using the following logic:

[0024]

[0025]

[0026] In the formula, Represents the diffusion model. The model structure adopts the Stable Diffusion model structure based on UNet. Representing text features Diffusion model The generated image, Representing text features The corresponding image features are Represents the vision branch of the CLIP model.

[0027] Furthermore, in S24, the content consistency constraint is calculated using the following logic:

[0028]

[0029] In the formula, Represents content consistency constraints, Indicates the semantic category is And it does not contain style text features;

[0030] The style space consistency loss is calculated using the following logic:

[0031]

[0032] In the formula, represents the style space consistency loss, Represents probability distribution and The KL divergence between represents the i-th semantic category and the style is The probability distribution of represents the jth semantic category and the style is The probability distribution of , represents the number of semantic categories, and K represents the number of style cue lexicons.

[0033] Further, the S4 includes:

[0034] S41, input the training set image into the instance-level style feature extraction model to obtain the instance-level style prompt word, input the instance-level style prompt word into the text branch of the CLIP model to obtain the text feature of the training set image input; input the training set image into the visual branch of the CLIP model to obtain the visual feature of the training set image input;

[0035] S42, searching for the nearest neighbor of the instance-level style cue word in the domain-level style information library, constructing the nearest neighbor matching constraint, constructing the cross-modal alignment loss for the text features and visual features of the training set image input, and performing gradient update on the instance-level style feature extraction model;

[0036] S43. Additional image data of electric power visual scenes is obtained as a test set, input into the instance-level style feature extraction model, and the accuracy of the model is calculated.

[0037] Furthermore, in S41, the text features of the training set image input are obtained using the following logic:

[0038]

[0039]

[0040]

[0041] In the formula, represents instance-level style hint words, represents the input training set image, Represents the instance-level style feature extraction model. The model structure adopts the ResNet18 feature extraction model, and the model parameters adopt the ImageNet pre-training parameters. Indicates the semantic category is And the style is A text description of Indicates the semantic category is And the style is Text features of

[0042] The following logic is used to obtain the visual features of the training set image input:

[0043]

[0044] In the formula, Represents the visual features of the training set image input.

[0045] Furthermore, in S42, the nearest neighbor matching constraint is constructed using the following logic:

[0046]

[0047] In the formula, represents the nearest neighbor matching constraint, Indicates the semantic category is The text features of Text features of

[0048] The cross-modal alignment loss is constructed using the following logic:

[0049]

[0050]

[0051] In the formula, represents the cross entropy loss, Representation calculation and The cosine similarity between Representation calculation and The cosine similarity between represents exponential operation, Indicates the semantic category is And the style is The text features of Represents the training set images Category probabilities output by the instance-level style feature extraction model.

[0052] A multimodal model style embedding system for complex electric power visual scenes, including a dataset building module, a domain-level style information model building module, a training set image building module, and an instance-level style feature extraction model building module;

[0053] The data set construction module is used to obtain image data and semantic category text data of power visual scenes and construct a training set, wherein the training set includes images and semantic category text;

[0054] The domain-level style information model construction module is used to establish a domain-level style information model based on the Pytorch deep learning framework, wherein the domain-level style information model includes a CLIP model, a diffusion model, and a learnable style hint vocabulary; the CLIP model and the diffusion model are jointly learned to perform gradient updates on the style hint vocabulary to obtain a domain-level style information library;

[0055] The training set image construction module is used to randomly crop the training set images, scale all the training set images to a uniform size, and then perform numerical normalization processing;

[0056] The instance-level style feature extraction model construction module is used to establish an instance-level style feature extraction model based on the Pytorch deep learning framework, extract style information of a single image, perform cross-modal knowledge interaction, and perform gradient update on the instance-level style feature extraction model.

[0057] The advantages of the present invention are:

[0058] The present invention utilizes a diffusion model to assist a multimodal model in establishing domain-level style prompt words in a continuous space, and transfers the knowledge of the diffusion model to the multimodal model. Compared with artificially defined style information, the present invention has richer style expression and modeling capabilities when facing complex power visual scenes, and can describe more detailed style information. In addition, the present invention establishes an instance-level style feature extraction model, aligns the instance-level style feature extraction model with the style information of a domain-level style information library, and can generate efficient and accurate style prompt words for a single image input during the reasoning process. The style prompt words are suitable for tasks such as defect recognition and target detection in real power scenes, and effectively enhance the style generalization performance of the visual text pre-training model in downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 is a flow chart of a multimodal model style embedding method for complex electric power visual scenes according to the first embodiment of the present invention;

[0060] Figure 2 is a schematic diagram of a domain-level style information model according to the first embodiment of the present invention;

[0061] Figure 3 Schematic diagram of an instance-level style feature extraction model according to the first embodiment of the present invention. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and specific embodiments:

[0064] Embodiment 1

[0065] like Figure 1 Specifically, a multimodal model style embedding method for complex electric power visual scenes is disclosed, comprising the following steps:

[0066] S1. Acquire image data and semantic category text data of electric power visual scenes and construct a training set, wherein the training set includes images and semantic category texts;

[0067] In this embodiment, the image data includes images of power transmission and distribution scenes taken by inspection robots and fixed cameras, and the images cover image data in different provinces and cities, different geographical environments, and different weather conditions (such as sunny days, rainy days, foggy days, and snowy days); a semantic category data set of power visual scenes is obtained, and the semantic category data includes semantic category texts that need to be considered in image recognition tasks, and a training set is constructed based on the image data and the semantic category text data, and the training set includes images and semantic category texts.

[0068] S2. Based on the Pytorch deep learning framework, a domain-level style information model is established. The domain-level style information model includes the CLIP model and the diffusion model. and a learnable vocabulary of style cues ; Use the CLIP model and the diffusion model to jointly learn, perform gradient updates on the style hint vocabulary, and obtain a domain-level style information library;

[0069] like Figure 2 As shown, S2 includes the following specific steps:

[0070] S21. Build a learnable style cue vocabulary;

[0071] In this embodiment, the style hint word library ,in, Respectively represent the 1st to style hint words, which are independent of the text category and are saved in the form of vectors.

[0072] In this embodiment, a style hint word library is constructed based on the fact that there is no style text description in the semantic category text data set. To further enhance the style information contained in the semantic category text dataset, the style prompt words are saved in the form of vectors rather than directly understandable natural language texts, which are different from the general artificially set style prompt words (such as sunny, rainy, evening, noon and other environmental style prompt words).

[0073] S22, combining the semantic category dataset and the style hint vocabulary to obtain text with a specific style, inputting the text branch of the CLIP model to obtain text features with a specific style;

[0074] In this embodiment, the CLIP model is trained based on large-scale image-text pairs by contrastive learning. The CLIP model includes a visual branch and text branch , where the visual branch Use ResNet or Transformer structure, text branch Adopt Transformer structure.

[0075] The text branch of the CLIP model It is often used to accept text input containing only categories but not styles, such as the unstyled text "A photo of a [class].". The text description is represented as an image containing a specific semantic category as class. Taking a specific downstream image recognition task as an example, "A photo of a transformer." means an image containing a transformer device.

[0076] In this embodiment, the following logical representation is used to obtain text features without style:

[0077]

[0078]

[0079] In the formula, represents the i-th semantic category, Indicates the semantic category is and does not contain a textual description of the style, Indicates the semantic category is And does not contain style text features.

[0080] In this embodiment, the style hint word library Each style cue word in the style cue word library represents a specific style. The style cue words in the style cue word library are combined with the semantic categories in the semantic category dataset to obtain a text “A [ ] style photo of a [class]. This embodiment uses the following logic to obtain text features of a specific style:

[0081]

[0082]

[0083] In the formula, represents the i-th semantic category in the semantic category dataset, Indicates the semantic category is And the style is A text description of Indicates the semantic category is And the style is The text features of Represents the jth style cue word in the style cue word library.

[0084] S23, inputting text features of a specific style into a diffusion model, generating an image corresponding to the text features, and inputting the image into a visual branch of a CLIP model to obtain image features of a specific style;

[0085] In this embodiment, the diffusion model The model structure adopts the Stable Diffusion model structure based on UNet; this embodiment uses the text features of a specific style to establish a communication diffusion model The bridge between the pre-trained model and the style-specific text features is fed into the diffusion model. Generate an image corresponding to the text feature of a specific style, and input the image feedback into the visual branch of the pre-trained model. This embodiment uses the following logic to obtain the image feature of a specific style:

[0086]

[0087]

[0088] In the formula, Representing text features The image generated by the diffusion model D, Representing text features The corresponding image features.

[0089] S24, based on the text features of a specific style and the image features of a specific style, calculate the content consistency constraint and the style space consistency loss, perform gradient update on the style hint word library, and obtain a domain-level style information library;

[0090] In this embodiment, due to the diffusion model There is a difference between the domain style knowledge of the CLIP model itself, and the visual features and text features of a specific style are not completely aligned. In order to implement the diffusion model The knowledge transfer to the pre-trained model takes the text features and visual features corresponding to the styleless text "A photo of a [class]." as anchor points and considers the content consistency constraint, so that the image features of a specific style are surrounded by the styleless text features, which has good semantic discriminability. This embodiment uses the following logic to calculate the content consistency constraint:

[0091]

[0092] In the formula, Represents a content consistency constraint.

[0093] For different semantic categories, the feature offset caused by the same style cue word should remain consistent, so as to learn a good spatial structural relationship between different style features. In order to construct the relationship between different styles within the same semantic category, this embodiment uses the similarity between image features of different styles to represent the relationship between different styles within the same semantic category and form a probability distribution.

[0094] In this embodiment, the following logic is used to represent the relationship between different styles within the same semantic category:

[0095]

[0096] In the formula, For the i-th semantic category and style and The similarity probability between represents the i-th semantic category and the style is The image features corresponding to the text features, represents the i-th semantic category and the style is The image features corresponding to the text features, represents the i-th semantic category and the style is The image features corresponding to the text features, cos( , )express and The cosine similarity of express and The cosine similarity of represents the temperature coefficient, which is 0.07, and K represents the number of style hint lexicons. According to this expression, , so As the i-th semantic category and style The probability distribution of .

[0097] In this embodiment, the following logic is used to represent the style space consistency loss:

[0098]

[0099] In the formula, represents the style space consistency loss, Represents probability distribution and The KL divergence between represents the jth semantic category and the style is The probability distribution of , Indicates the number of semantic categories.

[0100] In this embodiment, the style information is taken as a set of “basis vectors” in a continuous text space, and the learnable style cue vocabulary is updated to a domain-level style information library through the joint learning of the CLIP model and the diffusion model and gradient backpropagation update.

[0101] like Figure 2 As shown, style hint words are optimized in a continuous space. Style hint words can take values ​​in a continuous space, and thus can represent an infinite number of styles. This is different from the fact that the number of style description words set manually is limited. The style information represented by the domain-level style information library in this embodiment is more detailed than the manually defined style information. At the same time, a new style vector can be constructed by combination, thereby realizing the effective expression of style information.

[0102] S3, randomly crop the training set images, scale all the training set images to a uniform size, and then perform numerical normalization;

[0103] In this embodiment, conventional random cropping is performed on the training set images, the size of all training set images is scaled to 256×256, and then numerical normalization is performed.

[0104] S4. Build an instance-level style feature extraction model based on the Pytorch deep learning framework , extract the style information of a single image, conduct cross-modal knowledge interaction, and perform gradient updates on the instance-level style feature extraction model;

[0105] like Figure 3 As shown, S4 includes the following specific steps:

[0106] S41. Input the training set images into the instance-level style feature extraction model to obtain instance-level style prompt words, input the instance-level style prompt words into the text branch of the CLIP model to obtain the text features of the training set image input; input the training set images into the visual branch of the CLIP model to obtain the visual features of the training set image input.

[0107] In this embodiment, the structure of the instance-level style feature extraction model E adopts the feature extraction model of ResNet18, and the parameters adopt the ImageNet pre-training parameters. The text features of the training set image input are obtained using the following logic:

[0108]

[0109]

[0110]

[0111] In the formula, represents instance-level style hint words, represents the input training set image, Indicates the semantic category is And the style is A text description of Indicates the semantic category is And the style is The text features of .

[0112] In this embodiment, the following logic is used to obtain the visual features of the training set image input:

[0113]

[0114] In the formula, Represents the visual features of the training set image input.

[0115] S42, searching for the nearest neighbor of the instance-level style cue word in the domain-level style information library, constructing the nearest neighbor matching constraint, constructing the cross-modal alignment loss for the text features and visual features of the training set image input, and performing gradient update on the instance-level style feature extraction model;

[0116] In this embodiment, the text features corresponding to the domain-level style information library obtained in step S2 are searched for neighbor features to shorten the distance between the two in the feature space. This embodiment uses the following logic to construct the nearest neighbor matching constraint:

[0117]

[0118] In the formula, represents the nearest neighbor matching constraint, Indicates the semantic category is The text features of In this embodiment, the distance between the features is determined by calculating the cosine similarity.

[0119] In this embodiment, the cross-modal alignment loss is constructed using the following logic:

[0120]

[0121]

[0122] In the formula, represents the cross entropy loss, Representation calculation and The cosine similarity between Representation calculation and The cosine similarity between represents exponential operation, Indicates the semantic category is And the style is The text features of Represents the training set images Category probabilities output by the instance-level style feature extraction model.

[0123] Although this embodiment achieves effective expression of style information by means of domain-level style information modeling, in the reasoning process of performing image recognition tasks, how to select appropriate style prompt words is still a difficult problem that needs to be solved. This embodiment further studies how to extract style information for a single image by establishing an instance-level style feature extraction model, and performs cross-modal knowledge interaction by integrating the extended modal knowledge of CLIP text and visual branches, continuously updating the instance-level style feature extraction model, selecting appropriate style prompt words, and achieving effective embedding of style information.

[0124] S43, additionally obtaining image data of electric power visual scenes as a test set, inputting the image data into the instance-level style feature extraction model, and calculating the accuracy of the model;

[0125] In this embodiment, the test set includes additional image data of power visual scenes, whose domain style and category information are different from those of the training set images, and is used to evaluate the generalization performance of the model. After multiple iterations of the instance-level style feature extraction model, the test set is input into the instance-level style feature extraction model to calculate the accuracy of the model.

[0126] Embodiment 2

[0127] The present invention also provides a multimodal model style embedding system for complex electric power visual scenes, including a data set construction module, a domain-level style information model construction module, a training set image construction module, and an instance-level style feature extraction model construction module;

[0128] The data set construction module is used to obtain image data and semantic category text data of power visual scenes and construct a training set, wherein the training set includes images and semantic category text;

[0129] In this embodiment, the image data includes images of power transmission and distribution scenes taken by inspection robots and fixed cameras, and the images cover image data in different provinces and cities, different geographical environments, and different weather conditions (such as sunny days, rainy days, foggy days, and snowy days); a semantic category data set of power visual scenes is obtained, and the semantic category data includes semantic category texts that need to be considered in image recognition tasks, and a training set is constructed based on the image data and the semantic category text data, and the training set includes images and semantic category texts.

[0130] The domain-level style information model construction module is used to establish a domain-level style information model based on the Pytorch deep learning framework, wherein the domain-level style information model includes a CLIP model, a diffusion model, and a learnable style hint vocabulary; the CLIP model and the diffusion model are jointly learned to perform gradient updates on the style hint vocabulary to obtain a domain-level style information library;

[0131] The domain-level style information model construction module includes a vocabulary construction unit, a text feature unit, an image feature unit and a vocabulary update unit;

[0132] The word library construction unit is used to construct a learnable style prompt word library;

[0133] In this embodiment, the style hint word library ,in, Respectively represent the 1st to style hint words, which are independent of the text category and are saved in the form of vectors.

[0134] In this embodiment, a style hint word library is constructed based on the fact that there is no style text description in the semantic category text data set. To further enhance the style information contained in the semantic category text dataset, the style prompt words are saved in the form of vectors rather than directly understandable natural language texts, which are different from the general artificially set style prompt words (such as sunny, rainy, evening, noon and other environmental style prompt words).

[0135] The text features It is used to combine the semantic category dataset with the style hint vocabulary to obtain text with a specific style, input it into the text branch of the CLIP model, and obtain text features with a specific style;

[0136] In this embodiment, the CLIP model is trained based on large-scale image-text pairs by contrastive learning. The CLIP model includes a visual branch and text branch , where the visual branch Use ResNet or Transformer structure, text branch Adopt Transformer structure.

[0137] The text branch of the CLIP model It is often used to accept text input containing only categories but not styles, such as the unstyled text "A photo of a [class].". The text description is represented as an image containing a specific semantic category as class. Taking a specific downstream image recognition task as an example, "A photo of a transformer." means an image containing a transformer device.

[0138] In this embodiment, the following logical representation is used to obtain text features without style:

[0139]

[0140]

[0141] In the formula, represents the i-th semantic category, Indicates the semantic category is and does not contain a textual description of the style, Indicates the semantic category is And does not contain style text features.

[0142] In this embodiment, the style hint word library Each style cue word in the style cue word library represents a specific style. The style cue words in the style cue word library are combined with the semantic categories in the semantic category dataset to obtain a text “A [ ] style photo of a [class]. This embodiment uses the following logic to obtain text features of a specific style:

[0143]

[0144]

[0145] In the formula, represents the i-th semantic category in the semantic category dataset, Indicates the semantic category is And the style is A text description of Indicates the semantic category is And the style is The text features of Represents the jth style cue word in the style cue word library.

[0146] The image feature unit is used to input text features of a specific style into a diffusion model, generate an image corresponding to the text features, and input the image into the visual branch of the CLIP model to obtain image features of a specific style;

[0147] In this embodiment, the diffusion model The model structure adopts the Stable Diffusion model structure based on UNet; this embodiment uses the text features of a specific style to establish a communication diffusion model The bridge between the pre-trained model and the style-specific text features is fed into the diffusion model. Generate an image corresponding to the text feature of a specific style, and input the image feedback into the visual branch of the pre-trained model. This embodiment uses the following logic to obtain the image feature of a specific style:

[0148]

[0149]

[0150] In the formula, Representing text features The image generated by the diffusion model D, Representing text features The corresponding image features.

[0151] The word library updating unit is used to calculate the content consistency constraint and the style space consistency loss based on the text features of the specific style and the image features of the specific style, and to perform gradient update on the style hint word library to obtain a domain-level style information library;

[0152] In this embodiment, due to the diffusion model There is a difference between the domain style knowledge of the CLIP model itself, and the visual features and text features of a specific style are not completely aligned. In order to implement the diffusion model The knowledge transfer to the pre-trained model takes the text features and visual features corresponding to the styleless text "A photo of a [class]." as anchor points and considers the content consistency constraint, so that the image features of a specific style are surrounded by the styleless text features, which has good semantic discriminability. This embodiment uses the following logic to calculate the content consistency constraint:

[0153]

[0154] In the formula, Represents a content consistency constraint.

[0155] For different semantic categories, the feature offset caused by the same style cue word should remain consistent, so as to learn a good spatial structural relationship between different style features. In order to construct the relationship between different styles within the same semantic category, this embodiment uses the similarity between image features of different styles to represent the relationship between different styles within the same semantic category and form a probability distribution.

[0156] In this embodiment, the following logic is used to represent the relationship between different styles within the same semantic category:

[0157]

[0158] In the formula, For the i-th semantic category and style and The similarity probability between represents the i-th semantic category and the style is The image features corresponding to the text features, cos( , )express and The cosine similarity of represents the temperature coefficient, which is 0.07, and K represents the number of style hint lexicons. According to this expression, , so As the i-th semantic category and style The probability distribution of .

[0159] In this embodiment, the following logic is used to represent the style space consistency loss:

[0160]

[0161] In the formula, represents the style space consistency loss, Represents probability distribution and The KL divergence between represents the jth semantic category and the style is The probability distribution of , Indicates the number of semantic categories.

[0162] The style hint words are optimized in a continuous space. The style hint words can take values ​​in a continuous space, and thus can represent an infinite number of styles. This is different from the fact that the number of style description words set manually is limited. The style information represented by the domain-level style information library in this embodiment is more detailed than the manually defined style information. At the same time, a new style vector can be constructed by combination, thereby realizing the effective expression of style information.

[0163] The training set image construction module is used to randomly crop the training set images, scale all the training set images to a uniform size, and then perform numerical normalization processing;

[0164] In this embodiment, conventional random cropping is performed on the training set images, the size of all training set images is scaled to 256×256, and then numerical normalization is performed.

[0165] The instance-level style feature extraction model construction module is used to establish an instance-level style feature extraction model based on the Pytorch deep learning framework, extract style information of a single image, perform cross-modal knowledge interaction, and perform gradient update on the instance-level style feature extraction model.

[0166] The instance-level style feature extraction model construction module includes a feature extraction unit, a gradient updating unit and a testing unit;

[0167] The feature extraction unit is used to input the training set image into the instance-level style feature extraction model to obtain the instance-level style prompt words, input the instance-level style prompt words into the text branch of the CLIP model to obtain the text features of the training set image input; input the training set image into the visual branch of the CLIP model to obtain the visual features of the training set image input.

[0168] In this embodiment, the structure of the instance-level style feature extraction model E adopts the feature extraction model of ResNet18, and the parameters adopt the ImageNet pre-training parameters. The text features of the training set image input are obtained using the following logic:

[0169]

[0170]

[0171]

[0172] In the formula, represents instance-level style hint words, represents the input training set image, Indicates the semantic category is And the style is A text description of Indicates the semantic category is And the style is The text features of .

[0173] In this embodiment, the following logic is used to obtain the visual features of the training set image input:

[0174]

[0175] In the formula, Represents the visual features of the training set image input.

[0176] The gradient updating unit is used to search for the nearest neighbor of the instance-level style cue word in the domain-level style information library, construct the nearest neighbor matching constraint, construct the cross-modal alignment loss for the text features and visual features of the training set image input, and perform gradient update on the instance-level style feature extraction model;

[0177] In this embodiment, the text features corresponding to the domain-level style information library obtained in step S2 are searched for neighbor features to shorten the distance between the two in the feature space. This embodiment uses the following logic to construct the nearest neighbor matching constraint:

[0178]

[0179] In the formula, represents the nearest neighbor matching constraint, Indicates the semantic category is The text features of In this embodiment, the distance between the features is determined by calculating the cosine similarity.

[0180] In this embodiment, the cross-modal alignment loss is constructed using the following logic:

[0181]

[0182]

[0183] In the formula, represents the cross entropy loss, Representation calculation and The cosine similarity between Representation calculation and The cosine similarity between represents exponential operation, Indicates the semantic category is And the style is The text features of Represents the training set images Category probabilities output by the instance-level style feature extraction model.

[0184] Although this embodiment achieves effective expression of style information by means of domain-level style information modeling, in the reasoning process of performing image recognition tasks, how to select appropriate style prompt words is still a difficult problem that needs to be solved. This embodiment further studies how to extract style information for a single image by establishing an instance-level style feature extraction model, and performs cross-modal knowledge interaction by integrating the extended modal knowledge of CLIP text and visual branches, continuously updating the instance-level style feature extraction model, selecting appropriate style prompt words, and achieving effective embedding of style information.

[0185] The testing unit is used to additionally obtain image data of power visual scenes as a test set, input the image data into an instance-level style feature extraction model, and calculate the accuracy of the model;

[0186] In this embodiment, the test set includes additional image data of power visual scenes, whose domain style and category information are different from those of the training set images, and is used to evaluate the generalization performance of the model. After multiple iterations of the instance-level style feature extraction model, the test set is input into the instance-level style feature extraction model to calculate the accuracy of the model.

[0187] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal model style embedding method for complex power visual scenes, characterized by: include: S1. Acquire image data and semantic category text data of electric power visual scenes and construct a training set, wherein the training set includes images and semantic category texts; S2. Based on the Pytorch deep learning framework, a domain-level style information model is established, wherein the domain-level style information model includes a CLIP model, a diffusion model, and a learnable style cue vocabulary; the CLIP model and the diffusion model are jointly learned to perform gradient updates on the style cue vocabulary to obtain a domain-level style information library; The S2 includes: S21. Build a learnable style cue vocabulary; S22, combining the semantic category dataset and the style hint vocabulary to obtain text with a specific style, inputting the text branch of the CLIP model to obtain text features with a specific style; S23, inputting text features of a specific style into a diffusion model, generating an image corresponding to the text features, and inputting the image into a visual branch of a CLIP model to obtain image features of a specific style; S24, based on the text features of a specific style and the image features of a specific style, calculate the content consistency constraint and the style space consistency loss, perform gradient update on the style hint word library, and obtain a domain-level style information library; The diffusion model adopts a Stable Diffusion model structure based on UNet; S3, randomly crop the training set images, scale all the training set images to a uniform size, and then perform numerical normalization; S4. Based on the Pytorch deep learning framework, an instance-level style feature extraction model is established to extract the style information of a single image, conduct cross-modal knowledge interaction, and perform gradient updates on the instance-level style feature extraction model.

2. According to claim 1, the multimodal model style embedding method for complex electric power visual scenes is characterized in that: The image data includes images of power transmission, transformation and distribution scenes taken by inspection robots and fixed cameras, and the semantic category data includes semantic category texts that need to be considered in image recognition tasks.

3. The multimodal model style embedding method for complex electric power visual scenes according to claim 1 is characterized in that: In S22, the text features of a specific style are obtained using the following logic: In the formula, represents the i-th semantic category in the semantic category dataset, Indicates the semantic category And the style is A text description of Indicates the semantic category And the style is The text features of represents the jth style hint word in the style hint word library, Represents the text branch of the CLIP model.

4. The multimodal model style embedding method for complex electric power visual scenes according to claim 3 is characterized in that: In S23, the image features of a specific style are obtained using the following logic: In the formula, Representation Diffusion Model Representing text features Through the diffusion model The generated image, Representing text features The corresponding image features are Represents the vision branch of the CLIP model.

5. The multimodal model style embedding method for complex electric power visual scenes according to claim 4 is characterized in that: In S24, the content consistency constraint is calculated using the following logic: In the formula, Represents content consistency constraints, Indicates the semantic category And it does not contain style text features; The style space consistency loss is calculated using the following logic: In the formula, represents the style space consistency loss, Represents probability distribution and The KL divergence between represents the i-th semantic category and the style is The probability distribution of represents the jth semantic category and the style is The probability distribution of , represents the number of semantic categories, and K represents the number of style cue lexicons.

6. The multimodal model style embedding method for complex electric power visual scenes according to claim 1 is characterized in that: The S4 includes: S41, input the training set image into the instance-level style feature extraction model to obtain the instance-level style prompt word, input the instance-level style prompt word into the text branch of the CLIP model to obtain the text feature of the training set image input; input the training set image into the visual branch of the CLIP model to obtain the visual feature of the training set image input; S42, searching for the nearest neighbor of the instance-level style cue word in the domain-level style information library, constructing the nearest neighbor matching constraint, constructing the cross-modal alignment loss for the text features and visual features of the training set image input, and performing gradient update on the instance-level style feature extraction model; S43. Additional image data of electric power visual scenes is obtained as a test set, input into the instance-level style feature extraction model, and the accuracy of the model is calculated.

7. The multimodal model style embedding method for complex electric power visual scenes according to claim 6 is characterized in that: In S41, the text features of the training set image input are obtained using the following logic: In the formula, represents instance-level style hint words, represents the input training set image, Represents the instance-level style feature extraction model. The model structure adopts the ResNet18 feature extraction model, and the model parameters adopt the ImageNet pre-training parameters. Indicates the semantic category And the style is A text description of Indicates the semantic category And the style is Text features of The following logic is used to obtain the visual features of the training set image input: In the formula, Represents the visual features of the training set image input.

8. The multimodal model style embedding method for complex electric power visual scenes according to claim 7 is characterized in that: In S42, the nearest neighbor matching constraint is constructed using the following logic: In the formula, represents the nearest neighbor matching constraint, Indicates the semantic category The text features of Text features of The cross-modal alignment loss is constructed using the following logic: In the formula, represents the cross entropy loss, Representation calculation and The cosine similarity between Representation calculation and The cosine similarity between represents exponential operation, Indicates the semantic category And the style is The text features of Represents the training set images Category probabilities output by the instance-level style feature extraction model.

9. A multimodal model style embedding system for complex power visual scenes, characterized by: It includes a dataset construction module, a domain-level style information model construction module, a training set image construction module, and an instance-level style feature extraction model construction module; The data set construction module is used to obtain image data and semantic category text data of power visual scenes and construct a training set, wherein the training set includes images and semantic category text; The domain-level style information model construction module is used to establish a domain-level style information model based on the Pytorch deep learning framework, wherein the domain-level style information model includes a CLIP model, a diffusion model, and a learnable style hint vocabulary; the CLIP model and the diffusion model are jointly learned to perform gradient updates on the style hint vocabulary to obtain a domain-level style information library; The domain-level style information model construction module includes a vocabulary construction unit, a text feature unit, an image feature unit and a vocabulary update unit; The word library construction unit is used to construct a learnable style cue word library; The text features It is used to combine the semantic category dataset with the style hint vocabulary to obtain text with a specific style, input it into the text branch of the CLIP model, and obtain text features with a specific style; The image feature unit is used to input text features of a specific style into a diffusion model, generate an image corresponding to the text features, and input the image into the visual branch of the CLIP model to obtain image features of a specific style; The word library updating unit is used to calculate the content consistency constraint and the style space consistency loss based on the text features of the specific style and the image features of the specific style, and to perform gradient update on the style hint word library to obtain a domain-level style information library; The diffusion model adopts a Stable Diffusion model structure based on UNet; The training set image construction module is used to randomly crop the training set images, scale all the training set images to a uniform size, and then perform numerical normalization processing; The instance-level style feature extraction model construction module is used to establish an instance-level style feature extraction model based on the Pytorch deep learning framework, extract style information of a single image, perform cross-modal knowledge interaction, and perform gradient update on the instance-level style feature extraction model.

Images