Establishment method and device of power image sample generation model, equipment and medium

Abstract

The invention discloses a power image sample generation model establishment method and device, equipment and a medium, and the method comprises the steps: obtaining initial power image data and defect type text data, respectively carrying out image feature extraction and text feature extraction on the initial power image data and the defect type text data based on a preset visual large model, and generating an image feature vector and a text feature vector; based on a modal alignment mechanism, performing feature fusion on the image feature vector and the text feature vector to generate a fused feature vector; based on the fusion feature vector and the training sample image data, training a classifier model and a discriminator model in the visual large model until a preset model training ending condition is met; and taking the trained visual large model as a power image sample generation model. According to the technical scheme, the efficient generation of the electric power scarce sample image data and the expansion of the sample scale are realized.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device, equipment and medium for establishing a power image sample generation model. Background Art

[0002] With the development of smart grids and the digital transformation of power systems, the power industry has an increasing demand for data-driven intelligent decision-making. Visual big models are being used more and more widely in the power industry to optimize equipment maintenance and fault diagnosis by analyzing image data of power systems.

[0003] In the process of training large visual models, the training effect of the model needs to rely on a large amount of training sample data. The more sufficient the training sample data, the better the training effect of the model. If there is insufficient training sample data for a certain sample label, the model cannot learn the data features of the sample label well, which leads to insufficient learning ability of the model for the sample label and reduced model performance. Therefore, it is very important to ensure sufficient training sample data in the model training process. Summary of the invention

[0004] The present invention provides a method, device, equipment and medium for establishing a power image sample generation model to increase the data scale of training sample data.

[0005] According to one aspect of the present invention, a method for establishing a power image sample generation model is provided, the method comprising:

[0006] Acquire initial power image data and defect type text data, perform image feature extraction and text feature extraction on the initial power image data and the defect type text data respectively based on a preset visual macro model, and generate an image feature vector and a text feature vector; wherein the power image data is an equipment image of a power device without defects; and the defect type text data is used to characterize defect information of the power device;

[0007] Based on a modality alignment mechanism, feature fusion is performed on the image feature vector and the text feature vector to generate a fused feature vector;

[0008] Based on the fused feature vector and the training sample image data, the classifier model and the discriminator model in the visual large model are trained until the preset model training end condition is met; wherein the classifier model is used to generate simulated power defect image data; the discriminator model is used to detect the similarity between the simulated power defect image data and the training sample image data; the simulated power defect image data and the training sample image data have the same power equipment defect;

[0009] The trained visual large model is used as the power image sample generation model.

[0010] According to another aspect of the present invention, a device for establishing a power image sample generation model is provided, the device comprising:

[0011] A feature extraction module is used to obtain initial power image data and defect type text data, and perform image feature extraction and text feature extraction on the initial power image data and the defect type text data respectively based on a preset visual macro model to generate an image feature vector and a text feature vector; wherein the power image data is an equipment image of a power device without defects; and the defect type text data is used to characterize defect information of the power device;

[0012] A feature fusion module, used for performing feature fusion on the image feature vector and the text feature vector based on a modality alignment mechanism to generate a fused feature vector;

[0013] A training module, used to train the classifier model and the discriminator model in the visual large model based on the fused feature vector and the training sample image data until the preset model training end condition is met; wherein the classifier model is used to generate simulated power defect image data; the discriminator model is used to detect the similarity between the simulated power defect image data and the training sample image data; the simulated power defect image data and the training sample image data have the same power equipment defect;

[0014] The model generation module is used to generate a model using the trained visual large model as a power image sample.

[0015] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for establishing a power image sample generation model described in any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for establishing a power image sample generation model described in any embodiment of the present invention when executed.

[0020] According to another aspect of the present invention, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the method for establishing a power image sample generation model according to any embodiment of the present invention is implemented.

[0021] The technical solution of the embodiment of the present invention generates sample image data by fusing multimodal data with initial power image data and defect type text data, which solves the limitation of generating sample image data with traditional single-modal data, realizes the efficient generation of scarce power samples and expands the sample size of scarce sample image data, thereby improving the accuracy and robustness of the power fault detection and diagnosis model.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 is a flowchart of a method for establishing a power image sample generation model provided according to Embodiment 1 of the present invention;

[0025] Figure 2 is a flow chart of a method for establishing a power image sample generation model provided according to Embodiment 2 of the present invention;

[0026] Figure 3 is a flowchart of a method for establishing a power image sample generation model provided in accordance with Embodiment 3 of the present invention;

[0027] Figure 4 is a structural schematic diagram of a device for establishing a power image sample generation model provided according to a fourth embodiment of the present invention;

[0028] Figure 5 It is a structural schematic diagram of an electronic device for implementing the method for establishing a power image sample generation model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Embodiment 1

[0032] Figure 1 A flowchart of a method for establishing a power image sample generation model is provided for the first embodiment of the present invention. This embodiment is applicable to the case of generating training sample data for a power fault detection and diagnosis model. The method can be executed by an apparatus for establishing a power image sample generation model. The apparatus for establishing a power image sample generation model can be implemented in the form of hardware and / or software. The apparatus for establishing a power image sample generation model can be configured in various general-purpose computing devices, such as a client device or server device configured with a large visual model. Figure 1 As shown, the method includes:

[0033] S110, obtaining initial power image data and defect type text data, performing image feature extraction and text feature extraction on the initial power image data and defect type text data based on a preset visual macro model, and generating an image feature vector and a text feature vector.

[0034] Among them, the power image data can be an equipment image of the power equipment without defects; the defect type text data can be used to characterize the defect information of the power equipment; optionally, the defect type text data may include the defect type information of the power equipment, the location information of the defect of the power equipment and the defect level information of the defect of the power equipment, etc.

[0035] In an embodiment of the present invention, the pre-set visual macro model can first perform data preprocessing on the input initial power image data and defect type text data, and perform data cleaning on the initial power image data and the multimodal data represented by the defect type text data, respectively, to ensure the accuracy and consistency of the data. Exemplarily, the initial power image data can be subjected to denoising, normalization, resolution adjustment and cropping operations to ensure the data quality and consistency of the initial power image data; the defect type text data can be subjected to text processing (word segmentation, stop word filtering, etc.).

[0036] After performing the data preprocessing operation, the visual big model can perform feature extraction on the preprocessed initial power image data and the defect type text data, respectively, to generate an image feature vector and a text feature vector. Exemplarily, the visual big model can extract the low-level edge information and high-level semantic features of the preprocessed initial power image data to generate an image feature vector; and the visual big model can convert the preprocessed defect type text data into a vector representation to generate a text feature vector. Optionally, the text feature vector can include semantic information of the text data, such as defect type, defect location, and context information.

[0037] Optionally, in an embodiment of the present invention, the user can customize the initial power image data and the defect type text data. For example, the user can provide the initial power image data of a certain type of power equipment, and the user can select the defect type text data according to their own needs, such as the defect type information of the power equipment, the location information of the defect of the power equipment, and the defect level information of the defect of the power equipment, so as to generate a power image sample that meets the user's requirements.

[0038] S120: Based on the modal alignment mechanism, perform feature fusion on the image feature vector and the text feature vector to generate a fused feature vector.

[0039] In an embodiment of the present invention, due to the large modal differences between image data and text data, direct feature fusion will cause some feature information to be lost or mismatched. Therefore, it is necessary to map the image feature vector and the text feature vector to the same feature space through linear transformation according to the modal alignment mechanism, so that the feature data of different modalities are comparable in the same dimension.

[0040] Optionally, based on the modal alignment mechanism, feature fusion is performed on the image feature vector and the text feature vector to generate a fused feature vector, including: linearly transforming the image feature vector and the text feature vector to map the image feature vector and the text feature vector to the same feature space; determining the weight relationship between the image feature vector and the text feature vector in the same feature space in parallel according to the multi-head attention mechanism, and bidirectionally fusing the image feature vector and the text feature vector according to the weight relationship to generate a fused feature vector.

[0041] Specifically, after the image feature vector and the text feature vector are mapped to the same feature space, weights can be automatically assigned to the image feature vector and the text feature vector according to the importance of the image feature vector and the text feature vector according to the multi-head attention mechanism and the adaptive feature weighting mechanism, and the image feature vector and the text feature vector are weighted and summed according to the assigned weights to generate the final fused feature vector, so that the data of different modalities are not only aligned in the feature space, but also complement each other semantically. Further, considering the bidirectionality in the feature fusion process, a bidirectional fusion mechanism can be adopted in the feature fusion process, which not only integrates the text feature vector into the image feature vector, but also feeds the image feature vector back to the text feature vector. Optionally, the importance of the image feature vector and the text feature vector can be adaptively set according to those skilled in the art, and the importance of the image feature vector and the text feature vector can be dynamically adjusted according to the training results of the model during the training process to improve the data quality of the fused feature vector.

[0042] By performing feature fusion through bidirectional feature fusion, the data quality of the fused features is further enhanced, making the generated fused features more three-dimensional and accurate.

[0043] S130, based on the fused feature vector and the training sample image data, the classifier model and the discriminator model in the visual large model are trained until the preset model training end condition is met.

[0044] Among them, the classifier model can be used to generate simulated power defect image data; the discriminator model can be used to detect the similarity between the simulated power defect image data and the training sample image data; the simulated power defect image data and the training sample image data have the same power equipment defects, and the defect information in the simulated power defect image data is consistent with the defect type text data.

[0045] In an embodiment of the present invention, the training sample image data may be image data of a defect in the history of the power equipment, and the defect type of the defect in the history is consistent with the description in the defect type text data, and the training sample image data may be used to verify the authenticity of the model power defect image data. Based on the fused feature vector and the training sample image data, the classifier model and the discriminator model in the visual large model may be trained until the preset model training end condition is met. Optionally, the model training end condition may be adaptively set according to those skilled in the art.

[0046] S140, using the trained visual large model as a power image sample generation model.

[0047] In an embodiment of the present invention, the simulated power defect image data output by the power image sample generation model can be output into the image format required by the user according to user needs, and supports the export of batch simulated power defect image data for further analysis or training of power fault detection and diagnosis models.

[0048] The technical solution of the embodiment of the present invention generates sample image data by fusing initial power image data and defect type text data into multimodal data, which solves the limitation of generating sample image data by traditional single-modal data, realizes the efficient generation of scarce power samples and the sample scale of scarce sample image data, and thus improves the accuracy and robustness of the power fault detection and diagnosis model.

[0049] Embodiment 2

[0050] Figure 2 This is a flowchart of a method for establishing a power image sample generation model provided in the second embodiment of the present invention. This embodiment further refines the above embodiment and provides specific steps for training the classifier model and the discriminator model in the visual model based on the fusion feature vector and the training sample image data until the preset model training end conditions are met. It should be noted that for the undetailed parts of the embodiments of the present invention, please refer to the relevant descriptions of other embodiments, which will not be repeated here. Figure 2 As shown, the method includes:

[0051] S210, obtaining initial power image data and defect type text data, performing image feature extraction and text feature extraction on the initial power image data and defect type text data based on a preset visual macro model, and generating an image feature vector and a text feature vector.

[0052] S220: Based on the modal alignment mechanism, perform feature fusion on the image feature vector and the text feature vector to generate a fused feature vector.

[0053] S230 : Determine a high confidence region in the initial power image data based on the initial power image data and the fused feature vector.

[0054] The high confidence region may be a region in the initial power image data where defects are most likely to occur.

[0055] Optionally, in an embodiment of the present invention, based on the initial power image data and the fused feature vector, a high confidence area in the initial power image data is determined, including: extracting high-level features of the initial power image data to generate an activation heat map; annotating the activation heat map according to the defect location information in the fused feature vector, determining the injection area of ​​the defect in the activation heat map, and determining the injection area as a high confidence area.

[0056] Specifically, the high-level features of the initial power image data can be extracted through the image feature extraction models pre-trained in the visual big model, such as the ViT (Vision Transformer) model and the CLIP (Contrastive Language–Image Pretraining) model, and then the activation heat map of the initial power image data can be generated according to the extracted high-level features; the location information of the defects of the power equipment carried in the fused feature vector is extracted, the activation heat map is annotated according to the location information, the injection area for defect injection into the activation heat map is determined, and the injection area is used as the high-confidence area.

[0057] By determining the high confidence area, the rationality and authenticity of the defect location of the generated defects are ensured.

[0058] S240, input the fused feature vector into the generator model for multi-layer feature extraction, and add defect features consistent with the defect type text data in the corresponding area of ​​each feature level according to the high confidence area to generate simulated power defect image data.

[0059] Specifically, the fused feature vector can be input into the generator model to perform multi-layer feature extraction in a progressive manner of feature scale to generate multiple feature levels, such as low-level feature layer, mid-level feature layer, and high-level feature layer. The position area corresponding to the high-confidence area at each feature level is determined, and defect features corresponding to the defect type text data are added to the position area to generate simulated power defect image data.

[0060] S250, input the simulated power defect image data and the training sample image data into the discriminator model, extract image features respectively, and compare the extracted image features at the same area position to determine the image similarity between the simulated power defect image data and the training sample image data.

[0061] Specifically, the simulated power defect image data and training sample image data output by the generator model can be input into the classifier model, and the classifier model divides the simulated power defect image data and the training sample image data into multiple local areas, and samples the same area of ​​the simulated power defect image data and the training sample image data respectively, and compares the local areas of the simulated power defect image data and the training sample image data respectively, and determines the image similarity between the two according to the similarity of each local area of ​​the two, thereby judging the authenticity of the simulated power defect image data and outputting the authenticity score of the simulated power defect image data.

[0062] S260, iteratively training the discriminator model and the generator model based on the first loss function and the second loss function respectively until a preset model training end condition is met.

[0063] Among them, the first loss function can be used to characterize the image similarity between the simulated power defect image data and the training sample image data, and the second loss function can maximize the score of the discriminator model on the simulated power defect image data.

[0064] It should be noted that, in the embodiment of the present invention, the generator model and the discriminator model are updated in an alternating manner during the iterative training process.

[0065] Optionally, the generator model and the discriminator model are iteratively trained based on the first loss function and the second loss function, including: when the model parameters of the generator model are fixed, calculating the gradient value of the first loss function and updating the model parameters of the discriminator model; when the model parameters of the discriminator model are fixed, calculating the gradient value of the second loss function and updating the model parameters of the discriminator model.

[0066] Specifically, in the process of alternating updates of the generator model and the discriminator model, in the first iterative training process, the model parameters of the generator model can be fixed first, and the image similarity between the simulated power defect image data and the training sample image data can be determined by the discriminator model. The image similarity can be represented by a binary cross entropy loss function (first loss function), and the loss gradient of the binary cross entropy loss function is calculated. According to the back propagation algorithm and the loss gradient, the model parameters of the discriminator model are updated. In the second iterative training process, the model parameters of the discriminator model can be fixed, and the simulated power defect image data output by the generator model is input into the discriminator model. According to the score of the simulated power defect image data output by the discriminator model, the loss gradient of the second loss function is calculated, and the model parameters of the generator model are updated according to the back propagation algorithm and the loss gradient. Repeat the above steps, and update the discriminator model and the generator model alternately until the preset model training end conditions are met.

[0067] By alternately training the discriminator model and the generator model, the data quality of the generated sample data can be continuously improved, and ultimately the generator model can generate sample data similar to the real data.

[0068] S270, using the trained large visual model as a power image sample generation model.

[0069] The technical solution of the embodiment of the present invention determines a high-confidence area for defect injection, injects defect features into the corresponding position area of ​​each feature level after the fusion feature vector feature is extracted, generates simulated power defect image data, improves the accuracy and credibility of the generated sample image data, and adopts a generative adversarial network to dynamically update the generator model and the discriminant model, thereby optimizing the data quality of the generated sample image data.

[0070] Embodiment 3

[0071] Figure 3 This is a flowchart of a method for establishing a power image sample generation model provided in the third embodiment of the present invention. This embodiment further refines the above embodiment and provides specific steps for inputting the fused feature vector into the generator model for multi-layer feature extraction, and adding defect features that are consistent with the defect type text data in the corresponding area of ​​each feature level according to the high confidence area to generate simulated power defect image data. It should be noted that for the undetailed parts of the embodiments of the present invention, please refer to the relevant descriptions of other embodiments, which will not be repeated here. Figure 3 As shown, the method includes:

[0072] S310, inputting the fused feature vector into the generator model, performing multi-layer feature extraction in sequence to generate a preset number of feature levels; the feature levels are cascaded layer by layer.

[0073] S320. Generate corresponding primary defect features of different scales for the defect type text data; wherein the primary defect features can be used to describe defect information corresponding to the defect type text data.

[0074] S330 , according to the scale priority order of the primary defect features and the hierarchical order of the feature hierarchies, the initial defect features are sequentially merged with the feature hierarchies to generate simulated power defect image data.

[0075] In an embodiment of the present invention, by inputting the fused feature vector into the generator model, multi-layer feature extraction can be performed in a progressive manner of feature scale to generate multiple feature levels, for example, a low-level feature layer, a mid-level feature layer, and a high-level feature layer, etc. The feature scales of the low-level feature layer, the mid-level feature layer, and the high-level feature layer are progressive in a cascade manner. Optionally, the number of feature levels can be adaptively set according to those skilled in the art.

[0076] Furthermore, the generator model can determine the defect information corresponding to the defect type text data, and obtain multi-scale primary defect features corresponding to the defect information, such as edge information features and structural information features. By adopting the jump connection technology, the primary defect features of different scales can be fused in a scale-increasing manner with the feature levels of the feature scale that are progressively cascaded, to ensure that the generated defects are continuous and progressive. It should be noted that in an embodiment of the present invention, a mapping relationship between defect information and its corresponding primary defect features of different scales is pre-established, and the primary defect features of different scales corresponding to different defect information can be determined based on the mapping relationship. Optionally, the number of features of the primary defect features is consistent with the number of feature levels.

[0077] Optionally, in an embodiment of the present invention, the generator model may introduce random factors such as Gaussian noise or Poisson noise in the process of generating simulated power defect image data, so that the generated simulated power defect image data has diversity in shape, size and position, thereby improving the data quality of the generated sample data.

[0078] The technical solution of the embodiment of the present invention gradually generates the depth, shape and internal structure of the defect by fusing the primary defect features of different scales with the feature levels that progress in a cascade manner, so as to ensure the continuity and progression of the generated defects and improve the data quality of the generated sample data.

[0079] Embodiment 4

[0080] Figure 4 This is a schematic diagram of a structure of a device for establishing a power image sample generation model provided by the fourth embodiment of the present invention. Figure 4 As shown, the device comprises:

[0081] The feature extraction module 410 is used to obtain initial power image data and defect type text data, and perform image feature extraction and text feature extraction on the initial power image data and the defect type text data respectively based on a preset visual macro model to generate an image feature vector and a text feature vector; wherein the power image data is an equipment image of a power device without defects; and the defect type text data is used to characterize defect information of the power device;

[0082] A feature fusion module 420 is used to perform feature fusion on the image feature vector and the text feature vector based on a modality alignment mechanism to generate a fused feature vector;

[0083] The training module 430 is used to train the classifier model and the discriminator model in the visual large model based on the fused feature vector and the training sample image data until the preset model training end condition is met; wherein the classifier model is used to generate simulated power defect image data; the discriminator model is used to detect the similarity between the simulated power defect image data and the training sample image data; the simulated power defect image data and the training sample image data have the same power equipment defect;

[0084] The model generation module 440 is used to use the trained visual large model as a power image sample generation model.

[0085] The technical solution of the embodiment of the present invention generates sample image data by fusing initial power image data and defect type text data into multimodal data, which solves the limitation of generating sample image data by traditional single-modal data, realizes the efficient generation of scarce power samples and the sample scale of scarce sample image data, and thus improves the accuracy and robustness of the power fault detection and diagnosis model.

[0086] Optionally, the training module 430 includes:

[0087] A region determination unit, configured to determine a high confidence region in the initial power image data based on the initial power image data and the fused feature vector; wherein the high confidence region is a region in the initial power image data that is most prone to defects;

[0088] An image generation unit, used for inputting the fused feature vector into a generator model for multi-layer feature extraction, and adding defect features consistent with the defect type text data in corresponding areas of each feature level according to the high confidence area, to generate simulated power defect image data;

[0089] An image verification unit, used to input the simulated power defect image data and the training sample image data into a discriminator model, respectively extract image features, and compare the extracted image features at the same area position to determine the image similarity between the simulated power defect image data and the training sample image data;

[0090] An iterative unit is used to iteratively train the discriminator model and the generator model based on a first loss function and a second loss function, respectively, until a preset model training end condition is met; wherein the first loss function is used to characterize the image similarity between the simulated power defect image data and the training sample image data, and the second loss function is used to characterize the discriminator model's scoring of the simulated power defect image data.

[0091] Optionally, the image generating unit includes:

[0092] A feature extraction subunit, used for inputting the fused feature vector into a generator model, and sequentially performing multi-layer feature extraction to generate a preset number of feature levels; wherein the feature levels are cascaded layer by layer with each other;

[0093] A primary defect feature subunit is used to generate primary defect features of different scales corresponding to the defect type text data; wherein the primary defect features are used to describe the defect information corresponding to the defect type text data;

[0094] The image generation subunit is used to fuse the primary defect features with the feature levels in sequence according to the scale priority order of the primary defect features and the hierarchical order of the feature levels to generate simulated power defect image data.

[0095] Optionally, the generator model and the discriminator model may be updated in an alternating manner during the iterative training process.

[0096] Optionally, the iteration unit may be specifically used to: when the model parameters of the generator model are fixed, calculate the gradient value of the first loss function and update the model parameters of the discriminator model; when the model parameters of the discriminator model are fixed, calculate the gradient value of the second loss function and update the model parameters of the discriminator model.

[0097] Optionally, the region determination unit may be specifically used to: extract high-level features of the initial power image data to generate an activation heat map; annotate the activation heat map according to the defect location information in the fused feature vector, determine the defect injection region in the activation heat map, and determine the injection region as a high confidence region.

[0098] Optionally, the feature fusion module 420 can be specifically used to: perform linear transformation on the image feature vector and the text feature vector to map the image feature vector and the text feature vector to the same feature space; determine the weight relationship between the image feature vector and the text feature vector in the same feature space in parallel according to the multi-head attention mechanism, and perform bidirectional feature fusion on the image feature vector and the text feature vector according to the weight relationship to generate a fused feature vector.

[0099] The apparatus for establishing a power image sample generation model provided in the embodiment of the present invention can execute the method for establishing a power image sample generation model provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0100] Embodiment 5

[0101] Figure 5 A schematic diagram of an electronic device 510 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0102] like Figure 5 As shown, the electronic device 510 includes at least one processor 511, and a memory connected to the at least one processor 511 in communication, such as a read-only memory (ROM) 512, a random access memory (RAM) 513, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 511 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 512 or the computer program loaded from the storage unit 518 to the random access memory (RAM) 513. In the RAM 513, various programs and data required for the operation of the electronic device 510 can also be stored. The processor 511, the ROM 512, and the RAM 513 are connected to each other via a bus 514. An input / output (I / O) interface 515 is also connected to the bus 514.

[0103] Multiple components in the electronic device 510 are connected to the I / O interface 515, including: an input unit 516, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 518, such as a disk, an optical disk, etc.; and a communication unit 519, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 519 allows the electronic device 510 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0104] The processor 511 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 511 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 511 performs the various methods and processes described above, such as a method for establishing a power image sample generation model.

[0105] In some embodiments, the method for establishing the power image sample generation model may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 518. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 510 via the ROM 512 and / or the communication unit 519. When the computer program is loaded into the RAM 513 and executed by the processor 511, one or more steps of the method for establishing the power image sample generation model described above may be performed. Alternatively, in other embodiments, the processor 511 may be configured to execute the method for establishing the power image sample generation model in any other appropriate manner (e.g., by means of firmware).

[0106] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0107] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0108] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0109] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0110] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0111] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0112] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0113] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for establishing a power image sample generation model, characterized in that: include: Acquire initial power image data and defect type text data, perform image feature extraction and text feature extraction on the initial power image data and the defect type text data respectively based on a preset visual macro model, and generate an image feature vector and a text feature vector; wherein the power image data is an equipment image of a power device without defects; and the defect type text data is used to characterize defect information of the power device; Based on a modality alignment mechanism, feature fusion is performed on the image feature vector and the text feature vector to generate a fused feature vector; Based on the fused feature vector and the training sample image data, the classifier model and the discriminator model in the visual large model are trained until the preset model training end condition is met; wherein the classifier model is used to generate simulated power defect image data; the discriminator model is used to detect the similarity between the simulated power defect image data and the training sample image data; the simulated power defect image data and the training sample image data have the same power equipment defect; The trained visual large model is used as the power image sample generation model.

2. The method according to claim 1, characterized in that Based on the fused feature vector and the training sample image data, the classifier model and the discriminator model in the visual large model are trained until the preset model training end condition is met, including: Based on the initial power image data and the fused feature vector, a high confidence region in the initial power image data is determined; wherein the high confidence region is the region in the initial power image data that is most prone to defects; Inputting the fused feature vector into a generator model for multi-layer feature extraction, and adding defect features consistent with the defect type text data in corresponding areas of each feature level according to the high confidence area, to generate simulated power defect image data; Inputting the simulated power defect image data and the training sample image data into the discriminator model, respectively extracting image features, and comparing the extracted image features at the same area position to determine the image similarity between the simulated power defect image data and the training sample image data; The discriminator model and the generator model are iteratively trained based on a first loss function and a second loss function, respectively, until a preset model training end condition is met; wherein the first loss function is used to characterize the image similarity between the simulated power defect image data and the training sample image data, and the second loss function is used to characterize the discriminator model's scoring of the simulated power defect image data.

3. The method according to claim 2, characterized in that The fused feature vector is input into the generator model for multi-layer feature extraction, and defect features consistent with the defect type text data are added to the corresponding area of ​​each feature level according to the high confidence area to generate simulated power defect image data, including: The fused feature vector is input into the generator model, and multi-layer feature extraction is performed in sequence to generate a preset number of feature levels; wherein the feature levels are cascaded layer by layer; For the defect type text data, generating corresponding primary defect features of different scales; wherein the primary defect features are used to describe the defect information corresponding to the defect type text data; According to the scale priority order of the primary defect features and the hierarchical order of the feature levels, the primary defect features are fused with the feature levels in turn to generate simulated power defect image data.

4. The method according to claim 2, characterized in that: The generator model and the discriminator model are updated in an alternating manner during the iterative training process; Accordingly, the generator model and the discriminator model are iteratively trained based on the first loss function and the second loss function, including: When the model parameters of the generator model are fixed, the gradient value of the first loss function is calculated to update the model parameters of the discriminator model; When the model parameters of the discriminator model are fixed, the gradient value of the second loss function is calculated to update the model parameters of the discriminator model.

5. The method according to claim 2, characterized in that: Determining a high confidence region in the initial power image data based on the initial power image data and the fused feature vector includes: Extracting high-level features of the initial power image data to generate an activation heat map; The activation heat map is annotated according to the defect position information in the fused feature vector, an injection region of the defect is determined in the activation heat map, and the injection region is determined as a high confidence region.

6. The method according to claim 1, characterized in that Based on the modality alignment mechanism, the image feature vector and the text feature vector are subjected to feature fusion to generate a fused feature vector, including: Performing a linear transformation on the image feature vector and the text feature vector to map the image feature vector and the text feature vector to the same feature space; The weight relationship between the image feature vector and the text feature vector in the same feature space is determined in parallel according to the multi-head attention mechanism, and bidirectional feature fusion is performed on the image feature vector and the text feature vector according to the weight relationship to generate a fused feature vector.

7. A device for establishing a power image sample generation model, characterized in that: include: A feature extraction module is used to obtain initial power image data and defect type text data, and perform image feature extraction and text feature extraction on the initial power image data and the defect type text data respectively based on a preset visual macro model to generate an image feature vector and a text feature vector; wherein the power image data is an equipment image of a power device without defects; and the defect type text data is used to characterize defect information of the power device; A feature fusion module, used for performing feature fusion on the image feature vector and the text feature vector based on a modality alignment mechanism to generate a fused feature vector; A training module, used to train the classifier model and the discriminator model in the visual large model based on the fused feature vector and the training sample image data until the preset model training end condition is met; wherein the classifier model is used to generate simulated power defect image data; the discriminator model is used to detect the similarity between the simulated power defect image data and the training sample image data; the simulated power defect image data and the training sample image data have the same power equipment defect; The model generation module is used to generate a model using the trained visual large model as a power image sample.

8. The device according to claim 7, characterized in that The training module includes: A region determination unit, configured to determine a high confidence region in the initial power image data based on the initial power image data and the fused feature vector; wherein the high confidence region is a region in the initial power image data that is most prone to defects; An image generation unit, used for inputting the fused feature vector into a generator model for multi-layer feature extraction, and adding defect features consistent with the defect type text data in corresponding areas of each feature level according to the high confidence area, to generate simulated power defect image data; An image verification unit, used to input the simulated power defect image data and the training sample image data into a discriminator model, respectively extract image features, and compare the extracted image features at the same area position to determine the image similarity between the simulated power defect image data and the training sample image data; An iterative unit is used to iteratively train the discriminator model and the generator model based on a first loss function and a second loss function, respectively, until a preset model training end condition is met; wherein the first loss function is used to characterize the image similarity between the simulated power defect image data and the training sample image data, and the second loss function is used to characterize the discriminator model's scoring of the simulated power defect image data.

9. The device according to claim 8, characterized in that The image generating unit comprises: A feature extraction subunit, used for inputting the fused feature vector into a generator model, and sequentially performing multi-layer feature extraction to generate a preset number of feature levels; wherein the feature levels are cascaded layer by layer with each other; A primary defect feature subunit is used to generate primary defect features of different scales corresponding to the defect type text data; wherein the primary defect features are used to describe the defect information corresponding to the defect type text data; The image generation subunit is used to fuse the primary defect features with the feature levels in sequence according to the scale priority order of the primary defect features and the hierarchical order of the feature levels to generate simulated power defect image data.

10. The device according to claim 8, characterized in that The generator model and the discriminator model are updated in an alternating manner during the iterative training process.

11. The device according to claim 8, characterized in that The iteration unit is specifically used for: When the model parameters of the generator model are fixed, the gradient value of the first loss function is calculated to update the model parameters of the discriminator model; When the model parameters of the discriminator model are fixed, the gradient value of the second loss function is calculated to update the model parameters of the discriminator model.

12. The device according to claim 8, characterized in that The area determination unit is specifically used for: Extracting high-level features of the initial power image data to generate an activation heat map; The activation heat map is annotated according to the defect position information in the fused feature vector, an injection region of the defect is determined in the activation heat map, and the injection region is determined as a high confidence region.

13. The device according to claim 7, characterized in that The feature fusion module is specifically used for: Performing a linear transformation on the image feature vector and the text feature vector to map the image feature vector and the text feature vector to the same feature space; The weight relationship between the image feature vector and the text feature vector in the same feature space is determined in parallel according to the multi-head attention mechanism, and bidirectional feature fusion is performed on the image feature vector and the text feature vector according to the weight relationship to generate a fused feature vector.

14. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for establishing a power image sample generation model according to any one of claims 1 to 6.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for establishing a power image sample generation model according to any one of claims 1 to 6 when executed.

16. A computer program product, characterized in that It comprises a computer program, which, when executed by a processor, implements the method for establishing a power image sample generation model as described in any one of claims 1 to 6.

Images