A power distribution room scene training data augmentation method based on multi-modal image editing

Abstract

The application relates to the technical field of image processing, and discloses a power distribution room scene training data augmentation method based on multi-modal image editing. The method comprises the following steps: acquiring a plurality of monitoring images in a power distribution room scene as power distribution room scene training data; wherein each monitoring image comprises a background image and a target region image to be edited and synthesized; the background image and the target region image of each monitoring image are respectively subjected to feature extraction to obtain visual modal features of the background image and the target region image; a text description corresponding to a visual target to be edited and synthesized is acquired, and the text description is subjected to feature extraction to obtain a text modal feature; the text modal feature and the visual modal features of the background image and the target region image are spliced in a spatial dimension according to two-dimensional rotation position coding to obtain a multi-modal feature; and an edited and synthesized image of each monitoring image is generated through the multi-modal feature to augment the power distribution room scene training data.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an augmentation method, system, device, and medium for training data of a power distribution room scene based on multimodal image editing. Background Technology

[0002] With the rapid growth of the economy, the scale of power equipment is also increasing, and people's demand for electricity is constantly evolving. However, due to the diverse types and varying performance of power equipment in power distribution rooms, and the significant differences in their operating conditions and requirements, a series of safety issues inevitably arise during long-term power supply, thus affecting the normal operation of the power grid system. Therefore, using various monitoring devices to analyze the operating status of various power equipment in power distribution rooms and taking timely protective measures is of great significance for improving the operational stability of the power system and promoting its continued development.

[0003] Against this backdrop, promoting the transformation of the power grid towards "digital intelligence" and "green and low-carbon" development, developing "new-quality productivity," and accelerating the construction of a cloud-edge integrated collaborative system have become an industry consensus. A key step in achieving this goal is shifting from "passive emergency repair" to "proactive maintenance," which places higher demands on intelligent monitoring methods. Most existing high-performance real-time monitoring algorithms rely on high-quality labeled data to train high-performance deep neural networks to meet the actual monitoring needs of this scenario. However, at power plants such as photovoltaic stations and energy storage stations, due to the lack of data on abnormal operating conditions, most acquired images do not show abnormal operating states. This leads to models obtained with limited training data failing to meet performance requirements, and visual algorithms struggling to improve performance through online learning. These issues have become the core bottleneck restricting the rapid optimization and adaptation of algorithm models.

[0004] Image editing aims to modify the overall visual distribution information of a region in an image to meet user requirements. Existing methods attempt to artificially synthesize images with abnormal distributions by simply copying and pasting a specific object onto a target image, resulting in an unnatural overall appearance and impacting the model's monitoring performance in real-world scenarios. Although some methods have attempted to use generative deep neural networks to better synthesize edited images, these methods all suffer from varying degrees of image realism issues, limiting the performance of subsequent monitoring methods. Summary of the Invention

[0005] The purpose of this invention is to provide an augmentation method, system, device, and medium for training data of a power distribution room scene based on multimodal image editing, which can solve the problem of insufficient image realism generated by existing image editing methods.

[0006] To address the aforementioned technical problems, embodiments of the present invention provide an augmentation method for training data of a power distribution room scene based on multimodal image editing, comprising the following steps:

[0007] Acquire several monitoring images in a power distribution room scenario as training data for the power distribution room scenario; each monitoring image includes a background image and a target area image to be edited and synthesized.

[0008] For each monitoring image, feature extraction is performed on the background image and the target area image of the monitoring image to obtain the visual modal features of the background image and the target area image;

[0009] Obtain the text description corresponding to the visual target to be edited and synthesized, and extract features from the text description to obtain text modality features;

[0010] Based on the preset two-dimensional rotational position encoding for the Transformer structure, the text modal features and the visual modal features of the background image and the target region image are spliced ​​in the spatial dimension to obtain multimodal features. Among them, the two-dimensional rotational position encoding takes the upper left corner of the preset image divided into multiple regions of the same size as the center of the coordinate system, the horizontal axis of the coordinate system is to the right, and the vertical axis is downward. The text modal features are assigned to the region in the preset image where the starting position of the coordinate system is located, and the visual modal features of the background image and the target region image are assigned to the region in the preset image where the diagonal position of the coordinate system is located.

[0011] Multimodal features are input into an image generation network based on a Transformer structure to generate edited composite images of each monitoring image, thereby augmenting the training data for the power distribution room scene.

[0012] Optionally, the step of extracting features from the background image and the target area image of the surveillance image respectively to obtain the visual modal features of the background image and the target area image includes:

[0013] The background image and target region image of the monitoring image are respectively input into the trained visual modality feature extraction network. The visual modality feature extraction network compresses the background image and target region image into the latent space through the encoder network of the variational autoencoder, and obtains the visual representation in the latent space of each, which are used as the visual modality features of the background image and the target region image respectively.

[0014] Optionally, the step of extracting features from the text description to obtain text modal features includes:

[0015] The text description is input into the trained text modality feature extraction network. The text description is lexicalized by the text modality feature extraction network to obtain the edited text lexical sequence. The edited text lexical sequence is then passed through 12 Transformer blocks to obtain the text modality features. Each Transformer block contains a self-attention module, a feedforward neural network module, and a layer normalization module.

[0016] Optionally, before splicing the text modal features and the visual modal features of the background image and the target region image in the spatial dimension, the method further includes:

[0017] By using Reshape and dimension swapping operations, the spatial dimensions of the visual modal features of the background image and the target region image are changed to match the input format of the Transformer architecture.

[0018] Optionally, the two-dimensional rotational position is encoded as follows:

[0019] ;

[0020] In the formula, and These represent the horizontal and vertical directions of the image centered at the top left corner of the coordinate system. When, it means that the current feature corresponds to the region in the 0th row and the first column after the image is divided into regions of the same size.

[0021] Optionally, after generating the edited composite image of each monitoring image using multimodal features, the method further includes:

[0022] The edited and synthesized image is input into a trained open-set detection network to obtain the category and coordinates of the target in the edited and synthesized image;

[0023] If the category of the detected target does not match the category of the actual target in the visual target, the composite image is discarded and edited.

[0024] Otherwise, obtain the difference between the coordinates of the detected target and the coordinate center point of the actual target in the visual target, and normalize it to obtain the coordinate error score;

[0025] The edited synthetic image is input into the trained visual language network to obtain the cosine similarity between the text description of the edited synthetic image and the text description corresponding to the visual target, and the semantic error score is determined based on the cosine similarity.

[0026] Based on the coordinate error score and semantic error score, determine whether to discard the edited composite image.

[0027] Optionally, both the visual modality feature extraction network and the text modality feature extraction network are trained in the following way:

[0028] The visual modal features and text modal features extracted by the visual modal feature extraction network and the text modal feature extraction network, respectively, are obtained. The visual modal features and text modal features are then projected into the latent space of the image generation network through two linear layers to obtain their respective latent space features.

[0029] Based on their respective latent space features, the network parameters of the visual modality extraction network and the text modality extraction network are updated using a low-rank matrix fine-tuning method, in order to train the visual modality extraction network and the text modality extraction network.

[0030] Embodiments of the present invention also provide an augmentation system for training data of a power distribution room scene based on multimodal image editing, comprising:

[0031] The image acquisition module is used to acquire several monitoring images in the power distribution room scenario as training data for the power distribution room scenario; each monitoring image includes a background image and a target area image to be edited and synthesized.

[0032] The image feature extraction module is used to extract features from the background image and the target area image of each monitoring image to obtain the visual modal features of the background image and the target area image.

[0033] The text feature extraction module is used to obtain the text description corresponding to the visual target to be edited and synthesized, and to extract features from the text description to obtain text modal features;

[0034] The feature encoding module is used to splice the text modal features and the visual modal features of the background image and the target region image in the spatial dimension according to the preset two-dimensional rotational position encoding for the Transformer structure to obtain multimodal features. The two-dimensional rotational position encoding takes the upper left corner of the preset image, which is divided into multiple regions of the same size, as the center of the coordinate system. The horizontal axis of the coordinate system is to the right and the vertical axis is downward. The text modal features are assigned to the region in the preset image where the starting position of the coordinate system is located. The visual modal features of the background image and the target region image are assigned to the region in the preset image where the diagonal position of the coordinate system is located.

[0035] The image augmentation module is used to input multimodal features into an image generation network based on a Transformer structure to generate edited composite images of each monitoring image, thereby augmenting the training data of the power distribution room scene.

[0036] Embodiments of the present invention also provide a computer device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described method for augmenting training data for a power distribution room scenario.

[0037] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for augmenting training data for a power distribution room scenario.

[0038] The augmentation method for training data of a power distribution room scene based on multimodal image editing provided by this invention has at least the following beneficial effects:

[0039] For the training data of the power distribution room scenario, firstly, the visual modal features of the background image and the target area image in each monitoring image are extracted. The target area is the area in the monitoring image that needs to be edited and synthesized. Then, for the visual target edited and synthesized in the target area of ​​the monitoring image, the text modal features of its text description are extracted. Then, the visual modal features and text modal features are spatially concatenated and feature embedded by two-dimensional rotational position encoding (so that the Transformer structure can recognize the spatial position information of the features) to obtain multimodal features. Using these multimodal features, the image generation network based on the Transformer structure can obtain the complex spatial semantic features of the image that needs to be edited and synthesized. Based on this, more realistic edited and synthesized images can be generated.

[0040] In other words, the present invention can edit and synthesize each monitoring image using the above method, thereby augmenting the training data of the power distribution room scene. The augmented training data obtained by this augmentation scheme is more realistic, better reflects the real power distribution room scene, and is more conducive to the implementation of power distribution room safety monitoring. Attached Figure Description

[0041] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0042] Figure 1 A flowchart illustrating a method for augmenting training data of a power distribution room scene based on multimodal image editing, provided by the present invention;

[0043] Figure 2 A schematic diagram illustrating the principle of an augmentation method for training data of a power distribution room scene based on multimodal image editing provided by the present invention;

[0044] Figure 3 This invention provides a schematic diagram of an operation for encoding multimodal features using two-dimensional rotational position encoding;

[0045] Figure 4 This is a comparative schematic diagram of an edited and synthesized image provided by the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0047] The technical solutions provided by the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0048] One embodiment of the present invention relates to an augmentation method for training data of a power distribution room scene based on multimodal image editing. The specific process of the augmentation method for training data of a power distribution room scene based on multimodal image editing in this embodiment can be as follows: Figure 1 As shown, it includes:

[0049] Step 101: Obtain several monitoring images of the power distribution room scenario as training data for the power distribution room scenario; wherein, each monitoring image includes a background image and a target area image to be edited and synthesized.

[0050] In the specific implementation, the monitoring image containing the target to be edited and synthesized (assuming a size of...) will be used. The SAM network is used to segment and obtain an image of the region containing the target (size: ), namely the target area image and the background image (size is ).

[0051] Step 102: For each monitoring image, feature extraction is performed on the background image and the target area image of the monitoring image to obtain the visual modal features of the background image and the target area image.

[0052] Specifically, the background image and target region image of the surveillance image are input into a pre-trained visual modality feature extraction network. The network uses a variational autoencoder to compress the background image and target region image into their respective latent spaces, obtaining their visual representations within those latent spaces. These representations serve as the visual modality features for the background image and target region image, respectively, with feature sizes of [missing data]. as well as .

[0053] Step 103: Obtain the text description corresponding to the visual target to be edited and synthesized, and extract features from the text description to obtain text modality features.

[0054] Specifically, the visual target requiring image editing and synthesis is transformed into a corresponding text description, which is then input into a trained text modality feature extraction network. The text modality feature extraction network performs lexicalization on the text description, resulting in an edited text lexical sequence (size: [size missing]). The edited text word sequence is processed through 12 Transformer blocks to obtain the corresponding text modality features (size: ). Each Transformer block contains a self-attention module, a feedforward neural network module, and a layer normalization module.

[0055] Step 104: Based on the preset two-dimensional rotational position encoding for the Transformer structure, the text modal features and the visual modal features of the background image and the target region image are spliced ​​in the spatial dimension to obtain multimodal features; wherein, the two-dimensional rotational position encoding takes the upper left corner of a preset image divided into multiple regions of the same size as the center of the coordinate system, the horizontal axis of the coordinate system is to the right, and the vertical axis is downward. The text modal features are assigned to the region in the preset image where the starting position of the coordinate system is located, and the visual modal features of the background image and the target region image are assigned to the region in the preset image where the diagonal position of the coordinate system is located.

[0056] In the specific implementation, the two-dimensional rotational position is encoded as follows:

[0057] ;

[0058] In the formula, and These represent the horizontal and vertical directions of the image centered at the top left corner of the coordinate system. When, it means that the current feature corresponds to the region in the 0th row and the first column after the image is divided into regions of the same size.

[0059] For text modal features, two-dimensional rotational position encoding always assigns a starting point. Position; for visual modal features, two-dimensional rotational position coding assigns position codes to visual representations in the latent space (including the background image of the power distribution room and the representation of the target region image that needs to be edited and synthesized) in diagonal order. This diagonal allocation strategy can also naturally adapt to multiple sets of target region features that need to be edited and synthesized in different positions, thereby enabling multiple target regions to be synthesized on the same background image at the same time.

[0060] By splicing the text modal features, the substation background image, and the altered dimensional latent space visual representation of the target region image to be edited and synthesized in space, and embedding the features using a preset two-dimensional rotational position encoding, as shown in the following equation:

[0061]

[0062] In the formula, These represent the text modal features, the background image of the power distribution room after changing the spatial dimension, and the latent spatial visual representation of the target region image to be edited and synthesized, respectively. This represents a multimodal unified position encoding operation (implemented through two-dimensional rotational position encoding). This represents the multimodal input features after position encoding and concatenation, with a feature size of [value missing]. .

[0063] Step 105: Input the multimodal features into the image generation network based on the Transformer structure to generate edited and synthesized images of each monitoring image to augment the training data of the power distribution room scene.

[0064] Specifically, the aforementioned multimodal input features are input into a pre-trained image generation network to obtain an edited image, the feature size of which is... The pre-trained image generation network can use the Diffusion Transformer (DiT) architecture to achieve image generation. During image generation, the aforementioned text modality features, the dimensionality-modified power distribution room background image, and the latent space visual representation of the target region image to be edited and synthesized are simultaneously injected into the pre-trained Diffusion Transformer (DiT)-based image generation network through a cross-attention method. Taking the description of text modality features as an example, the cross-attention operation is as follows:

[0065]

[0066] In the formula, These represent the parameter matrices of the linear layers used for the corresponding features. These represent the text modal features and the intermediate features of the image generation network, respectively. The dimension representing the intermediate features of the image generation network. This is a commonly used operation in neural networks, used to normalize a numerical vector into a probability distribution vector. This indicates a cross-attention operation.

[0067] Deep neural networks have made significant progress in the field of image generation. This invention can employ various image generation networks as the base network for a multimodal image editing and synthesis module, including VQGAN, Stable Diffusion, and FLUX.1. Extensive experiments have demonstrated that FLUX.1, based on the Diffusion Transformer architecture, achieves the best image generation results. FLUX.1 is an image generation model developed by Black Forest Labs in 2024. Its main idea lies in using flow matching to optimize the efficiency of the Diffusion Transformer-based diffusion model in terms of generation.

[0068] In one example, the visual representation in the latent space obtained in step 102 (i.e., visual modality features) has its spatial dimensions modified through Reshape and dimension swapping operations to match the input format of the Diffusion Transformer architecture used by the image generation network in step 104. The modified spatial dimensions are as follows: as well as .

[0069] In some embodiments, the present invention performs quality verification on the generated edited composite image. Specifically, the edited composite image is input into a trained open-set detection network to obtain the category and coordinates of the target in the edited composite image. If the category of the detected target does not match the category of the actual target in the visual target, the edited composite image is discarded. Otherwise, the difference between the coordinates of the detected target and the center point of the actual target in the visual target is obtained and normalized to obtain a coordinate error score. The edited composite image is input into a trained visual language network to obtain the cosine similarity between the text description of the edited composite image and the text description corresponding to the visual target, and the semantic error score is determined based on the cosine similarity. Based on the coordinate error score and the semantic error score, it is determined whether to discard the edited composite image.

[0070] The detection error fraction is as follows:

[0071]

[0072] In the formula, This represents the final detection error score. as well as These represent the square roots of the center point coordinates of the predicted target coordinates and the labeled true coordinates, respectively. and These represent the pixel height and width of the composite image, respectively.

[0073] The semantic error score is shown in the following formula:

[0074]

[0075] In the formula, This represents the final semantic error score. Represents cosine similarity. and These represent the edited and synthesized image features and descriptive text features extracted by the visual language network, respectively.

[0076] In a specific example, if the detection error score is greater than 0.1, the semantic error score is greater than 0.3, and the sum of the two is greater than 0.4, then the current synthesized image is discarded; otherwise, it is not discarded.

[0077] In some embodiments, the present invention provides an efficient parameter fine-tuning method for the supervised training process of partial network layers in a deep neural network:

[0078] Based on the text modality features extracted from the text modality feature extraction network, two linear layers are designed to project their features into the latent space of a pre-trained image generation network, and a low-rank matrix fine-tuning method is used to update only the parameters of this part of the network layer; and based on the visual representation in the latent space of the visual modality feature extraction network, two linear layers are designed to project their features into the latent space of a pre-trained image generation network, and a low-rank matrix fine-tuning method is used to update only the parameters of this part of the network layer.

[0079] This embodiment employs efficient parameter fine-tuning methods such as low-rank matrices to update the parameters of the linear projection layers in the text modality feature extraction network and the visual modality feature extraction network, which are used to project the output modality-related features onto the pre-trained image generation network, thereby improving the ability to edit and synthesize images in the power distribution room scenario.

[0080] Deep neural networks have made significant progress in text information extraction. This embodiment can use various deep neural networks as the base network for text modality feature extraction, including CLIP Text Encoder, T5 Encoder, and BERT. After multiple experiments, T5 Encoder demonstrates that removing the bias parameters of the LayerNorm layer and employing a self-supervised pre-training method can achieve better text feature extraction performance.

[0081] Furthermore, this embodiment can employ various low-rank matrix fine-tuning methods as training strategies for updating projection layer parameters, including LoRA, DoRA, and LoRA+. Extensive experiments have demonstrated that LoRA+, as an efficient parameter fine-tuning method, achieves the best training results. LoRA+ is an efficient parameter fine-tuning method whose main idea lies in updating the parameters of the two matrices after low-rank decomposition with different learning rates, decomposing the parameter update paths of the two matrices, thereby achieving more stable training results.

[0082] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the protection scope of this invention. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, without changing the core design of the algorithm and process, are also within the protection scope of this invention.

[0083] Another embodiment of the present invention relates to an augmentation system for training data of a power distribution room scene based on multimodal image editing. The implementation details of this embodiment's augmentation system for training data of a power distribution room scene based on multimodal image editing are described below. The following details are provided for ease of understanding and are not essential for implementing this solution. This embodiment's augmentation system for training data of a power distribution room scene based on multimodal image editing includes:

[0084] The image acquisition module is used to acquire several monitoring images in the power distribution room scenario as training data for the power distribution room scenario; each monitoring image includes a background image and a target area image to be edited and synthesized.

[0085] The image feature extraction module is used to extract features from the background image and the target area image of each monitoring image to obtain the visual modal features of the background image and the target area image.

[0086] The text feature extraction module is used to obtain the text description corresponding to the visual target to be edited and synthesized, and to extract features from the text description to obtain text modal features;

[0087] The feature encoding module is used to splice the text modal features and the visual modal features of the background image and the target region image in the spatial dimension according to the preset two-dimensional rotational position encoding for the Transformer structure to obtain multimodal features. The two-dimensional rotational position encoding takes the upper left corner of the preset image, which is divided into multiple regions of the same size, as the center of the coordinate system. The horizontal axis of the coordinate system is to the right and the vertical axis is downward. The text modal features are assigned to the region in the preset image where the starting position of the coordinate system is located. The visual modal features of the background image and the target region image are assigned to the region in the preset image where the diagonal position of the coordinate system is located.

[0088] The image augmentation module is used to input multimodal features into an image generation network based on a Transformer structure to generate edited composite images of each monitoring image, thereby augmenting the training data of the power distribution room scene.

[0089] It is not difficult to see that this embodiment is a system embodiment corresponding to the above method embodiments, and this embodiment can be implemented in conjunction with the above method embodiments. The relevant technical details and technical effects mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0090] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this invention, this embodiment does not introduce units that are not closely related to solving the technical problem proposed by this invention; however, this does not mean that other units are absent from this embodiment.

[0091] Another embodiment of the present invention relates to a computer device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the augmentation method for training data of a power distribution room scene based on multimodal image editing in the above embodiments.

[0092] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0093] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0094] Another embodiment of the present invention relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the method embodiments described above.

[0095] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0096] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing the present invention, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of the present invention.

Claims

1. A method for augmenting training data of a power distribution room scene based on multimodal image editing, characterized in that, The method includes: Acquire several monitoring images in a power distribution room scenario as training data for the power distribution room scenario; each monitoring image includes a background image and a target area image to be edited and synthesized. For each monitoring image, feature extraction is performed on the background image and the target area image of the monitoring image to obtain the visual modal features of the background image and the target area image; Obtain the text description corresponding to the visual target to be edited and synthesized, and extract features from the text description to obtain text modality features; Based on the preset two-dimensional rotational position encoding for the Transformer structure, the text modal features and the visual modal features of the background image and the target region image are spliced ​​in the spatial dimension to obtain multimodal features. Among them, the two-dimensional rotational position encoding takes the upper left corner of the preset image divided into multiple regions of the same size as the center of the coordinate system, the horizontal axis of the coordinate system is to the right, and the vertical axis is downward. The text modal features are assigned to the region in the preset image where the starting position of the coordinate system is located, and the visual modal features of the background image and the target region image are assigned to the region in the preset image where the diagonal position of the coordinate system is located. Multimodal features are input into an image generation network based on a Transformer structure to generate edited composite images of each monitoring image, thereby augmenting the training data for the power distribution room scene.

2. The method for augmenting training data of a power distribution room scene based on multimodal image editing according to claim 1, characterized in that, The step of extracting features from the background image and the target area image of the monitoring image respectively to obtain the visual modal features of the background image and the target area image includes: The background image and target region image of the monitoring image are respectively input into the trained visual modality feature extraction network. The visual modality feature extraction network compresses the background image and target region image into the latent space through the encoder network of the variational autoencoder, and obtains the visual representation in the latent space of each, which are used as the visual modality features of the background image and the target region image respectively.

3. The method for augmenting training data of a power distribution room scene based on multimodal image editing according to claim 2, characterized in that, The step of extracting features from the text description to obtain text modal features includes: The text description is input into the trained text modality feature extraction network. The text description is lexicalized by the text modality feature extraction network to obtain the edited text lexical sequence. The edited text lexical sequence is then passed through 12 Transformer blocks to obtain the text modality features. Each Transformer block contains a self-attention module, a feedforward neural network module, and a layer normalization module.

4. The method for augmenting training data of a power distribution room scene based on multimodal image editing according to claim 1, characterized in that, Before the spatial concatenation of the text modal features and the visual modal features of the background image and the target region image, the following steps are included: By using Reshape and dimension swapping operations, the spatial dimensions of the visual modal features of the background image and the target region image are changed to match the input format of the Transformer architecture.

5. The method for augmenting training data of a power distribution room scene based on multimodal image editing according to claim 1, characterized in that, The two-dimensional rotational position code is: ； In the formula, and These represent the horizontal and vertical directions of the image centered at the top left corner of the coordinate system. When, it means that the current feature corresponds to the region in the 0th row and the first column after the image is divided into regions of the same size.

6. The method for augmenting training data of a power distribution room scene based on multimodal image editing according to claim 1, characterized in that, Following the generation of the edited composite image for each surveillance image, the process also includes: The edited and synthesized image is input into a trained open-set detection network to obtain the category and coordinates of the target in the edited and synthesized image; If the category of the detected target does not match the category of the actual target in the visual target, the composite image is discarded and edited. Otherwise, obtain the difference between the coordinates of the detected target and the coordinate center point of the actual target in the visual target, and normalize it to obtain the coordinate error score; The edited synthetic image is input into the trained visual language network to obtain the cosine similarity between the text description of the edited synthetic image and the text description corresponding to the visual target, and the semantic error score is determined based on the cosine similarity. Based on the coordinate error score and semantic error score, determine whether to discard the edited composite image.

7. The method for augmenting training data of a power distribution room scene based on multimodal image editing according to claim 3, characterized in that, Both the visual modality feature extraction network and the text modality feature extraction network were trained in the following way: The visual modal features and text modal features extracted by the visual modal feature extraction network and the text modal feature extraction network, respectively, are obtained. The visual modal features and text modal features are then projected into the latent space of the image generation network through two linear layers to obtain their respective latent space features. Based on their respective latent space features, the network parameters of the visual modality extraction network and the text modality extraction network are updated using a low-rank matrix fine-tuning method, in order to train the visual modality extraction network and the text modality extraction network.

8. An augmentation system for training data of a power distribution room scene based on multimodal image editing, characterized in that, The system includes: The image acquisition module is used to acquire several monitoring images in the power distribution room scenario as training data for the power distribution room scenario; each monitoring image includes a background image and a target area image to be edited and synthesized. The image feature extraction module is used to extract features from the background image and the target area image of each monitoring image to obtain the visual modal features of the background image and the target area image. The text feature extraction module is used to obtain the text description corresponding to the visual target to be edited and synthesized, and to extract features from the text description to obtain text modal features; The feature encoding module is used to splice the text modal features and the visual modal features of the background image and the target region image in the spatial dimension according to the preset two-dimensional rotational position encoding for the Transformer structure to obtain multimodal features. The two-dimensional rotational position encoding takes the upper left corner of the preset image, which is divided into multiple regions of the same size, as the center of the coordinate system. The horizontal axis of the coordinate system is to the right and the vertical axis is downward. The text modal features are assigned to the region in the preset image where the starting position of the coordinate system is located. The visual modal features of the background image and the target region image are assigned to the region in the preset image where the diagonal position of the coordinate system is located. The image augmentation module is used to input multimodal features into an image generation network based on a Transformer structure to generate edited composite images of each monitoring image, thereby augmenting the training data of the power distribution room scene.

9. A computer device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the augmentation method for training data of a power distribution room scene based on multimodal image editing as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the augmentation method for training data of a power distribution room scene based on multimodal image editing as described in any one of claims 1 to 7.

Images