A method and device for generating power transmission inspection images based on multimodal data

Abstract

The present invention discloses a method and device for generating power transmission inspection images based on multimodal data. The method comprises: inputting an original image into a first network model for target detection, obtaining text description data of at least one target power transmission component in the original image; obtaining a small number of matching reference images from a historical image acquisition database based on the first text information; determining a target generation area for the target power transmission component in a newly generated image based on the coordinate position information of the target power transmission component in the original image; inputting the original image, a small number of reference images, and the text description data into an image generation network, and generating an image of the target power transmission component in the target generation area of the newly generated image. The present invention generates the target power transmission component at the corresponding position in the newly generated image based only on the position of the target power transmission component in the original image, without requiring the generation of a new image based on the entire original image, thereby improving the efficiency of power transmission component image generation.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and device for generating power transmission inspection images based on multimodal data. Background Art

[0002] With the in-depth application of artificial intelligence technology in the power grid sector, defect detection for transmission lines is gradually shifting from manual detection to artificial intelligence identification. Defect recognition models often require large amounts of data for training, but the data volume of some component images simply does not meet the training model requirements. Current image generation methods primarily rely on fusion-based, transformation-based, and optimization-based methods to achieve small sample generation. These methods can introduce numerous artifacts during the image generation process or lose the diversity of the generated images. Transmission line images typically capture large scenes, while the target components often occupy a very small portion of the image. Generating the entire image would waste significant time and resources. Summary of the Invention

[0003] In response to the above-mentioned problems in the prior art, the present invention provides a method and device for generating power transmission inspection images based on multimodal data, which improves the image generation efficiency and the quality of the generated images.

[0004] The present invention provides a method for generating a power transmission inspection image based on multimodal data, the method comprising:

[0005] Performing target detection based on an original image input into a first network model, obtaining text description data of at least one target power transmission component in the original image, wherein the text description data includes coordinate position information of the target power transmission component in the original image and first text information, wherein the first text information represents associated attribute information of the target power transmission component in the original image;

[0006] Acquiring a small number of matching reference images from a historical image acquisition database based on the first text information;

[0007] determining a target generation area of the target power transmission component in the newly generated image based on the coordinate position information of the target power transmission component in the original image;

[0008] Based on the original image, a small amount of the reference images and text description data input into the image generation network, the target power transmission component image is generated in the target generation area of the newly generated image.

[0009] In some embodiments, the step of inputting an image generation network based on an original image, a small amount of reference images, and text description data, and generating the target power transmission component image in the target generation area of the newly generated image includes:

[0010] Obtain pixel features of the global image and local pixel features of the target coordinate area based on the original image and a small number of reference images;

[0011] generating corresponding text features based on the first text information;

[0012] The text features and pixel features are fused and input into the generator model to generate simulated images;

[0013] Authenticity discrimination is performed based on the simulated image input to the discriminator model;

[0014] After the discriminator model determines that the simulated image is a real image, the simulated image is output as a newly generated image.

[0015] In some implementations, the fusing based on text features and pixel features includes:

[0016] Measuring the matching of the text features and the image features based on the similarity between the text features and the image features;

[0017] Assign weights to text features based on the degree of matching. When the matching between text features and image features is high, a high weight is assigned to the text feature; when the matching between text features and image features is low, a low weight is assigned to the text feature.

[0018] Extract high-level image features based on text features, text feature weights and image features.

[0019] In some implementations, extracting high-level image features based on text features, text feature weights, and image features further includes:

[0020] (41) Analyze the image features obtained in the previous step and update the weight of the text features at the current moment;

[0021] (42) Based on the text features and the updated text feature weights, the image features obtained in the previous step are updated;

[0022] (43) Repeat steps (41)-(42) to update the text feature weights and image features.

[0023] In some embodiments, acquiring pixel features of a global image and local pixel features of a target coordinate region based on an original image and a small number of reference images includes:

[0024] Inputting the original image and a small number of reference images into a first image processing module and a second image processing module respectively;

[0025] Extracting pixel features of a global image from an original image and a small number of reference images based on a first image processing module;

[0026] Based on the coordinate position information of the target power transmission component in the original image output by the first network model, the second image processing module extracts image features of a local area where the target power transmission component is located in the original image to obtain local pixel features;

[0027] Pixel features are obtained by fusing global image features and local pixel features.

[0028] In some embodiments, the fusing based on global image features and local pixel features includes:

[0029] Extracting attention image features of the local pixel feature map based on a preset attention model;

[0030] Obtaining a target power transmission component feature map based on multiplying the attention image feature and the local pixel feature map;

[0031] Multiplying the attention image features based on the local pixel feature map and the global pixel feature map to obtain a first intermediate image;

[0032] Based on the addition of the first intermediate image and the global pixel feature map, a fused global image feature is obtained.

[0033] In some embodiments, the discriminator model of the image generation network, when performing image authenticity discrimination, includes:

[0034] Encoding the simulated image generated by the generator model, the encoding comprising: converting the encoded data distribution into normal distribution data based on mean and variance characteristics of the encoded data;

[0035] A comparison analysis is performed based on the coded data, the coordinate position information, the first text information and the original image to obtain a result of authenticity judgment of the simulated image.

[0036] In some embodiments, the converting of the coded data distribution into normally distributed data based on the mean and variance characteristics of the coded data includes: transforming the coded data into a Gaussian distribution based on the mean and variance of the coded data, and performing a normalization transformation after the Gaussian distribution transformation.

[0037] The present invention also provides a device for generating a power transmission inspection image based on multimodal data, comprising:

[0038] a target detection unit, configured to perform target detection based on an original image input into a first network model, and obtain text description data of at least one target power transmission component in the original image, wherein the text description data includes coordinate position information of the target power transmission component in the original image and first text information, wherein the first text information represents associated attribute information of the target power transmission component in the original image;

[0039] A reference image acquisition unit, configured to acquire a small number of matching reference images from a historical image acquisition database based on the first text information;

[0040] An image generation position acquisition unit, configured to determine a target generation area of the target power transmission component in the newly generated image based on the coordinate position information of the target power transmission component in the original image;

[0041] A new image generating unit is configured to generate the target power transmission component image in the target generation area of the newly generated image based on an original image, a small amount of the reference images and text description data input into an image generation network.

[0042] The present invention also provides a power transmission inspection image generation device, the device comprising:

[0043] processor;

[0044] a memory for storing processor-executable instructions;

[0045] The processor implements the above-mentioned method for generating power transmission inspection images based on multimodal data by running the executable instructions.

[0046] The present invention provides a method and device for generating power transmission inspection images based on multimodal data, achieving the following beneficial effects: When generating images from a captured original image, a local generation method is employed. The target power transmission component is generated at the corresponding position in the newly generated image based solely on its position in the original image, without requiring the entire original image to be generated. This improves the efficiency of generating images of the power transmission component. Furthermore, the present invention uses the original image as a basis, combining it with a small number of reference images and first text information, which are input into an image generation network. This increases the diversity and high value of the features extracted by the image generation network, correspondingly improving the diversity and quality of the newly generated images. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 1 is a flow chart of a method for generating a power transmission inspection image based on multimodal data in an embodiment of the present application;

[0048] Figure 2 This is a flow chart of an implementation method of a method for generating a power transmission inspection image in an embodiment of the present application;

[0049] Figure 3 Schematic diagram of the process of multimodal data fusion of text features and pixel features in an embodiment of the present application;

[0050] Figure 4 This is a flow chart of a method for acquiring image pixel features based on the fusion of local features and global features in an embodiment of the present application;

[0051] Figure 5 It is a structural diagram of a power transmission inspection image generation device based on multimodal data in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to enable those skilled in the art to better understand the present invention, the following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0053] See also Figure 1 , an embodiment of the present application provides a method for generating a power transmission inspection image based on multimodal data, the method comprising:

[0054] (1) performing target detection based on an original image input into a first network model, and obtaining text description data of at least one target power transmission component in the original image, wherein the text description data includes coordinate position information of the target power transmission component in the original image and first text information, wherein the first text information represents associated attribute information of the target power transmission component in the original image;

[0055] (2) obtaining a small number of matching reference images from a historical image acquisition database based on the first text information;

[0056] (3) Determining a target generation area of the target power transmission component in the newly generated image based on the coordinate position information of the target power transmission component in the original image;

[0057] (4) Based on the original image, a small amount of the reference image and text description data input into the image generation network, the target power transmission component image is generated in the target generation area of the newly generated image.

[0058] Taking into account the fact that in the field of power transmission scenarios, power transmission line equipment mainly consists of small hardware, the target power transmission components in the captured real image are too small in the captured image, accounting for less than 5% of the original image. In the embodiment of the present application, when generating the captured original image, a local generation method is adopted. The target power transmission components are generated at the corresponding positions in the newly generated image based only on the positions of the target power transmission components in the original image, without the need to generate a new image based on the entire original image, thereby improving the efficiency of generating images of power transmission components.

[0059] In addition, in the present application, the original image is used as the basis, combined with a small number of reference images and the first text information, and inputted into the image generation network together, thereby improving the diversity and high value of the features extracted by the image generation network, and correspondingly improving the diversity of types and image quality of the newly generated images. At the same time, the use of multimodal data for image generation reduces the number of samples required for image generation network training, and only the original image and a small number of reference images are required to obtain the generated image.

[0060] The coordinate position information of the target power transmission component in the original image includes the coordinate data of the target power transmission component in the original image and the relative positional structural relationship between the target power transmission component and multiple other target power transmission components in the original image. The first text information includes information such as the type, color, and angle of the target power transmission component.

[0061] Specifically, a Transformer network can be used to detect targets in the original image. Based on this network, all target power transmission components in the original image can be detected, and then the text feature information of all target power transmission components in the original image can be obtained, including type, color, coordinates, angle, whether there are defects, and other information.

[0062] For further information, see Figure 2 In the above step (4), based on the original image, a small amount of the reference image and text description data input into the image generation network, the target power transmission component image is generated in the target generation area of the newly generated image, including:

[0063] (401) obtaining pixel features of the global image and local pixel features of the target coordinate area based on the original image and a small number of reference images;

[0064] (402) generating corresponding text features based on the first text information;

[0065] (403) Fusion-based input of text features and pixel features into the generator model generates a simulated image;

[0066] (404) performing authenticity discrimination based on the simulated image input to the discriminator model;

[0067] (405) After the discriminator model determines that the simulated image is a real image, the simulated image is output as a newly generated image.

[0068] In an embodiment of the present application, during the entire image generation process, an image is generated only for the region of the target power transmission component in the original image. In one embodiment, the image generation process can be directly based on the local area image features of the region where the target power transmission component is located. However, considering the matching of the positional structure of the target power transmission component and other target power transmission components generated in the generated image with that in the original image, in order to make the positional structure of the target power transmission component and other target power transmission components generated in the generated image closer to the original image, in an embodiment of the present application, features of the entire original image are also used in the image generation process.

[0069] The image generation network extracts features from the original image, a small amount of the reference image and the first text information respectively, and then performs feature fusion. A simulated image is generated through the generator model, and the authenticity of the simulated image is judged based on the discriminator model. After being judged as a real image by the discriminator model, the simulated image is output as a newly generated image.

[0070] Correspondingly, when the generator model generates simulated images, it uses the image features of the original image as a foundation, combines the image features of a small number of reference images, and also incorporates text features to enhance the feature information used to generate the newly generated image. This rich and diverse feature information improves image generation efficiency and the diversity of generated images.

[0071] For further information, see Figure 3 In the above step (403), the fusion based on text features and pixel features includes:

[0072] (4031) measuring the matching of the text feature and the image feature based on the similarity between the text feature and the image feature;

[0073] (4032) assigning weights to text features based on the degree of matching, assigning high weights to text features when the matching between text features and image features is high, and assigning low weights to text features when the matching between text features and image features is low;

[0074] (4033) Extracting high-level image features based on text features, text feature weights, and image features.

[0075] In an embodiment of the present application, it is necessary to fuse the features of multiple modal data to achieve the generation of new images. Due to the differences between text features and image features in the expression of information and the representation of features, it is necessary to manage the synergy between text features and image features. When the matching between text features and image features is high, a high weight is assigned to the text feature to improve the contribution of the text feature to the generation of new images. When the matching between text features and image features is low, a low weight is assigned to the text feature to reduce the impact of the mismatch between text features and image features on the quality of the generated new images.

[0076] In this application, high-level image features are extracted based on text features, text feature weights and image features, and the high-level image features are input into an image generation network to obtain a generated image.

[0077] Specifically, the above-mentioned measurement of the matching of text features and image features can be used to calculate the size of the matching based on similarity.

[0078] See also Figure 3 In one embodiment, in the above step (4033), extracting high-level image features based on text features, text feature weights, and image features further includes:

[0079] (41) Analyze the image features obtained in the previous step and update the weight of the text features at the current moment;

[0080] (42) Based on the text features and the updated text feature weights, the image features obtained in the previous step are updated;

[0081] (43) Repeat steps (41)-(42) to update the text feature weights and image features.

[0082] In the embodiment of the present application, in the process of extracting high-level image features, in the image generation network, after the previous feature extraction layer outputs the image features, the image features obtained in the previous layer are analyzed and fed back to the correction of the text feature weights. The correction process also measures the matching of the text features and the image features based on the similarity of the text features and the image features to determine the correction value, and then updates the image features obtained in the previous step based on the text features and the updated text feature weights, that is, the image feature update process is guided by the text features and the updated text feature weights. By repeating steps (41)-(42), the text feature weights and the image features are updated until the output layer of the high-level image feature extraction model outputs the final extracted high-level image features. By repeating steps (41)-(42), in the process of the high-level image features, the next image features can be formed each time based on the text features and image features with high value, so as to achieve a high degree of matching between the text features and the image features in each image feature update process, so as to achieve more accurate image feature generation.

[0083] For further information, see Figure 4 In the above step (401), the pixel features of the global image and the local pixel features of the target coordinate area are obtained based on the original image and a small number of reference images, including:

[0084] (4011) inputting the original image and a small amount of reference images into a first image processing module and a second image processing module respectively;

[0085] (4012) extracting pixel features of a global image from the original image and a small number of reference images based on the first image processing module;

[0086] (4013) performing image feature extraction on a local area where the target power transmission component is located in the original image by a second image processing module based on the coordinate position information of the target power transmission component in the original image output by the first network model to obtain local pixel features;

[0087] (4014) Pixel features are obtained by fusing global image features and local pixel features.

[0088] In an embodiment of the present application, a first image processing module and a second image processing module are used to extract image features from an original image and a small number of reference images, respectively. The first image processing module is used to extract pixel features from the entire image, and the second image processing module is used to extract pixel features from the area where the target power transmission component is located. Based on the second image processing module, more accurate texture feature information and contour feature information of the target power transmission component can be obtained, thereby improving the authenticity of texture details and contour clarity of the target power transmission component in the generated image.

[0089] Furthermore, in the embodiment of the present application, the global image features and local pixel features are fused, and the generation of the image of the local area where the target power transmission component is located can be guided based on the global image features. At the same time, the fusion of the global image features and the local pixel features can improve the closeness of the entire generated image to the original image.

[0090] Specifically, see Figure 4 , the above (4014) is based on the fusion of global image features and local pixel features, including:

[0091] Extracting attention image features of the local pixel feature map based on a preset attention model;

[0092] Obtaining a target power transmission component feature map based on multiplying the attention image feature and the local pixel feature map;

[0093] Multiplying the attention image features based on the local pixel feature map and the global pixel feature map to obtain a first intermediate image;

[0094] Based on the addition of the first intermediate image and the global pixel feature map, a fused global image feature is obtained.

[0095] In an embodiment of the present application, for the fusion of global image features and local pixel features, on the one hand, an attention mechanism is used to retain the features of interest in the image features and extract important features. On the other hand, the attention image features of the local pixel features are fused into the global image features, so that the fused global image features add more detailed features, making the final generated image more refined in the target area.

[0096] In one embodiment, in the above step (404), the discriminator model of the image generation network, when performing image authenticity discrimination, includes:

[0097] (4041) Encoding the simulated image generated by the generator model, wherein the encoding comprises: converting the encoded data distribution into normal distribution data based on the mean and variance characteristics of the encoded data;

[0098] (4042) Comparing and analyzing the coded data, the coordinate position information, the first text information and the original image to obtain a result of authenticity judgment of the simulated image.

[0099] In an embodiment of the present application, when encoding the simulated image generated by the generator model, the complexity of the simulated image encoding data is reduced by normalizing the encoded data, so as to improve the efficiency of the discriminator model in judging the authenticity of the simulated image.

[0100] Specifically, the above-mentioned conversion of the coded data distribution into normally distributed data based on the mean and variance characteristics of the coded data includes: transforming the coded data into a Gaussian distribution based on the mean and variance of the coded data, and performing a normalization transformation after the Gaussian distribution transformation.

[0101] The present application also provides a device for generating a power transmission inspection image based on multimodal data, such as Figure 5 As shown, the device includes:

[0102] a target detection unit, configured to perform target detection based on an original image input into a first network model, and obtain text description data of at least one target power transmission component in the original image, wherein the text description data includes coordinate position information of the target power transmission component in the original image and first text information, wherein the first text information represents associated attribute information of the target power transmission component in the original image;

[0103] A reference image acquisition unit, configured to acquire a small number of matching reference images from a historical image acquisition database based on the first text information;

[0104] An image generation position acquisition unit, configured to determine a target generation area of the target power transmission component in the newly generated image based on the coordinate position information of the target power transmission component in the original image;

[0105] A new image generating unit is configured to generate the target power transmission component image in the target generation area of the newly generated image based on an original image, a small amount of the reference images and text description data input into an image generation network.

[0106] It should be noted that the multimodal data-based power transmission inspection image generation device provided in this embodiment uses the aforementioned division of functional units as an example when generating images. In actual applications, the aforementioned functions can be assigned to different functional units as needed, that is, the internal structure of the device can be divided into different functional units to complete all or part of the functions described above. In addition, the multimodal data-based power transmission inspection image generation device provided in this embodiment and the multimodal data-based power transmission inspection image generation method provided in the above embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0107] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0108] The present application also provides a device for generating a power transmission inspection image, the device comprising:

[0109] processor;

[0110] a memory for storing processor-executable instructions;

[0111] The processor implements the above-mentioned method for generating power transmission inspection images based on multimodal data by running the executable instructions.

[0112] The present invention is not limited to the above-mentioned specific implementation methods. Various changes made by ordinary technicians in this field based on the above-mentioned concept without creative work are all within the scope of protection of the present invention.

Claims

1. A method for generating a power transmission inspection image based on multimodal data, characterized in that: include: Performing target detection based on an original image input into a first network model, obtaining text description data of at least one target power transmission component in the original image, wherein the text description data includes coordinate position information of the target power transmission component in the original image and first text information, wherein the first text information represents associated attribute information of the target power transmission component in the original image; Acquiring a small number of matching reference images from a historical image acquisition database based on the first text information; determining a target generation area of the target power transmission component in the newly generated image based on the coordinate position information of the target power transmission component in the original image; Based on the original image, a small amount of the reference images and text description data input into the image generation network, the target power transmission component image is generated in the target generation area of the newly generated image.

2. The method for generating a power transmission inspection image based on multimodal data according to claim 1, characterized in that: The step of inputting an image generation network based on the original image, a small amount of the reference images, and text description data, and generating the target power transmission component image in the target generation area of the newly generated image comprises: Obtain pixel features of the global image and local pixel features of the target coordinate area based on the original image and a small number of reference images; generating corresponding text features based on the first text information; The text features and pixel features are fused and input into the generator model to generate simulated images; Authenticity discrimination is performed based on the simulated image input to the discriminator model; After the discriminator model determines that the simulated image is a real image, the simulated image is output as a newly generated image.

3. The method for generating a power transmission inspection image based on multimodal data according to claim 2, characterized in that: The fusion based on text features and pixel features includes: Measuring the matching of the text features and the image features based on the similarity between the text features and the image features; Assign weights to text features based on the degree of matching. When the matching between text features and image features is high, a high weight is assigned to the text feature; when the matching between text features and image features is low, a low weight is assigned to the text feature. Extract high-level image features based on text features, text feature weights and image features.

4. The method for generating a power transmission inspection image based on multimodal data according to claim 3, characterized in that: The extracting of high-level image features based on text features, text feature weights, and image features further includes: (41) Analyze the image features obtained in the previous step and update the weight of the text features at the current moment; (42) updating the image features obtained in the previous step based on the text features and the updated text feature weights; (43) Repeat steps (41)-(42) to update the text feature weights and image features.

5. The method for generating a power transmission inspection image based on multimodal data according to claim 2, characterized in that: The method of obtaining pixel features of a global image and local pixel features of a target coordinate area based on an original image and a small number of reference images includes: Inputting the original image and a small number of reference images into a first image processing module and a second image processing module respectively; Extracting pixel features of a global image from an original image and a small number of reference images based on a first image processing module; Based on the coordinate position information of the target power transmission component in the original image output by the first network model, the second image processing module extracts image features of a local area where the target power transmission component is located in the original image to obtain local pixel features; Pixel features are obtained by fusing global image features and local pixel features.

6. The method for generating a power transmission inspection image based on multimodal data according to claim 5, characterized in that: The fusing based on global image features and local pixel features includes: Extracting attention image features of the local pixel features based on a preset attention model; Obtaining a target power transmission component feature map based on multiplying the attention image feature and the local pixel feature; Multiplying the attention image feature based on the local pixel feature and the global image feature to obtain a first intermediate image; Based on the addition of the first intermediate image and the global image feature, a fused global image feature is obtained.

7. The method for generating a power transmission inspection image based on multimodal data according to claim 2, characterized in that: The discriminator model of the image generation network includes: Encoding the simulated image generated by the generator model, the encoding comprising: converting the encoded data distribution into normal distribution data based on mean and variance characteristics of the encoded data; A comparison analysis is performed based on the coded data, the coordinate position information, the first text information and the original image to obtain a result of authenticity judgment of the simulated image.

8. The method for generating a power transmission inspection image based on multimodal data according to claim 7, characterized in that: The method of converting the coded data distribution into normally distributed data based on the mean and variance characteristics of the coded data includes: transforming the coded data into a Gaussian distribution based on the mean and variance of the coded data, and performing a normalization transformation after the Gaussian distribution transformation.

9. A device for generating power transmission inspection images based on multimodal data, characterized in that: include: a target detection unit, configured to perform target detection based on an original image input into a first network model, and obtain text description data of at least one target power transmission component in the original image, wherein the text description data includes coordinate position information of the target power transmission component in the original image and first text information, wherein the first text information represents associated attribute information of the target power transmission component in the original image; A reference image acquisition unit, configured to acquire a small number of matching reference images from a historical image acquisition database based on the first text information; An image generation position acquisition unit, configured to determine a target generation area of the target power transmission component in the newly generated image based on the coordinate position information of the target power transmission component in the original image; The new image generating unit is configured to generate the target power transmission component image in the target generation area of the newly generated image based on an original image, a small amount of the reference images and text description data input into an image generation network.

10. A power transmission inspection image generation device, characterized in that: The device comprises: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 8 by running the executable instructions.

Images