A method and device for generating a countermeasure defect sample based on a graphic-text linkage, a terminal device, and a storage medium

Abstract

The application discloses a kind of based on graphic-text linkage's countermeasure defect sample generation method, device, terminal equipment and storage medium, belong to defect sample generation technical field, above-mentioned method is: obtaining power equipment target scene image and defect description text, defect description text is carried out semantic analysis, obtain the defect type, defect location and appearance feature of the defect to be generated countermeasure defect;From the pre-set knowledge base, match the real defect location feature corresponding to defect type, and generate local scene image;According to the visual feature of local scene image, the visual parameter of the defect to be generated countermeasure defect is adjusted, and defect texture image is generated;Finally, according to defect texture image, image embedding and edge pixel fusion are carried out on target scene image, and defect sample image of countermeasure is obtained. By implementing the present application, the problem that the accuracy of the detection result of the trained model is reduced when generating defect samples based on the existing technology of generative adversarial network or diffusion model can be solved.

Description

Technical Field

[0001] This invention relates to the field of defect sample generation technology, and in particular to a method, apparatus, terminal device and storage medium for generating countermeasure defect samples based on image and text linkage. Background Technology

[0002] In the investigation and detection of countermeasures in the field of power-related public safety, a large amount of high-quality defect sample data is usually needed to train the model in order to improve the performance of intelligent detection models. However, in actual power operation and maintenance scenarios, the probability of certain types of defects (such as "minor damage to the tower foundation" and "lack of anti-collision measures") is extremely low, resulting in a severe scarcity of real defect samples and making it difficult to collect them comprehensively. To solve this data bottleneck problem, existing technologies often use image generation methods based on generative adversarial networks (GANs) or diffusion models to expand the training dataset. Specifically, existing methods take a text instruction describing the defect as input (e.g., "generate an image of a tower with red and white anti-collision blocks") and use a pre-trained text-based image model to directly synthesize a complete image of the power equipment containing the defect.

[0003] While this method can increase the number of samples to some extent, the spatial location of the generated defects in the image is entirely determined by the randomness within the model. This leads to a significant deviation between the generated defect samples and the physical laws governing the occurrence of defects in the real world, thereby reducing the effectiveness and relevance of the generated data for model training and resulting in a decrease in the accuracy of the intelligent detection model's detection results. Summary of the Invention

[0004] This invention provides a method, apparatus, terminal device, and storage medium for generating defect samples based on image-text linkage. It can solve the problem that the defect samples generated for defect detection model training using generative adversarial networks or diffusion models in the prior art deviate significantly from the physical laws of defect occurrence in the real world, resulting in reduced accuracy of model detection results.

[0005] An embodiment of the present invention provides a method for generating defect samples of countermeasures based on image-text linkage, comprising: Acquire target scene images of power equipment in normal condition, along with defect description text; Semantic parsing of the above defect description text yields the defect type, defect location, and appearance features of the defect to be generated as a countermeasure. Based on the aforementioned defect types, the actual defect location features corresponding to the aforementioned defect types are matched from the preset knowledge base; wherein, the aforementioned preset knowledge base is constructed based on the historical defect status of each power equipment; Based on the aforementioned defect location and the actual defect location features, the defect spatial range of the aforementioned countermeasure defect to be generated is located in the aforementioned target scene image, and a defect location mask is generated based on the aforementioned defect spatial range. Based on the aforementioned defect localization mask, a local scene image representing the spatial range of the aforementioned defect is extracted from the aforementioned target scene image; The corresponding visual features are extracted from the above local scene images, and the visual parameters of the above-mentioned defect to be generated are adjusted according to the above visual features so that the above visual parameters are consistent with the above visual features. Based on the adjusted visual parameters and appearance features, a defect texture image is generated. Based on the aforementioned defect texture image, the aforementioned target scene image is subjected to image embedding and edge pixel fusion to obtain the anti-defect sample image.

[0006] Furthermore, the visual features extracted from the aforementioned local scene images include: Obtain the gray values ​​of the red channel, green channel, and blue channel of each pixel in the aforementioned local scene image, as well as the pixel's gray value. Based on the red channel grayscale values, green channel grayscale values, and blue channel grayscale values ​​of all local scene pixels, the first average grayscale value, the maximum red channel grayscale value, the minimum red channel grayscale value, the second average grayscale value, the maximum green channel grayscale value, the minimum green channel grayscale value, the third average grayscale value, the maximum blue channel grayscale value, and the minimum blue channel grayscale value of the image in the above local scene are calculated. The first average gray value, the maximum red channel gray value, the minimum red channel gray value, the second average gray value, the maximum green channel gray value, the minimum green channel gray value, the third average gray value, the maximum blue channel gray value, and the minimum blue channel gray value are used as color features. For each local scene pixel, the brightness value is calculated based on the gray values ​​of the red channel, green channel, and blue channel of the current local scene pixel. Calculate the average brightness value of all local scene pixels and use the average value as the brightness feature; The brightness difference is calculated based on the brightness values ​​of all local scene pixels, and the brightness difference is used as the contrast feature. Based on the sliding window method and the gray values ​​of the pixels mentioned above, the variance within each window in the local scene image is calculated, and the variance is used as a texture feature. The aforementioned color features, brightness features, contrast features, and texture features are used as the aforementioned visual features.

[0007] Furthermore, the above-mentioned image embedding and edge pixel fusion of the target scene image based on the aforementioned defect texture image to obtain the anti-defect sample image includes: Obtain the pixel coordinates of the target pixel point used to identify the spatial location of the defect to be generated in the above defect localization mask, as well as the number of the first horizontal pixels in the horizontal direction and the number of the first vertical pixels in the vertical direction in the above defect texture image; Based on the pixel coordinates of all target pixels, determine the number of second horizontal pixels in the horizontal direction and the number of second vertical pixels in the vertical direction of the region to be embedded in the target scene image above. Based on the first horizontal pixel count, the second horizontal pixel count, the first vertical pixel count, and the second vertical pixel count, the defect texture image is sized to obtain a target texture image that is the same size as the region to be embedded. Based on the target texture image, the pixel information of the target scene image is replaced and connected to obtain the target defect synthetic image; Based on the aforementioned defect description text, edge fusion is performed on the embedded boundaries in the synthesized image of the aforementioned target defect to obtain a sample image of the reverse defect.

[0008] Furthermore, the above-mentioned pixel information replacement and stitching of the target scene image based on the target texture image to obtain the target defect synthetic image includes: From the above target scene image, determine the target scene image pixels that correspond to each target texture pixel in the above target texture image; For each pixel in the target scene image, the pixel information of the current target scene image pixel is replaced with the pixel information of the corresponding target texture pixel to obtain the initial defect synthesis image; Determine the boundary pixels between the region to be embedded and the region not to be embedded from the initial defect-synthesized image above; Based on the aforementioned boundary pixels, the initial defect composite image is subjected to boundary pixel joining processing to obtain the target defect composite image.

[0009] Furthermore, the above-mentioned boundary pixel stitching process on the initial defect composite image based on the aforementioned boundary pixels to obtain the target defect composite image includes: Separate a number of first boundary pixels that are located within the aforementioned region to be embedded from all boundary pixels, and a number of remaining second boundary pixels; For each first boundary pixel, construct a pixel pair based on the current first boundary pixel and the second boundary pixel that will be adjacent to the current first boundary pixel; Based on the coordinates of the first boundary pixel and the second boundary pixel in each pixel pair, the first coordinate difference in the horizontal direction and the second coordinate difference in the vertical direction of each pixel pair are calculated. For each pixel pair, if the first coordinate difference is 1 and the second coordinate difference is 0; or the first coordinate difference is 0 and the second coordinate difference is 1, then the current pixel pair is determined to be in a normal connection state; otherwise, the current pixel pair is determined to be in an abnormal connection state. Anomaly repair is performed on all pixel pairs in an abnormal connection state to obtain the above-mentioned target defect synthetic image.

[0010] Furthermore, the above-mentioned abnormal repair is performed on all pixel pairs in an abnormal connection state to obtain the above-mentioned target defect synthetic image, including: Get the coordinates of the abnormal pixels in all pixel pairs that are in an abnormal connection state; For each pixel pair in an abnormal connection state, based on the above abnormal pixel coordinates, calculate the first coordinate offset in the horizontal direction and the second coordinate offset in the vertical direction between the two pixels in the current pixel pair. Based on the above abnormal coordinates, the first coordinate offset, and the second coordinate offset, the abnormal type of each pixel pair in the abnormal connection state is determined; wherein, if there are no blank coordinates between the two pixels in the current pixel pair, the abnormal pixel coordinates are not repeated, and at least one of the above first coordinate offset and the second coordinate offset is greater than a preset offset threshold, the abnormal type of the pixel pair in the current abnormal connection state is determined to be pixel misalignment. If there is a blank coordinate between two pixels in the current pixel pair, the abnormal type of the pixel pair that is currently in an abnormal connection state is determined to be pixel gap. If the abnormal pixel coordinates of two pixels in the current pixel pair are the same, the abnormal type of the pixel pair currently in an abnormal connection state is determined to be pixel overlap. For all first abnormal pixels with the abnormality type of pixel misalignment, the coordinate deviation value of each first abnormal pixel is calculated according to the preset standard coordinates and the above-mentioned abnormal pixel coordinates. Then, the corresponding first abnormal pixel is translated at an equal distance according to each coordinate deviation value to generate the above-mentioned target defect composite image. For all second abnormal pixels with the abnormality type of pixel gap, obtain the abnormal texture features of each second abnormal pixel and the normal texture features of the normal boundary pixels adjacent to each second abnormal pixel; based on the above abnormal texture features and normal texture features, perform pixel filling processing on the corresponding gap region to generate the above target defect synthetic image. For all third-order anomalous pixels with pixel overlap as the anomalous type, redundant pixel removal is performed on all third-order anomalous pixels to generate the above-mentioned target defect synthetic image.

[0011] Furthermore, the aforementioned edge fusion of the embedded boundaries in the synthesized image of the target defect based on the aforementioned defect description text yields a countermeasure defect sample image, including: From the above synthesized image of the target defect, determine the defect boundary pixels between the defect area and the non-defect area; Extend a predetermined number of pixels outward from the aforementioned defect boundary pixels to obtain the edge neighborhood range; Based on the above defect description text, edge texture analysis is performed on the defect to be generated to determine the edge texture features of the defect to be generated. Extract the visual features of all edge neighborhood pixels within the aforementioned edge neighborhood range, and compare the visual features of all edge neighborhood pixels one by one with the aforementioned edge texture features to determine the coordinates of edge neighborhood pixels that conform to the aforementioned edge texture features, thereby obtaining a texture-related pixel set. Smooth transition processing is performed on the edge neighbor pixels in the texture-associated pixel set and the edge pixels in the defect region to obtain the preliminary fused edge neighbor pixel information. Obtain the defect pixel information of the aforementioned defect area, as well as the background pixel information of the background area excluding the aforementioned defect area and edge neighborhood; The above defect pixel information, background pixel information, and edge neighborhood pixel information are used to perform pixel information fusion transition to obtain the above-mentioned anti-measure defect sample image.

[0012] Based on the above method embodiments, the present invention provides corresponding apparatus embodiments; This invention provides a device for generating defect samples for countermeasures based on image-text linkage, comprising: The system includes modules for image and text data acquisition, text semantic parsing, real defect location feature matching, defect spatial range localization, local scene image generation, defect texture image generation, and countermeasure defect sample image generation. The aforementioned image and text data acquisition module is used to acquire target scene images of power equipment in normal condition, as well as defect description text; The aforementioned text semantic parsing module is used to perform semantic parsing on the aforementioned defect description text to obtain the defect type, defect location, and appearance features of the defect to be generated as a countermeasure. The aforementioned real defect location feature matching module is used to match the real defect location features corresponding to the aforementioned defect type from a preset knowledge base based on the aforementioned defect type; wherein, the aforementioned preset knowledge base is constructed based on the historical defect status of each power equipment; The aforementioned defect spatial range localization module is used to locate the defect spatial range of the defect to be generated in the target scene image based on the defect location and the actual defect location features, and to generate a defect localization mask based on the defect spatial range. The aforementioned local scene image generation module is used to extract a local scene image representing the spatial range of the defect from the aforementioned target scene image based on the aforementioned defect localization mask. The aforementioned defect texture image generation module is used to extract corresponding visual features from the aforementioned local scene image, and adjust the visual parameters of the defect to be generated according to the aforementioned visual features so that the aforementioned visual parameters are consistent with the aforementioned visual features, and generate a defect texture image according to the adjusted visual parameters and appearance features. The aforementioned defect sample image generation module is used to perform image embedding and edge pixel fusion on the aforementioned target scene image based on the aforementioned defect texture image to obtain the defect sample image.

[0013] Based on the above method embodiments, the present invention provides a corresponding terminal device embodiment; The present invention provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the above-described method for generating countermeasure defect samples based on image-text linkage according to any embodiment of the present invention.

[0014] Based on the above method embodiments, the present invention provides a corresponding storage medium embodiment; The present invention provides a storage medium including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the above-described method for generating countermeasure defect samples based on image-text linkage according to any embodiment of the present invention.

[0015] The embodiments of the present invention have the following beneficial effects: This invention provides a method, apparatus, terminal device, and storage medium for generating countermeasure defect samples based on image-text linkage. The method includes: acquiring a target scene image of power equipment in a normal state and defect description text; then performing semantic analysis on the defect description text to obtain the defect type, defect location, and appearance features of the defect to be generated as a countermeasure; then matching the actual defect location features corresponding to the defect type from a preset knowledge base based on the defect type; wherein the preset knowledge base is constructed based on the historical defect status of each power equipment; and locating the defect in the target scene image based on the defect location and the actual defect location features. The process involves defining the spatial range of the defect to be detected and generating a defect location mask based on this range. Using the defect location mask, a local scene image representing the spatial range of the defect is extracted from the target scene image. Subsequently, corresponding visual features are extracted from the local scene image, and the visual parameters of the defect to be detected are adjusted based on these visual features to ensure consistency. A defect texture image is then generated based on the adjusted visual parameters and appearance features. Finally, image embedding and edge pixel fusion are performed on the target scene image using the defect texture image to obtain a sample image of the defect. Therefore, this application utilizes a pre-defined knowledge base constructed based on the historical defect status of power equipment to represent defect types and corresponding real defect location features. This knowledge base allows for the location of defects in images that conform to real physical laws. Furthermore, adjusting the visual parameters of the defect to be detected based on the visual features of images under normal conditions ensures the visual realism of the final generated defect. Therefore, the model trained based on this sample improves the accuracy of defect detection results. Attached Figure Description

[0016] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating a method for generating defect samples based on image and text linkage, according to an embodiment of the present invention.

[0018] Figure 2 This is a schematic diagram of a device for generating defect samples based on image and text linkage, according to an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0021] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0022] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0023] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0024] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0025] In the description of the embodiments of this application, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0026] See Figure 1 To address the issue that existing technologies using generative adversarial networks (GANs) or diffusion models to generate defect samples for defect detection model training often deviate significantly from the physical laws governing defect occurrence in the real world, leading to reduced accuracy in model detection results, this invention provides a method for generating defect samples based on image-text linkage, comprising: Step S101: Obtain a target scene image of the power equipment in a normal state, and a defect description text; Specifically, the aforementioned target scene image refers to an image that fully presents the normal appearance characteristics, installation environment, and surrounding components of the power equipment, without containing any defect information. The defect description text refers to the text content used to clearly explain the defect type and semantic information of the defect to be addressed in the countermeasure.

[0027] Specifically, the aforementioned defect types refer to the specific types of defects that are the focus of countermeasure work, while semantic information refers to relevant descriptive information used to assist in determining the location, morphology, etc., of defects. Countermeasure defects refer to power equipment defects for which samples are specifically generated for targeted research or testing training to meet the requirements of power equipment anti-accident measures.

[0028] Preferably, after acquiring the target scene image and defect description text, the target scene image can be preprocessed. The preprocessing process includes image denoising and image size standardization. Image denoising refers to removing useless information such as random noise and shooting interference from the target scene image to ensure clear and discernible images. Specifically, a mean filtering method is used to process the target scene image. For each pixel in the image, a nine-pixel region consisting of that pixel and its eight neighboring pixels is taken. The average grayscale value of all pixels within this nine-pixel region is calculated, and this average value is used as the new grayscale value for the current pixel. This process is repeated for all pixels in the image to complete denoising. Image size standardization refers to adjusting the denoised target scene image to a preset fixed size. The preset fixed size is determined according to the input requirements of the subsequent defect localization model to ensure the consistency of subsequent processing.

[0029] Step S102: Perform semantic parsing on the above defect description text to obtain the defect type, defect location, and appearance features of the defect to be generated as a countermeasure. Specifically, the aforementioned semantic parsing refers to converting the natural language in the defect description text into structured information. The process involves first semantically splitting the defect description text into independent words, then performing part-of-speech tagging and key information extraction on these words. The key information includes the defect type, defect location, and semantic information related to the defect's appearance features. In this invention, the defect location obtained directly from text parsing is used as the initial defect location. This location is then combined with real defect location features obtained from a pre-set knowledge base to achieve more accurate defect localization. The aforementioned appearance features include the defect's shape, color, and degree of damage.

[0030] Step S103: Based on the above defect types, match the actual defect location features corresponding to the above defect types from the preset knowledge base; wherein, the above preset knowledge base is constructed based on the historical defect status of each power equipment; Specifically, the pre-set knowledge base refers to a pre-stored set of correspondences between different defect types and their real and common locations on power equipment. This set is obtained by summarizing a large number of historical defect cases of power equipment.

[0031] Step S104: Based on the above defect location and the actual defect location features, locate the defect spatial range of the above-mentioned countermeasure defect to be generated in the above-mentioned target scene image, and generate a defect location mask based on the above-mentioned defect spatial range. Specifically, based on the defect location obtained through semantic parsing, the corresponding power equipment component region is identified in the target scene image. Then, according to the real defect location features corresponding to this defect type in a pre-set knowledge base, the specific spatial range of the defect to be generated within the identified component region is determined. The pixel value of the region corresponding to this specific spatial range is set to 255, and the pixel value of the remaining regions is set to 0, generating a defect localization mask with the same size as the target scene image. The region of the defect localization mask corresponding to the spatial location of the defect to be generated is identified with a specific pixel value, and the remaining regions are identified with another specific pixel value.

[0032] Indicatively, if the target scene image is a normal utility pole image, it clearly shows the complete structure of the pole, the concrete pole body, and the crossarm connecting components in the middle of the pole body, without any defect information. The obtained defect description text is "utility pole body crack defect, the crack is located in the lower middle area of ​​the utility pole". The utility pole image is then preprocessed, using a mean filtering method for noise reduction. For each pixel in the image, eight surrounding pixels are taken to form a nine-pixel region, and the average gray value of these nine pixels is calculated as the new gray value of the current pixel. This process is repeated for all pixels to complete the noise reduction. Finally, the denoised utility pole image is standardized to a fixed size of 1024*1024.

[0033] Subsequently, semantic parsing was performed on the defect description text to obtain words such as "telephone pole," "pole body crack defect," "crack location," "located in," and "lower-middle area of ​​the telephone pole." After part-of-speech tagging, key information was extracted as follows: defect type: "telephone pole body crack defect," defect location: "lower-middle area of ​​the telephone pole." The extracted defect location was matched with a pre-set knowledge base, which stores the correspondence that "telephone pole body crack defects are commonly found in the lower-middle and root areas of telephone poles." Further, based on the defect location, the pole body area was located in the standardized telephone pole image. Combining the matched information "commonly found in the lower-middle area of ​​the telephone pole" and the semantically parsed information "lower-middle area of ​​the telephone pole," the specific spatial range of the pole body crack defect to be generated was determined to be a 200*200 pixel area in the lower-middle part of the telephone pole body. A defect location mask with the same size as the 1024*1024 standardized image was generated, and the pixel value of the 200*200 pixel area was set to 255, while the pixel value of the remaining areas was set to 0, thus completing the generation of the defect location mask.

[0034] Step S105: Based on the above defect localization mask, extract the local scene image representing the spatial range of the defect from the above target scene image; Specifically, the target scene image is isolated based on a defect localization mask to obtain a local scene image containing only the region corresponding to the defect to be generated as a countermeasure. Region isolation refers to the process of extracting the region in the target scene image corresponding to the defect identified by the defect localization mask using the pixel value differences in the defect localization mask. Illustratively, the pixel positions of the defect localization mask and the target scene image are compared pixel by pixel. All pixels in the target scene image corresponding to the region with a pixel value of 255 in the defect localization mask are extracted, while preserving the relative positional relationship of these pixels in the target scene image. The extracted pixels are then combined according to the preserved relative positional relationship to form the local scene image. Pixels in the target scene image corresponding to the region with a pixel value of 0 in the defect localization mask are directly discarded and not included in the composition of the local scene image.

[0035] Schematic illustration: the defect localization mask corresponds to a 200*200 pixel area in the lower middle section of the utility pole, where pixel value 255 is the target scene image, which is a normalized 1024*1024 image of a utility pole. By comparing the defect localization mask and the target scene image pixel by pixel, all pixels in the 200*200 pixel area corresponding to pixel value 255 in the defect localization mask are extracted. The relative positional relationships of these pixels in the target scene image are preserved, and the images are combined to form a local scene image containing only the 200*200 pixel area in the lower middle section of the utility pole. Pixels in the remaining areas corresponding to pixel value 0 in the defect localization mask are discarded.

[0036] Step S106: Extract the corresponding visual features from the above local scene image, and adjust the visual parameters of the above-mentioned defect to be generated according to the above visual features so that the above visual parameters are consistent with the above visual features, and generate a defect texture image according to the adjusted visual parameters and appearance features. Specifically, visual feature extraction refers to extracting key information from a local scene image that reflects its visual attributes, including color features, brightness features, contrast features, and texture features. Then, a joint analysis is performed based on the local scene image and the defect description text. Based on the extracted visual features of the local scene image, the visual parameters of the defect texture are adjusted to ensure that the average grayscale value of the defect texture's color channels, average brightness, contrast, and other parameters are consistent with the corresponding parameters in the local scene image. Finally, the adjusted visual parameters of the defect texture are combined with the appearance features to generate a defect texture image. Visual consistency refers to maintaining consistency in visual features such as color, brightness, contrast, and texture detail.

[0037] To illustrate, the defect description text "pole body crack defect, the crack is located in the lower middle part of the pole" obtained in the previous steps is called to perform defect feature analysis and extract the appearance feature as a linearly extending crack shape; based on the extracted local scene image visual features, the visual parameters of the linearly extending crack texture are adjusted so that the visual parameters of the crack texture are basically consistent with the visual features of the local scene image; the adjusted linearly extending crack texture is combined with the appearance features to generate a 200*200 pixel defect texture image.

[0038] In a preferred embodiment, the extraction of corresponding visual features from the aforementioned local scene image includes: Obtain the gray values ​​of the red channel, green channel, and blue channel of each pixel in the aforementioned local scene image, as well as the pixel's gray value. Based on the red channel grayscale values, green channel grayscale values, and blue channel grayscale values ​​of all local scene pixels, the first average grayscale value, the maximum red channel grayscale value, the minimum red channel grayscale value, the second average grayscale value, the maximum green channel grayscale value, the minimum green channel grayscale value, the third average grayscale value, the maximum blue channel grayscale value, and the minimum blue channel grayscale value of the image in the above local scene are calculated. The first average gray value, the maximum red channel gray value, the minimum red channel gray value, the second average gray value, the maximum green channel gray value, the minimum green channel gray value, the third average gray value, the maximum blue channel gray value, and the minimum blue channel gray value are used as color features. For each local scene pixel, the brightness value is calculated based on the gray values ​​of the red channel, green channel, and blue channel of the current local scene pixel. Calculate the average brightness value of all local scene pixels and use the average value as the brightness feature; The brightness difference is calculated based on the brightness values ​​of all local scene pixels, and the brightness difference is used as the contrast feature. Based on the sliding window method and the gray values ​​of the pixels mentioned above, the variance within each window in the local scene image is calculated, and the variance is used as a texture feature. Specifically, the sliding window method is used to traverse the local scene image. The sliding window method refers to setting a window of a fixed size, starting from the top left corner of the local scene image, moving the window in a fixed step size from left to right and from top to bottom, counting the gray values ​​of the pixels in each window, calculating the variance of the gray values ​​of the pixels in the window, and using the variance values ​​of all windows as the texture features of the local scene image.

[0039] The aforementioned color features, brightness features, contrast features, and texture features are used as the aforementioned visual features.

[0040] The process of visual feature extraction for a local scene image is illustrated as follows: The grayscale values ​​of the red, green, and blue color channels for each pixel in the local scene image are calculated. The average first grayscale value of the red channel is 180, the maximum red channel grayscale value is 220, and the minimum red channel grayscale value is 150. The average second grayscale value of the green channel is 185, the maximum green channel grayscale value is 225, and the minimum green channel grayscale value is 155. The average third grayscale value of the blue channel is 190, the maximum blue channel grayscale value is 230, and the minimum blue channel grayscale value is 155. The color channel grayscale value is 160; the average of the red, green, and blue color channel grayscale values ​​of each local scene pixel is calculated as the brightness value, and the average brightness of the local scene image is 185; the brightness difference of 76 between the maximum brightness value of 228 and the minimum brightness value of 152 is calculated as the contrast of the local scene image; a 3*3 window is set, and the window slides from the top left corner of the local scene image with a step size of 1 pixel, calculating the variance of the grayscale values ​​of each pixel in the window, and obtaining a series of variance values ​​as the texture features of the local scene image.

[0041] In this preferred embodiment, the visual features in the local scene image are calculated using the grayscale values ​​of the three colors red, green, and blue in the local scene image.

[0042] Step S107: Based on the above defect texture image, perform image embedding and edge pixel fusion on the above target scene image to obtain the anti-defect sample image.

[0043] Specifically, firstly, the defect texture image is embedded into the target scene image at the spatial location defined by the defect location mask based on the defect location mask to obtain the target defect synthetic image. Then, based on the defect location mask and the defect description text, edge fusion is performed on the region defined by the defect location mask and its edge neighborhood in the target defect synthetic image to obtain the anti-defect sample image.

[0044] Specifically, by clearly defining the spatial location of the defect to be generated in the target scene image as defined by the defect localization mask, the defect texture image is aligned with this spatial location to ensure a one-to-one correspondence between the pixels of the defect texture image and the pixels of the corresponding spatial location in the target scene image. Finally, the pixel information of the defect texture image replaces the pixel information of the corresponding spatial location in the target scene image to complete the embedding and obtain the target defect synthetic image. Subsequently, the edge position and edge neighborhood range of the defect region in the target defect synthetic image are determined according to the defect localization mask. The edge neighborhood refers to the area extending outward from the edge of the defect region by a certain distance, which is determined according to the defect type and image resolution. Further, by combining the morphology, material, and other information of the defect in the defect description text, an appropriate edge blending strategy is selected. Further, based on the selected edge blending strategy, the pixel information of the defect region and its edge neighborhood is smoothed to allow the defect region to naturally connect with the surrounding area of ​​the target scene image, resulting in the countermeasure defect sample image.

[0045] In a preferred embodiment, the above-mentioned image embedding and edge pixel fusion of the target scene image based on the defect texture image to obtain the anti-defect sample image includes: Obtain the pixel coordinates of the target pixel point used to identify the spatial location of the defect to be generated in the above defect localization mask, as well as the number of the first horizontal pixels in the horizontal direction and the number of the first vertical pixels in the vertical direction in the above defect texture image; Specifically, firstly, embedding region analysis is performed based on the first pixel distribution features of the defect localization mask to obtain the pixel coordinate range of the region to be embedded in the target scene image. Embedding region analysis refers to the process of determining the specific region in the target scene image where the defect texture image needs to be embedded by identifying the distribution range of target pixels in the defect localization mask. The region to be embedded refers to the specific area in the target scene image used to place the defect texture image, which completely corresponds to the area defined by the target pixels in the defect localization mask. The pixel coordinate range refers to the boundary range of the region to be embedded, expressed in pixel coordinate values. Pixel coordinates are defined with the top-left corner of the target scene image as the origin, with the horizontal axis pointing to the right and the vertical axis pointing downwards.

[0046] Subsequently, a full pixel traversal scan is performed on the defect localization mask. This scan starts from the top-left corner pixel and proceeds sequentially from left to right and top to bottom, examining each pixel in the mask. During this scan, the pixel coordinates of all target pixels are recorded. A target pixel is a pixel in the defect localization mask whose specific pixel value identifies the spatial location of the defect to be generated. Similarly, for the defect texture image, a full pixel traversal scan is performed to obtain the first horizontal pixel count and the first vertical pixel count.

[0047] Based on the pixel coordinates of all target pixels, determine the number of second horizontal pixels in the horizontal direction and the number of second vertical pixels in the vertical direction of the region to be embedded in the target scene image above. Specifically, statistical analysis is performed on the pixel coordinates of all recorded target pixels to determine the minimum and maximum horizontal coordinate values, as well as the minimum and maximum vertical coordinate values. Then, these values ​​are combined to obtain the pixel coordinate range of the region to be embedded. This range is a rectangular area where the horizontal coordinates extend from the minimum to the maximum horizontal coordinate value, and the vertical coordinates extend from the minimum to the maximum vertical coordinate value.

[0048] To illustrate, the defect localization mask is 1024*1024 pixels in size (consistent with the target scene image size), with target pixels having a pixel value of 255, and the remaining areas having a pixel value of 0. A full pixel scan is then performed on this defect localization mask, starting from the top left corner (horizontal coordinate 0, vertical coordinate 0), checking the pixel value of each pixel sequentially, and recording the pixel coordinates of all target pixels with a pixel value of 255. Statistically, among the recorded target pixels, the minimum horizontal coordinate is 300 and the maximum is 500, and the minimum vertical coordinate is 400 and the maximum is 600. Combining these coordinate values ​​yields a rectangular area with pixel coordinates ranging from 300 to 500 horizontally and from 400 to 600 vertically, corresponding to the lower part of the utility pole in the target scene image.

[0049] Then, the pixel size of the region to be embedded is calculated to obtain the second horizontal pixel count and the second vertical pixel count. The calculation method is as follows: the second horizontal pixel count is equal to its maximum horizontal coordinate value minus its minimum horizontal coordinate value, and the second vertical pixel count is equal to its maximum vertical coordinate value minus its minimum vertical coordinate value.

[0050] Based on the first horizontal pixel count, the second horizontal pixel count, the first vertical pixel count, and the second vertical pixel count, the defect texture image is sized to obtain a target texture image that is the same size as the region to be embedded. Specifically, firstly, based on the number of first horizontal pixels, the number of second horizontal pixels, the number of first vertical pixels, and the number of second vertical pixels, the size matching relationship between the defect texture image and the region to be embedded is determined. Then, based on the size matching relationship, the defect texture image is cropped to obtain a target texture image whose size matches the region to be embedded.

[0051] Specifically, the number of first horizontal pixels in the defect texture image is compared with the number of second horizontal pixels in the region to be embedded, and the number of first vertical pixels in the defect texture image is also compared with the number of second vertical pixels in the region to be embedded. The size matching relationship is determined based on the results of the two comparisons: if both the number of first horizontal pixels and the number of first vertical pixels are equal to the corresponding number of pixels in the region to be embedded, the size matching relationship is that the sizes are consistent; if the number of first horizontal pixels in the defect texture image is greater than the number of second horizontal pixels in the region to be embedded, or the number of first vertical pixels is greater than the number of second vertical pixels in the region to be embedded, the size matching relationship is that the size is too large; if the number of first horizontal pixels in the defect texture image is less than the number of second horizontal pixels in the region to be embedded, or the number of first vertical pixels is less than the number of second vertical pixels in the region to be embedded, the size matching relationship is that the size is too small.

[0052] To illustrate, if the pixel coordinates of the region to be embedded range from 300 to 500 horizontally and from 400 to 600 vertically, the pixel size of the region to be embedded is calculated as follows: the number of pixels in the second horizontal direction is 500 minus 300, which equals 200; the number of pixels in the second vertical direction is 600 minus 400, which equals 200. Therefore, the pixel size of the region to be embedded is 200*200 pixels. The pixel size information of the defect texture image is 250*250 pixels (250 pixels horizontally and 250 pixels vertically). Comparing the first horizontal pixel count of 250 in the defect texture image with the second horizontal pixel count of 200 in the region to be embedded, and comparing the first vertical pixel count of 250 with the second vertical pixel count of 200 in the region to be embedded, since the horizontal and vertical pixel counts of the defect texture image are both greater than the corresponding pixel counts of the region to be embedded, the size matching relationship is determined to be too large.

[0053] Specifically, pixel cropping refers to the process of cropping a section from the defect texture image that is exactly the same size as the region to be embedded, based on the pixel dimensions of the region to be embedded. The target texture image refers to the defect texture image that, after pixel cropping, is sized to fit the region to be embedded and can be directly embedded into it. Size fit means that the number of horizontal and vertical pixels in the target texture image is equal to the corresponding number of pixels in the region to be embedded.

[0054] Specifically, the need for pixel cropping can be determined based on the size matching relationship. Specifically: if the size matching relationship is that the sizes are the same, no cropping is needed, and the defect texture image can be directly used as the target texture image; if the size matching relationship is that the size is too large or too small, cropping is required (if the size is too small, the defect texture image needs to be enlarged first. The enlargement method is to use pixel interpolation, which means to add new pixels between existing pixels to increase the image size. The gray value of the added pixels is calculated based on the gray values ​​of the surrounding existing pixels. The calculation method is to take the average of the gray values ​​of two adjacent pixels as the gray value of the added pixel, and then cropping is performed).

[0055] Specifically, if cropping is required, the cropping range needs to be determined. Specifically: when the size is too large, use the center of the defect texture image as a reference to determine a cropping range consistent with the pixel size of the region to be embedded. The number of horizontal pixels in the cropping range equals the number of the second horizontal pixels of the region to be embedded, and the number of vertical pixels equals the number of the second vertical pixels of the region to be embedded. When the size is too small, first enlarge the defect texture image to a size not smaller than the pixel size of the region to be embedded, then use the center of the enlarged image as a reference to determine a cropping range consistent with the pixel size of the region to be embedded.

[0056] Specifically, the defect texture image (or the magnified defect texture image) is cropped according to the determined cropping range, removing pixels outside the cropping range and retaining pixels within the cropping range to obtain the target texture image.

[0057] Indicatively, if the size matching relationship indicates the size is too large, pixel cropping of the defect texture image is required. If the defect texture image is 250*250 pixels and the area to be embedded is 200*200 pixels, then the cropping range is determined based on the center of the defect texture image (125 pixels horizontally and 125 pixels vertically): horizontally from 125 minus 100 (25) to 125 plus 100 (225 pixels), and vertically from 125 minus 100 (25) to 125 plus 100 (225 pixels). That is, the cropping range is a 200*200 pixel area, with pixels ranging from 25 to 225 pixels horizontally and 25 to 225 pixels vertically. The 250*250 pixel defect texture image is cropped according to this range, removing pixels outside the cropping range and retaining those within it, resulting in a 200*200 pixel target texture image. The size of this target texture image perfectly matches the size of the area to be embedded.

[0058] Based on the target texture image, the pixel information of the target scene image is replaced and connected to obtain the target defect synthetic image; Specifically, firstly, pixel fusion is performed based on the pixel arrangement order of the target texture image and the pixel distribution order of the target pixels used to define the region to be embedded in the defect localization mask to obtain an initial defect synthesis image. Here, the pixel arrangement order refers to the arrangement of pixels in the image from left to right and from top to bottom. The pixel distribution order of the target pixels refers to the distribution order of all target pixels in the defect localization mask from left to right and from top to bottom. Pixel fusion refers to the process of replacing and fusing the pixel information of the target texture image with the pixel information of the region to be embedded in the target scene image.

[0059] Specifically, by matching the pixel arrangement of the target texture image with the pixel distribution order of the target pixels in the defect localization mask, the pixel information of each pixel in the target texture image is used to replace the pixel information of the corresponding target pixel position in the target scene image, thus completing pixel fusion and obtaining the initial defect synthesis image.

[0060] Subsequently, pixel-by-pixel stitching is performed based on the pixel boundary features of the target pixels and the pixel distribution features of the preliminary defect synthesis image to obtain the target defect synthesis image. The pixel boundary features of the target pixels refer to the pixel distribution features of the edges of the target pixel region in the defect localization mask, including the coordinates of edge pixels and the pixel value variation patterns. The pixel distribution features of the preliminary defect synthesis image refer to the pixel value types of all pixels in the preliminary defect synthesis image and the spatial distribution of each type of pixel. Pixel stitching refers to the process of making preliminary pixel adjustments at the junction of the target texture image and the target scene image in the initial defect synthesis image to reduce abrupt transitions.

[0061] Specifically, by identifying the pixel boundary features of the target pixels, the connection area between the target texture image and the target scene image in the initial defect synthesis image is determined. Combined with the second pixel distribution features of the initial defect synthesis image, the pixel information of the connection area is adjusted to complete the pixel connection and obtain the target defect synthesis image.

[0062] Based on the aforementioned defect description text, edge fusion is performed on the embedded boundaries in the synthesized image of the aforementioned target defect to obtain a sample image of the reverse defect.

[0063] Preferably, the present invention can ensure that the defect is accurately and completely embedded in a specific position in the target scene image, and the generated target defect synthetic image has good basic visual consistency. After edge fusion, a high-quality anti-defect sample image can be obtained. Therefore, in the anti-defect sample image, the shape and position of small targets and defects in specific positions are consistent with the actual defect characteristics, which can provide targeted training data for intelligent detection models and improve the accuracy of intelligent detection models in recognizing small targets and defects in specific positions.

[0064] In this preferred embodiment, the target scene image is image embedded and edge pixel fusion is performed based on the defect texture image to obtain the anti-defect sample image.

[0065] In another preferred embodiment, the above-mentioned pixel information replacement and splicing of the target scene image based on the target texture image to obtain the target defect synthetic image includes: From the above target scene image, determine the target scene image pixels that correspond to each target texture pixel in the above target texture image; Specifically, firstly, based on the pixel arrangement order and distribution order of the target texture image, a pixel association relationship is constructed between the target texture image and the region to be embedded in the target scene image. This then yields the corresponding pixels in the target scene image. Here, the pixel association relationship refers to the one-to-one mapping between each target texture pixel in the target texture image and each pixel in the region to be embedded. The pixel arrangement order refers to the arrangement of the target texture pixels in the target texture image from left to right and from top to bottom. The distribution order refers to the distribution of all target pixels in the defect localization mask from left to right and from top to bottom.

[0066] Specifically, a full-pixel traversal scan is performed on both the target texture image and the defect localization mask. This full-pixel traversal scan refers to the process of checking and recording each pixel sequentially from left to right and top to bottom, starting from the top-left corner of the image or region. Secondly, during the traversal scan, each target texture pixel in the target texture image is sequentially numbered according to the traversal order, starting from the first number and increasing sequentially to obtain the texture pixel sequence number for each target texture pixel. Simultaneously, each mask pixel in the defect localization mask is sequentially numbered according to the same traversal order, also starting from the first number and increasing sequentially to obtain the mask pixel sequence number for each mask. Finally, texture pixel sequences with the same sequence number are associated with mask pixel sequences, establishing a one-to-one correspondence between the target texture image pixel corresponding to each texture pixel sequence number and the pixel in the region to be embedded corresponding to each mask pixel sequence number. This correspondence is the pixel association relationship between the target texture image and the region to be embedded, thus obtaining the target scene image pixels corresponding to each target texture pixel in the target texture image.

[0067] Indicatively, the target texture image is 200*200 pixels, showing a linearly extending crack texture on a utility pole. The defect localization mask is 1024*1024 pixels. A full pixel traversal scan is performed on the target texture image, starting from the top left corner (horizontal 0, vertical 0), and each pixel is numbered in a left-to-right, top-to-bottom order. The first pixel (horizontal 0, vertical 0) is numbered 1, the second pixel (horizontal 1, vertical 0) is numbered 2, and so on, up to the 40,000th pixel (horizontal 199, vertical 199) is numbered 40,000, resulting in a number of texture pixel numbers. Simultaneously, a full pixel traversal scan is performed on the defect localization mask. Each target pixel is numbered in the same left-to-right and top-to-bottom order: the first target pixel (horizontal 300, vertical 400) is numbered 1, the second target pixel (horizontal 301, vertical 400) is numbered 2, and so on, up to the 40,000th target pixel (horizontal 499, vertical 599) is numbered 40,000, resulting in a number of mask pixel numbers. Subsequently, texture pixel number 1 is associated with mask pixel number 1, texture pixel number 2 is associated with mask pixel number 2, and so on, up to texture pixel number 40,000 is associated with mask pixel number 40,000, thus establishing the pixel association relationship between the target texture image and the region to be embedded in the target scene image.

[0068] For each pixel in the target scene image, the pixel information of the current target scene image pixel is replaced with the pixel information of the corresponding target texture pixel to obtain the initial defect synthesis image; Specifically, the aforementioned pixel information refers to the inherent visual feature information of the corresponding pixel, including the grayscale values ​​and brightness values ​​of the red, green, and blue color channels of the pixel. In this replacement process, firstly, the pixel coordinates of all pixels in the target scene image are individually determined to see if they fall within the pixel coordinate range of the region to be embedded. The criterion is: if the horizontal coordinate value of a pixel is between the minimum and maximum horizontal coordinate values ​​within the pixel coordinate range, and the vertical coordinate value is between the minimum and maximum vertical coordinate values ​​within the pixel coordinate range, then the pixel coordinate belongs to the region to be embedded; otherwise, the pixel coordinate does not belong to the region to be embedded. All pixel coordinates that do not belong to the region to be embedded are filtered out, resulting in the background pixel coordinate set, and the pixel coordinates that belong to the region to be embedded are obtained, resulting in the set of pixel coordinates to be embedded.

[0069] Indicatively, the pixel coordinates of the region to be embedded range from 300 to 500 horizontally and 400 to 600 vertically. The target scene image (a normal 1024*1024 pixel image of a utility pole) contains all pixel coordinates from 0 to 1023 horizontally and 0 to 1023 vertically. Each pixel coordinate in the target scene image is evaluated individually. For example, pixel coordinates (horizontal 299, vertical 399) have a horizontal coordinate of 299 less than the minimum horizontal coordinate of 300 and a vertical coordinate of 399 less than the minimum vertical coordinate of 400, therefore they belong to the background pixel coordinate set. Pixel coordinates (horizontal 350, vertical 450) have a horizontal coordinate of 350 between 300 and 500 and a vertical coordinate of 450 between 400 and 600, therefore they belong to the pixel coordinate set corresponding to the region to be embedded. Pixel coordinates (horizontal 501, vertical 550) have a horizontal coordinate of 501 greater than the maximum horizontal coordinate of 500, therefore they belong to the background pixel coordinate set.

[0070] Then, the pixel correlation relationship and pixel information of the target texture image are invoked, and the pixel coordinates of each mask pixel number in the defect localization mask corresponding to the region to be embedded are obtained. Based on the pixel correlation relationship, the mask pixel number corresponding to each texture pixel number is found, and the pixel coordinates of the region to be embedded corresponding to the target texture image pixel point of each texture pixel number are determined, thus establishing a one-to-one correspondence between the coordinates of the target texture image pixel point and the coordinates of the region to be embedded.

[0071] Then, following the pixel arrangement order of the target texture image, the pixel information of each target texture pixel is extracted sequentially. The extracted pixel information is then filled into the corresponding embedding coordinates in the target scene image, replacing the original pixel information of the target scene image pixels at the embedding coordinates, thus completing the filling of all pixels of the target texture image into the embedding area. For the background pixel coordinate set, its original pixel information is retained.

[0072] To illustrate, pixel relationships are as follows: texture pixel number 1 corresponds to mask pixel number 1, texture pixel number 2 corresponds to mask pixel number 2, and so on. Pixel information of the target texture image includes the target texture pixel corresponding to texture pixel number 1, with a red channel grayscale value of 178, a green channel grayscale value of 183, a blue channel grayscale value of 188, and a brightness value of 183. The pixel coordinates of the pixels corresponding to each mask pixel number in the defect localization mask (i.e., the corresponding target scene image pixels) are as follows: mask pixel number 1 corresponds to pixel coordinates (horizontal 300, vertical 400), mask pixel number 2 corresponds to pixel coordinates (horizontal 301, vertical 400), and so on. Based on these pixel relationships, the embedding coordinates corresponding to texture pixel number 1 are determined to be (horizontal 300, vertical 400), texture pixel number 2 corresponds to (horizontal 301, vertical 400), and so on, establishing a correspondence between all target texture image pixels and their embedding coordinates. Then, following the order from left to right and from top to bottom, the pixel information corresponding to texture pixel number 1 is extracted first and filled into the target scene image at the embedding coordinate (horizontal 300, vertical 400), replacing the original pixel information at that coordinate; then the pixel information corresponding to texture pixel number 2 is extracted and filled into the embedding coordinate (horizontal 301, vertical 400)... until the pixel information corresponding to the 40,000th texture pixel number is filled into the embedding coordinate (horizontal 499, vertical 599), completing the filling of all pixels of the target texture image into the embedding area.

[0073] Finally, the pixel information of each pixel in the preserved set of background pixel coordinates and the pixel information after filling and replacement are integrated to obtain the initial defect synthesis image.

[0074] Determine the boundary pixels between the region to be embedded and the region not to be embedded from the initial defect-synthesized image above; Specifically, in the preliminary defect synthesis image, all pixels at the boundary between the region to be embedded and the background region (i.e., the non-embedded region) are the aforementioned boundary pixels. These boundary pixels directly constitute the connecting boundary between the two regions. The specific process for determining the boundary pixels is as follows: First, extract the edge coordinate range at the boundary between the target pixel (i.e., the pixel with a grayscale value of 255) and the non-target pixel (i.e., the pixel with a grayscale value of 0) in the defect localization mask. Second, based on the extracted edge coordinate range, determine the corresponding candidate pixel region in the preliminary defect synthesis image. The candidate pixel region refers to the region in the preliminary defect synthesis image whose coordinates are within the edge coordinate range of the target pixel and extend outward by 1 pixel. The purpose of extending by 1 pixel is to cover the possible connecting transition region between the region to be embedded and the background region. Next, for each pixel within the candidate pixel region, pixel attribution is determined. Pixel attribution refers to determining whether a pixel belongs to the region to be embedded or the background region based on its pixel value type. The criteria are: if the pixel value type of a pixel is consistent with the pixel value type of the target texture image, then the pixel belongs to the region to be embedded; if the pixel value type of a pixel is consistent with the original pixel value type of the background region of the target scene image, then the pixel belongs to the background region. Therefore, pixels at the boundary where both the region to be embedded and the background region exist within the candidate pixel region can be selected. These boundary pixels are the aforementioned boundary pixels.

[0075] For illustration, the edge coordinates range from 300 to 500 horizontally and 400 to 600 vertically. Based on this range, the candidate pixel region is determined to be from 299 to 501 horizontally and from 399 to 601 vertically (extending the original edge range outward by 1 pixel). For each pixel within the candidate pixel region, pixel classification is determined. For example, a pixel with coordinates (300 horizontally, 400 vertically) whose pixel value type is crack texture pixel value is classified as a pixel in the region to be embedded; a pixel with coordinates (299 horizontally, 400 vertically) whose pixel value type is normal concrete like a utility pole pixel value is classified as a pixel in the background region; and a pixel with coordinates (300 horizontally, 399 vertically) whose pixel value type is normal concrete like a utility pole pixel value is classified as a pixel in the background region. Filter out the pixels at the intersection, such as (horizontal 300, vertical 400), (horizontal 301, vertical 400) ... (horizontal 500, vertical 400), (horizontal 300, vertical 401), (horizontal 500, vertical 401) ... (horizontal 300, vertical 600), (horizontal 500, vertical 600), etc., which are the boundary pixels.

[0076] Based on the aforementioned boundary pixels, the initial defect composite image is subjected to boundary pixel joining processing to obtain the target defect composite image.

[0077] Preferably, the present invention can ensure the accurate and complete embedding of small targets and defects in specific locations in the target scene image, and retain the original features of the scene to the greatest extent to obtain an initial defect synthesis image. After edge fusion, a high-quality anti-defect sample image can be formed, providing targeted training data for the intelligent detection model and improving the accuracy of the intelligent detection model in recognizing small targets and defects in specific locations.

[0078] In this preferred embodiment, the target scene image is replaced and stitched together with pixel information based on the target texture image to obtain a composite image of the target defect.

[0079] In another preferred embodiment, the above-mentioned boundary pixel concatenation processing of the initial defect composite image based on the boundary pixels to obtain the target defect composite image includes: Separate a number of first boundary pixels that are located within the aforementioned region to be embedded from all boundary pixels, and a number of remaining second boundary pixels; Specifically, all boundary pixels are divided into first boundary pixels and second boundary pixels according to their category. The first boundary pixels are pixels belonging to the region to be embedded, and the second boundary pixels are pixels belonging to the background region.

[0080] For each first boundary pixel, construct a pixel pair based on the current first boundary pixel and the second boundary pixel that will be adjacent to the current first boundary pixel; Specifically, a correspondence is established between the first boundary pixel and the second boundary pixel. The correspondence is established as follows: for each first boundary pixel, find its adjacent pixels in the horizontal and vertical directions. If the adjacent pixel is a second boundary pixel, then establish a correspondence between the first boundary pixel and the corresponding second boundary pixel to obtain several pixel pairs.

[0081] Based on the coordinates of the first boundary pixel and the second boundary pixel in each pixel pair, the first coordinate difference in the horizontal direction and the second coordinate difference in the vertical direction of each pixel pair are calculated. Specifically, for the first boundary pixel and the second boundary pixel in each pixel pair, their positional relationship is determined by calculating the first coordinate difference in the horizontal direction and the second coordinate difference in the vertical direction.

[0082] For each pixel pair, if the first coordinate difference is 1 and the second coordinate difference is 0; or the first coordinate difference is 0 and the second coordinate difference is 1, then the current pixel pair is determined to be in a normal connection state; otherwise, the current pixel pair is determined to be in an abnormal connection state. Specifically, if the first coordinate difference is 1 and the second coordinate difference is 0, or the first coordinate difference is 0 and the second coordinate difference is 1, then the two are judged to be in a normal connection state; if the first coordinate difference and the second coordinate difference do not meet the above conditions, then the two are judged to be in an abnormal connection state.

[0083] Indicatively, boundary pixels are divided into first boundary pixels (e.g., (horizontal 300, vertical 400), (horizontal 301, vertical 400) etc.) and second boundary pixels (e.g., (horizontal 299, vertical 400), (horizontal 300, vertical 399) etc.). For a first boundary pixel (horizontal 300, vertical 400), find its horizontal left-hand neighbor (horizontal 299, vertical 400), which belongs to the second boundary pixel, and establish a correspondence between them; find its vertical upper-hand neighbor (horizontal 300, vertical 399), which also belongs to the second boundary pixel, and establish a correspondence between them. The positional relationship between the first boundary pixel (horizontal 300, vertical 400) and the second boundary pixel (horizontal 299, vertical 400) is determined. The horizontal coordinate difference is 300-299=1, and the vertical coordinate difference is 400-400=0, indicating a normal connection. The positional relationship between the first boundary pixel (horizontal 300, vertical 400) and the second boundary pixel (horizontal 300, vertical 399) is determined. If the horizontal coordinate difference is 300-300=0 and the vertical coordinate difference is 400-399=1, it is judged as a normal connection state. If the second boundary pixel corresponding to a certain first boundary pixel (horizontal 300, vertical 402) is (horizontal 298, vertical 402), the horizontal coordinate difference is 300-298=2 and the vertical coordinate difference is 0, then it is judged as an abnormal connection state. The positional connection state of all pixel pairs is then obtained.

[0084] Anomaly repair is performed on all pixel pairs in an abnormal connection state to obtain the above-mentioned target defect synthetic image.

[0085] In this preferred embodiment, the initial defect composite image is processed by boundary pixel joining based on the boundary pixels to obtain the target defect composite image.

[0086] In another preferred embodiment, the above-mentioned abnormal repair of all pixel pairs in an abnormal connection state to obtain the above-mentioned target defect synthetic image includes: Get the coordinates of the abnormal pixels in all pixel pairs that are in an abnormal connection state; Specifically, for each pixel pair in an abnormal connection state, the abnormal pixel coordinates of the first boundary pixel and the second boundary pixel are obtained respectively.

[0087] For each pixel pair in an abnormal connection state, based on the above abnormal pixel coordinates, calculate the first coordinate offset in the horizontal direction and the second coordinate offset in the vertical direction between the two pixels in the current pixel pair. Specifically, all pixel pairs in abnormal connection state are treated as a set, and each pixel pair in the set is analyzed one by one. The difference in horizontal coordinates and vertical coordinates between the first boundary pixel and the second boundary pixel in the pixel pair are calculated to obtain the first coordinate offset and the second coordinate offset.

[0088] Based on the above abnormal coordinates, the first coordinate offset, and the second coordinate offset, the abnormal type of each pixel pair in the abnormal connection state is determined; wherein, if there are no blank coordinates between the two pixels in the current pixel pair, the abnormal pixel coordinates are not repeated, and at least one of the above first coordinate offset and the second coordinate offset is greater than a preset offset threshold, the abnormal type of the pixel pair in the current abnormal connection state is determined to be pixel misalignment. Specifically, the anomaly type refers to the specific manifestation of the connection anomaly, including three types: pixel misalignment, pixel gap, and pixel overlap. If the abnormal pixel coordinates of the first boundary pixel and the second boundary pixel in the pixel pair in the set do not overlap, and there is no blank coordinate between them, but at least one coordinate offset is greater than the preset offset threshold (set to 1 in this invention), then the anomaly type is determined to be pixel misalignment.

[0089] If there is a blank coordinate between two pixels in the current pixel pair, the abnormal type of the pixel pair that is currently in an abnormal connection state is determined to be pixel gap. Specifically, if there is a blank coordinate position between the first boundary pixel and the second boundary pixel that is not occupied by any pixel, the anomaly type is determined to be a pixel gap.

[0090] If the abnormal pixel coordinates of two pixels in the current pixel pair are the same, the abnormal type of the pixel pair currently in an abnormal connection state is determined to be pixel overlap. Specifically, if the horizontal and vertical coordinates of the first boundary pixel and the second boundary pixel are exactly the same, i.e., the coordinates are repeated, then the anomaly type is determined to be pixel overlap.

[0091] Schematic representation: Construct a set to represent all pixel pairs in an anomalous connection state, such as (first boundary pixel (horizontal 300, vertical 402) and second boundary pixel (horizontal 298, vertical 402)), (first boundary pixel (horizontal 302, vertical 405) and second boundary pixel (horizontal 302, vertical 408)), (first boundary pixel (horizontal 305, vertical 410) and second boundary pixel (horizontal 305, vertical 410)), etc. Analyzing the first pixel pair, its first coordinate offset is 300-298=2, and its second coordinate offset is 0. The coordinates between the two (horizontal 299, vertical 402) are blank coordinates, and the anomaly type is determined to be pixel gap. Analyzing the second pixel pair, the first coordinate offset is 0, and the second coordinate offset is 408-402=6. There are no blank coordinates between the two, and at least one coordinate offset is greater than 1, so the anomaly type is pixel misalignment. Analyzing the third pixel pair, the horizontal and vertical coordinates of the first boundary pixel and the second boundary pixel are exactly the same, so the anomaly type is pixel overlap.

[0092] For all first abnormal pixels with the abnormality type of pixel misalignment, the coordinate deviation value of each first abnormal pixel is calculated according to the preset standard coordinates and the above-mentioned abnormal pixel coordinates. Then, the corresponding first abnormal pixel is translated at an equal distance according to each coordinate deviation value to generate the above-mentioned target defect composite image. Specifically, for three different anomaly types—pixel misalignment, pixel gap, and pixel overlap—corresponding pixel position adjustment or pixel information supplementation strategies are used to process boundary pixels with abnormal positional connections, thereby eliminating the connection anomalies between the region to be embedded and the background region, and obtaining a target defect synthetic image with a natural connection between the region to be embedded and the background region.

[0093] Specifically, if the anomaly type is pixel misalignment, then based on the coordinate deviation between the abnormal pixel coordinates and the standard coordinates of these boundary pixels, each boundary pixel belonging to the pixel misalignment is translated in the same direction at equal distances. Then, the translated pixels are stitched together to obtain the composite image of the target defect. Here, the standard coordinates refer to the correct pixel coordinates of the boundary pixels of the pixel misalignment, determined based on the normal stitching relationship, that is, the pixel coordinates that conform to the normal adjacency relationship between the region to be embedded and the background region.

[0094] Specifically, after determining the anomaly type as pixel misalignment, the standard coordinates are first determined: for the first boundary pixel belonging to the region to be embedded, the standard coordinates are the coordinates within the pixel coordinate range of the region to be embedded, and the pixel is directly adjacent to the corresponding background pixel (the horizontal coordinate differs by 1 or the vertical coordinate differs by 1); for the second boundary pixel belonging to the background region, the standard coordinates are the coordinates directly adjacent to the corresponding first boundary pixel. Next, the difference between the current horizontal coordinate value and the standard horizontal coordinate value (horizontal coordinate deviation value) and the difference between the current vertical coordinate value and the standard vertical coordinate value (vertical coordinate deviation value) of each boundary pixel are calculated. Then, the translation direction and translation distance are determined based on the coordinate deviation values. The translation direction is the direction to eliminate the deviation: if the horizontal coordinate deviation value is positive, the horizontal direction is shifted to the left; if the horizontal coordinate deviation value is negative, the horizontal direction is shifted to the right; if the vertical coordinate deviation value is positive, the vertical direction is shifted upwards; if the vertical coordinate deviation value is negative, the vertical direction is shifted downwards; the translation distance is the absolute value of the coordinate deviation value. Finally, perform equidistant translation on all first anomalous pixels in the same direction, moving each first anomalous pixel from its current coordinates to the standard coordinates, so that the region to be embedded and the background region are properly connected, and the target defect synthetic image is obtained.

[0095] Indicatively, after determining the anomaly type to be pixel misalignment, all first-abnormal pixels are extracted, including the first boundary pixel (horizontal 302, vertical 405) and the corresponding second boundary pixel (horizontal 302, vertical 408). Simultaneously, the pixel coordinate range of the region to be embedded (horizontal 300 to 500, vertical 400 to 600) is retrieved. Based on the pixel coordinate range and normal connection relationships, standard coordinates are determined: the standard vertical coordinate of the first boundary pixel (horizontal 302, vertical 405) should be 407 (normally adjacent to the second boundary pixel), with standard coordinates of (horizontal 302, vertical 407); the standard vertical coordinate of the second boundary pixel (horizontal 302, vertical 408) should be 406 (normally adjacent to the first boundary pixel), with standard coordinates of (horizontal 302, vertical 406). Calculate the coordinate deviation values: The horizontal coordinate deviation of the first boundary pixel is 302-302=0, and the vertical coordinate deviation is 405-407=-2; the horizontal coordinate deviation of the second boundary pixel is 302-302=0, and the vertical coordinate deviation is 408-406=2. Determine the translation direction and distance: The vertical coordinate deviation of the first boundary pixel is -2, so it is translated downwards by 2 pixels; the vertical coordinate deviation of the second boundary pixel is 2, so it is translated upwards by 2 pixels. The translation distance for both is 2 pixels, and the translation direction is the same to eliminate deviation (the direction of mutual approach). Perform translation on the two pixels: The first boundary pixel is translated from (horizontal 302, vertical 405) to (horizontal 302, vertical 407), and the second boundary pixel is translated from (horizontal 302, vertical 408) to (horizontal 302, vertical 406). The area to be embedded and the background area are normally connected, resulting in the synthesized image of the target defect.

[0096] For all second abnormal pixels with the abnormality type of pixel gap, obtain the abnormal texture features of each second abnormal pixel and the normal texture features of the normal boundary pixels adjacent to each second abnormal pixel; based on the above abnormal texture features and normal texture features, perform pixel filling processing on the corresponding gap region to generate the above target defect synthetic image. Specifically, if the anomaly type is a pixel gap, the filling pixel texture information for the gap region is determined based on the texture features of all second-abnormal pixels and adjacent normal boundary pixels. Then, pixels with textures consistent with adjacent pixels are selected based on this filling pixel texture information, and pixel filling is performed on the gap region to obtain the synthesized image of the target defect. Here, the gap region refers to the area formed by the blank coordinate positions between the first and second boundary pixels in the corresponding pixel pair that are not occupied by any pixels. Adjacent normal boundary pixels refer to the boundary pixels surrounding all second-abnormal pixels that are in a normal connection state. Texture features refer to the visual texture attributes of pixels, including the pixel's color grayscale value, texture direction, texture density, and other feature information that reflects the pixel's appearance texture. Filling pixel texture information refers to the texture feature information that the pixels used to fill the gap region should possess, and this information is consistent with the texture features of adjacent normal boundary pixels. Pixel filling refers to the process of supplementing the blank coordinate positions of the gap region with pixels possessing filling pixel texture information.

[0097] Specifically, after determining the anomaly type to be a pixel gap, the blank coordinate range of the gap region is defined. Next, all adjacent normal boundary pixels surrounding the second anomalous pixel are searched, and the texture features of these adjacent normal boundary pixels are extracted, including color grayscale values, texture direction, and texture density. Then, the extracted texture features of the adjacent normal boundary pixels are statistically analyzed, calculating the average value of the texture features (such as the average color grayscale value), and determining the mainstream features of texture direction and texture density. These statistically obtained feature information are used as the filling pixel texture information for the gap region. Subsequently, based on the filling pixel texture information, pixels with this texture information are selected from a preset texture pixel library, or pixels with this texture information are generated through interpolation of the texture features of adjacent normal boundary pixels; the interpolation method is to take the average of the texture feature values ​​of two adjacent normal boundary pixels as the texture feature value of the filling pixel. The selected or generated pixels are then used to fill the gap region one by one according to the blank coordinate range of the gap region. After filling, the region to be embedded and the background region can be normally connected, resulting in the synthesized image of the target defect.

[0098] Indicatively, after determining the anomaly type to be a pixel gap, a corresponding set of pixel pairs is generated, containing the first boundary pixel (horizontal 300, vertical 402), the second boundary pixel (horizontal 298, vertical 402), and the gap region between them (blank coordinates (horizontal 299, vertical 402)). Adjacent normal boundary pixels are found: (horizontal 300, vertical 401), (horizontal 300, vertical 403), (horizontal 298, vertical 401), and (horizontal 298, vertical 403). Texture features of these pixels are extracted: color grayscale values ​​are 183, 181, 180, and 178 respectively; the texture direction is horizontal; and the texture density is one texture line per 10 pixels. Statistical analysis of the texture features shows that the average color grayscale value is (183+181+180+178) / 4=180.5, rounded to the integer 181. The main texture direction is horizontal, and the main texture density is one texture line per 10 pixels. These are determined as the filling pixel texture information. Filler pixels are generated through interpolation. The grayscale values ​​of adjacent normal boundary pixels (horizontal 300, vertical 402) and (horizontal 298, vertical 402) are taken as 183 and 180, respectively. The average value is calculated as (183+180) / 2=181.5, rounded to the integer 181. The texture direction and density are consistent with the adjacent normal pixels. The generated filler pixels are then used to fill the blank coordinates (horizontal 299, vertical 402) in the gap region, so that the region to be embedded and the background region are properly connected, resulting in the synthesized image of the target defect.

[0099] For all third-order anomalous pixels with pixel overlap as the anomalous type, redundant pixel removal is performed on all third-order anomalous pixels to generate the above-mentioned target defect synthetic image.

[0100] Specifically, if the anomaly type is pixel overlap, then based on the preset judgment conditions of all third-abnormal pixels and the target pixels in the defect localization mask, redundant pixels in the overlapping region are identified. Redundant pixel removal is then performed on the overlapping region based on the coordinate information of these redundant pixels to obtain the composite image of the target defect. Here, the overlapping region refers to the area where multiple pixels among all third-abnormal pixels have identical coordinates. The preset judgment conditions refer to the pre-defined criteria used to determine whether a third-abnormal pixel belongs to the region to be embedded. This criterion is based on the coordinate range of the target pixels in the defect localization mask; if the coordinates of the third-abnormal pixel are within the coordinate range of the target pixels, it is determined to be a pixel belonging to the region to be embedded; otherwise, it is determined to be a pixel belonging to the background region. Redundant pixels refer to pixels in the overlapping region that do not belong to the preset region corresponding to their coordinates—that is, pixels within the target pixel coordinate range but belonging to the background region, or pixels not within the target pixel coordinate range but belonging to the region to be embedded. Redundant pixel removal refers to the process of deleting redundant pixels in the overlapping region, retaining only pixels that meet the preset judgment conditions.

[0101] Specifically, after determining the anomaly type to be pixel overlap, the coordinates of the overlapping area are first identified. Next, the coordinate range of the target pixel in the defect localization mask is obtained, and a preset judgment condition is determined: if the pixel coordinates of the third abnormal pixel are within the target pixel coordinate range, it should belong to the region to be embedded; otherwise, if they are not within the target pixel coordinate range, they should belong to the background region. Then, based on the preset judgment condition, normal pixels that meet the condition and redundant pixels that do not meet the condition are distinguished from the overlapping area: if the coordinates of the third abnormal pixel are within the target pixel coordinate range but belong to the background region, it is a redundant pixel; if the coordinates of the third abnormal pixel are not within the target pixel coordinate range but belong to the region to be embedded, it is also a redundant pixel. Finally, all redundant pixels are deleted from the overlapping area to generate a composite image of the target defect.

[0102] In a schematic manner, after determining the anomaly type as pixel overlap, the relevant information set of all third-abnormal pixels is extracted, including the first boundary pixel (belonging to the region to be embedded) and the second boundary pixel (belonging to the background region) at the coordinates (horizontal 305, vertical 410) of the overlapping area. Simultaneously, the coordinate range of the target pixel in the defect localization mask (horizontal 300 to 500, vertical 400 to 600) is called to determine the preset judgment conditions: pixels with coordinates within the range of 300 to 500 horizontal coordinates and 400 to 600 vertical coordinates should be first boundary pixels; otherwise, they should be second boundary pixels. The pixels in the overlapping area are judged: if the coordinates (horizontal 305, vertical 410) of the overlapping area are within the target pixel coordinate range, the first boundary pixel meets the preset judgment conditions and is a normal pixel; the second boundary pixel is within this coordinate range but belongs to the background region and is a redundant pixel. The second boundary pixel is removed from the overlapping area, leaving only the first boundary pixel, so that the region to be embedded and the background region are properly connected, resulting in the synthesized image of the target defect.

[0103] Preferably, the present invention can accurately connect the preliminary defect synthesis image, so that the region to be embedded in the generated target defect synthesis image is naturally connected with the background region, the defect shape is complete and has a high degree of integration with the scene. After edge fusion, a high-quality anti-defect sample image is formed, which provides high-quality targeted training data for the intelligent detection model and improves the accuracy of the intelligent detection model in recognizing small targets and defects in specific locations.

[0104] Preferably, by achieving normal connection between the region to be embedded and the background region, the present invention ensures that the defect region in the target defect synthesis image has a complete shape and a high degree of integration with the background scene, without any abrupt connection or structural defects. This ensures that the final generated anti-defect sample image can realistically simulate the actual defect scene, providing high-quality targeted training data for the intelligent detection model, thereby improving the accuracy of the intelligent detection model in recognizing small targets and defects in specific locations.

[0105] In this preferred embodiment, a composite image of the target defect is obtained by performing abnormal repair on all pixel pairs in an abnormal connection state.

[0106] In another preferred embodiment, the above-mentioned edge fusion of the embedded boundaries in the target defect synthetic image based on the above-mentioned defect description text to obtain the anti-defect sample image includes: From the above synthesized image of the target defect, determine the defect boundary pixels between the defect area and the non-defect area; Specifically, in the synthesized image of the target defect, all pixels at the edge of the region (i.e., the defect area) defined by the defect localization mask (i.e., the defect boundary pixels) constitute the initial boundary between the defect area and the surrounding area. The defect area is determined as follows: all pixel coordinates of the synthesized image of the target defect are obtained; these coordinates are matched with the target pixels of the defect localization mask to determine the pixel region in the synthesized image corresponding to the target pixels; this region is the defect area defined by the defect localization mask.

[0107] Specifically, a full pixel traversal scan is performed on all pixels within the defect area. Starting from the top left pixel of the defect area, each pixel is checked sequentially from left to right and from top to bottom. During the traversal scan, it is determined whether each pixel has at least one adjacent pixel that does not belong to the defect area (adjacent pixels refer to pixels whose coordinates differ by 1 in the horizontal or vertical direction). If such a pixel exists, it is considered the defect boundary pixel of the defect area, and the specific coordinates of all defect boundary pixels are recorded.

[0108] To illustrate, the target pixels (pixel value 255) used to define the area to be embedded in the defect localization mask are obtained, and the coordinate set of the target pixels is extracted as all pixel coordinates from horizontal 300 to 500 and vertical 400 to 600. At the same time, all pixel coordinates (including all pixel coordinates from horizontal 0 to 1023 and vertical 0 to 1023) of the target defect composite image (1024*1024 pixels, a composite image of the crack defect in the pole body) are obtained, and matched with the coordinate set of the target pixels to determine that the area from horizontal 300 to 500 and vertical 400 to 600 in the target defect composite image is the crack defect area defined by the defect localization mask. A full pixel traversal scan is performed on the crack defect area, checking whether each pixel's adjacent pixels belong to the defect area. For example, if pixel coordinates (horizontal 300, vertical 400) has a left-hand adjacent pixel (horizontal 299, vertical 400) that does not belong to the defect area, then this pixel is a defect boundary pixel. If pixel coordinates (horizontal 350, vertical 450) has four adjacent pixels that belong to the defect area, then it is not a defect boundary pixel. If pixel coordinates (horizontal 500, vertical 500) has a right-hand adjacent pixel (horizontal 501, vertical 500) that does not belong to the defect area, then it is a defect boundary pixel. The coordinates of all such pixels are recorded to form a set of defect boundary pixels, such as (horizontal 300, vertical 400), (horizontal 301, vertical 400)...(horizontal 500, vertical 400), (horizontal 300, vertical 401), (horizontal 500, vertical 401)...etc.

[0109] Extend a predetermined number of pixels outward from the aforementioned defect boundary pixels to obtain the edge neighborhood range; Specifically, the preset number of pixels refers to the pre-defined number of extended pixels used to form the edge neighborhood. This number is determined based on the resolution and defect type of the synthesized image of the target defect. The higher the resolution and the smaller the defect size, the more appropriate the preset number can be, generally ranging from 3 to 5 pixels. For example, for small target defects such as cracks in utility poles and images with a resolution of 1024*1024, the preset number is set to 3 pixels. The edge neighborhood range refers to the range of all pixel coordinates covered by extending the preset number of pixels outward from the defect boundary pixels. This range includes the defect boundary pixels and the pixels in the extended area.

[0110] Specifically, using the coordinates of each defect boundary pixel as a reference, a predetermined number of pixels are extended outwards from the defect region to determine the extended pixel coordinate range corresponding to each defect boundary pixel: for horizontal defect boundary pixels (such as pixels on the left edge of the defect region), the range is extended outwards horizontally by a predetermined number of pixels; for vertical defect boundary pixels (such as pixels on the upper edge of the defect region), the range is extended outwards vertically by a predetermined number of pixels; for corner defect boundary pixels (such as pixels in the upper left corner of the defect region), the range is extended outwards both horizontally and vertically by a predetermined number of pixels. Finally, the extended pixel coordinate ranges corresponding to all defect boundary pixels and the coordinates of the defect boundary pixels themselves are integrated, and duplicate coordinates are removed to obtain the edge neighborhood range.

[0111] This example illustrates the extraction of pixel coordinates for each defect boundary, such as (horizontal 300, vertical 400), (horizontal 500, vertical 450), (horizontal 380, vertical 600), etc. The preset extension pixel count is 3 pixels (set for crack defects on utility poles and 1024*1024 resolution images). Taking the defect boundary pixel (horizontal 300, vertical 400) (the left edge pixel of the defect area) as an example, it extends 3 pixels horizontally outward (to the left), with corresponding extension coordinates ranging from horizontal 297 to 299 and vertical 400; taking the defect boundary pixel (horizontal 500, vertical 450) (the right edge pixel of the defect area) as an example, it extends 3 pixels horizontally outward (to the right), with extension coordinates ranging from horizontal 501 to 503 and vertical 450; taking the defect boundary pixel (horizontal 380, vertical 600) (the lower edge pixel of the defect area) as an example, it extends 3 pixels vertically outward (downward), with extension coordinates ranging from horizontal 380 and vertical 601 to 603; taking the defect boundary pixel (horizontal 300, vertical 600) (the lower left edge pixel of the defect area) as an example, it extends 3 pixels horizontally to the left (horizontal 297 to 299) and vertically downward (vertical 601 to 603), with extension coordinates ranging from horizontal 297 to 299 and vertical 601 to 603. By integrating all extended coordinate ranges and defect boundary pixel coordinates and removing duplicate coordinates, the edge neighborhood range is obtained, such as a portion of the pixel area within the horizontal range of 297 to 503 and the vertical range of 397 to 603.

[0112] Based on the above defect description text, edge texture analysis is performed on the defect to be generated to determine the edge texture features of the defect to be generated. Extract the visual features of all edge neighborhood pixels within the aforementioned edge neighborhood range, and compare the visual features of all edge neighborhood pixels one by one with the aforementioned edge texture features to determine the coordinates of edge neighborhood pixels that conform to the aforementioned edge texture features, thereby obtaining a texture-related pixel set. Specifically, edge texture analysis refers to the process of analyzing and determining the texture features that the edges of the defect to be generated should possess by combining the defect appearance information in the defect description text and the visual features of the defect area. The texture-related pixel set refers to the set of pixels within the edge neighborhood where the visual features are related to the texture features of the defect edge (i.e., the visual features can form a smooth transition with the texture features of the defect edge).

[0113] Specifically, defect appearance information is extracted from the defect description text. Defect appearance information refers to descriptive information related to the defect edges, such as "irregular edges, darker color than the rod body, and linear extension" for crack defects. Then, visual features of the area defined by the defect location mask are extracted, including the color grayscale value, texture direction, texture density, brightness value, and contrast of the pixels in the area. Statistical values ​​of these visual features (such as the average, maximum, and minimum values ​​of the color grayscale value) are calculated.

[0114] Then, by combining the extracted defect appearance information and the visual features of the defect area, edge texture analysis is performed to determine the edge texture features corresponding to the defect to be generated as a countermeasure: taking the crack defect as an example, by combining the "linearly extending crack" in the defect description text and the "average color gray value 181, horizontal texture direction" of the visual features of the defect area, the edge texture features are determined to be "horizontal texture direction, color gray value 179 to 183, brightness value 181 to 185, and uniform texture density".

[0115] Then, all pixels within the edge neighborhood are traversed, the visual features of each pixel are extracted, and they are matched with the determined defect edge texture features. If the visual features of a pixel are within the range of the edge texture features, the pixel is determined to be a texture-associated pixel associated with the defect texture. The coordinates of all texture-associated pixels are recorded to form a texture-associated pixel set.

[0116] Schematic illustration: The defect description text "pole body crack defect, crack location is in the lower middle area of ​​the pole" is obtained, and the defect appearance information is extracted as "pole body crack, linearly extending crack, color darker than the concrete body of the pole". Visual features of the crack defect area defined by the defect location mask are extracted: average grayscale value 181, maximum 185, minimum 178, horizontal texture direction, average brightness value 183, contrast 75. Combining the defect appearance information and the visual features of the defect area, edge texture analysis is performed, determining the edge texture features of the pole body crack defect to be generated as "horizontal texture direction, grayscale value 179 to 183, brightness value 181 to 185, uniform texture density (1 texture line per 10 pixels)". Traverse all pixels within the edge neighborhood (e.g., a portion of the area between horizontal 297 and 503, and vertical 397 and 603) and extract the visual features of each pixel: for example, a pixel with coordinates (horizontal 299, vertical 400) has a grayscale value of 182, a horizontal texture direction, and a brightness value of 184, all of which fall within the range of edge texture features and is therefore identified as a texture-related pixel; a pixel with coordinates (horizontal 297, vertical 400) has a grayscale value of 175 and a vertical texture direction, which does not meet the edge texture features and is therefore not identified as a texture-related pixel; a pixel with coordinates (horizontal 502, vertical 450) has a grayscale value of 181, a horizontal texture direction, and a brightness value of 183 and is identified as a texture-related pixel. Record the coordinates of all pixels that meet the conditions to form a set of texture-related pixels.

[0117] Smooth transition processing is performed on the edge neighbor pixels in the texture-associated pixel set and the edge pixels in the defect region to obtain the preliminary fused edge neighbor pixel information. Specifically, the pixel information of texture-related pixels and edge pixels of defective regions undergoes smoothing transition processing to achieve a visually natural connection between the edge of the defective region and the texture-related pixels in its neighborhood, eliminating any abruptness between them. The edge neighborhood pixel information after preliminary fusion refers to the visual characteristics of all pixels within the edge neighborhood after smoothing transition processing, including color grayscale values, brightness values, etc.

[0118] Specifically, firstly, each defect boundary pixel is processed individually to extract its specific coordinates. Using these coordinates as a reference, an analysis sub-range is defined within the edge neighborhood. This sub-range is the coordinate area extended outwards from the defect boundary pixel coordinates by a predetermined number of pixels (consistent with the edge neighborhood extension, which is 3 pixels). Next, within this sub-range, pixels in the texture-related pixel set are searched, and their positional relationship with the current defect boundary pixel is determined: if the texture-related pixel and the defect boundary pixel differ by 1 in horizontal coordinate but have the same vertical coordinate, or differ by 1 in vertical coordinate but have the same horizontal coordinate, they are considered directly adjacent; if they are in the same extension direction (horizontal or vertical) and their coordinates are 2 or 3 pixels apart, they are considered indirectly adjacent. Finally, the texture-related pixels corresponding to each defect boundary pixel and their specific positional correspondences are recorded, forming a positional correspondence set.

[0119] This illustration shows the set of defect boundary pixels (e.g., (horizontal 300, vertical 400), (horizontal 500, vertical 450) etc.), the set of texture-related pixels (e.g., (horizontal 299, vertical 400), (horizontal 298, vertical 400), (horizontal 501, vertical 450) etc.), and the edge neighborhood range (horizontal 297 to 503, vertical 397 to 603). Taking the defect boundary pixel (horizontal 300, vertical 400) (the left edge pixel of the defect region) as an example, its coordinates are extracted and the analysis sub-range is defined as horizontal 297 to 300, vertical 400 (extending 3 pixels to the left). Within this sub-range, texture-related pixels are searched, and (horizontal 299, vertical 400) and (horizontal 298, vertical 400) are found: the horizontal coordinates of (horizontal 299, vertical 400) and (horizontal 300, vertical 400) differ by 1, thus establishing a direct adjacent relationship; the horizontal coordinates of (horizontal 298, vertical 400) and (horizontal 300, vertical 400) are separated by 2 pixels and are in the same horizontal extension direction, thus establishing an indirect adjacent relationship. Taking the defect boundary pixel (horizontal 500, vertical 450) (the right edge pixel of the defect area) as an example, the sub-range is analyzed as horizontal 500 to 503 and vertical 450, and texture-related pixels (horizontal 501, vertical 450) are found. The horizontal coordinates of the two pixels differ by 1, thus establishing a direct adjacent relationship.

[0120] Subsequently, the positional correspondences are categorized according to the extension direction (horizontal or vertical) of the defect boundary pixels, into two types: horizontal correspondences and vertical correspondences. Next, for each defect boundary pixel in each type of correspondence, starting from that defect boundary pixel, all its corresponding texture-related pixels are included in the same region along the extension direction of the correspondence: for horizontal correspondences, the region is a horizontal coordinate range extending 3 pixels outward from the defect boundary pixel under the same vertical coordinate; for vertical correspondences, the region is a vertical coordinate range extending 3 pixels outward from the defect boundary pixel under the same horizontal coordinate. Finally, each region is defined as a boundary fusion sub-region, each containing 1 defect boundary pixel (first quantity) and 3 texture-related pixels (second quantity), ensuring that the sub-regions are uniformly sized and independent, forming a set of boundary fusion sub-regions.

[0121] Schematic, the horizontal correspondence (e.g., the correspondence between (horizontal 300, vertical 400) and surrounding texture-related pixels) and the vertical correspondence are categorized. For the defect boundary pixel (horizontal 300, vertical 400), taking it as the starting point, extend horizontally to the left by 3 pixels, defining a region ranging from horizontal 297 to 300 and vertical 400. This region contains one defect boundary pixel (horizontal 300, vertical 400) and three texture-related pixels (horizontal 299, vertical 400), (horizontal 298, vertical 400), and (horizontal 297, vertical 400), and is defined as boundary fusion sub-region 1. For the defect boundary pixel (horizontal 500, vertical 450), extend 3 pixels horizontally to the right, the region ranges from horizontal 500 to 503 and vertical 450, including 1 defect boundary pixel (horizontal 500, vertical 450) and 3 texture-related pixels (horizontal 501, vertical 450), (horizontal 502, vertical 450), and (horizontal 503, vertical 450), and is determined as boundary fusion sub-region 2. Number the two sub-regions and record the coordinate range and pixel points to form a boundary fusion sub-region set.

[0122] Then, the first texture information of the defect boundary pixels and the second texture information of all texture-related pixels within each boundary fusion sub-region are extracted. Next, the texture matching degree is calculated: for color grayscale values, the absolute value of the difference between the color grayscale value in the first texture information and the color grayscale value in each second texture information is calculated, and the average of all absolute differences is taken; for texture direction, it is determined whether it is consistent with the texture direction of the first texture information; if the average difference of color grayscale values ​​is less than or equal to 5, and the texture direction of all texture-related pixels is consistent with the defect boundary pixels, it is determined to be a high matching degree; if the average difference of color grayscale values ​​is greater than 5, or there are texture-related pixels whose texture direction is inconsistent with the defect boundary pixels, it is determined to be a low matching degree, thus obtaining the texture matching degree determination result for each sub-region. Finally, the region fusion method is determined based on the determination result: if the determination result is a high matching degree, the region fusion method is texture sequential fusion; if the determination result is a low matching degree, the region fusion method is texture transition fusion, and the region fusion method corresponding to each boundary fusion sub-region is recorded.

[0123] Indicatively, the texture information of the boundary fusion sub-region 1 (horizontal 297 to 300, vertical 400) is extracted: the first texture information of the defect boundary pixel (horizontal 300, vertical 400) is a grayscale value of 181, a horizontal texture direction, and a brightness value of 183; the second texture information of the texture-related pixels (horizontal 299, vertical 400), (horizontal 298, vertical 400), and (horizontal 297, vertical 400) are grayscale values ​​of 182, 180, and 179 respectively, all with a horizontal texture direction and brightness values ​​of 184, 182, and 181. The average difference of the grayscale values ​​is calculated as: (|181-182|+|181-180|+|181-179|) / 3=(1+1+2) / 3=1.33, which is less than 5, and the texture direction is consistent, so it is judged as a high matching degree, and the region fusion method is texture continuation fusion. Texture information of boundary fusion sub-region 2 (horizontal 500 to 503, vertical 450) is extracted: the defect boundary pixel (horizontal 500, vertical 450) has a grayscale value of 180 and a horizontal texture direction; the texture-related pixel (horizontal 501, vertical 450) has a grayscale value of 173 and a horizontal texture direction, and (horizontal 502, vertical 450) has a grayscale value of 172 and a vertical texture direction. The average difference in grayscale value is (7+8) / 2=7.5, which is greater than 5, and there are pixels with inconsistent texture directions. Therefore, it is judged as low matching degree, and the region fusion method is texture transition fusion.

[0124] Finally, after determining that the region fusion method is texture continuation fusion, the texture continuation direction of the corresponding boundary fusion sub-region is determined, and it is determined to be horizontal or vertical based on the extension direction of the sub-region. Next, the first texture information of the defect boundary pixels and the second texture information of texture-related pixels within the sub-region are extracted. The changing trend of texture features is analyzed to determine the continuation pixel arrangement rules: for color grayscale values ​​and brightness values, the original gradient trend is maintained along the continuation direction, and the feature difference between adjacent pixels does not exceed 2; for texture direction, it is kept consistent with the texture direction of the defect boundary pixels. Third, according to the continuation direction and arrangement rules, the pixel information within the boundary fusion sub-region is adjusted: if the original texture feature changes are discontinuous, the feature values ​​of the intermediate pixels are adjusted so that the feature values ​​gradually change along the continuation direction. Finally, the pixel texture continuation of the sub-region is completed, and the fused pixel information of all sub-regions using texture continuation fusion is integrated to form partially fused edge neighborhood pixel information.

[0125] Indicatively, the fusion method for boundary fusion sub-region 1 is determined to be texture extension fusion. Since the extension direction of this sub-region is horizontal, the texture extension direction is horizontal to the left. Texture features of pixels within the sub-region are extracted: defect boundary pixels (horizontal 300, vertical 400) have a grayscale value of 181 and a brightness value of 183; texture-related pixels (horizontal 299, vertical 400) have values ​​of 182 and 184, (horizontal 298, vertical 400) have values ​​of 180 and 182, and (horizontal 297, vertical 400) have values ​​of 179 and 181. Analysis shows that the grayscale value gradually decreases from 181 to the left, and the brightness value gradually decreases from 183 to the left. The rule for extending pixels is determined to be that the difference in grayscale value between adjacent pixels does not exceed 2, the difference in brightness value does not exceed 2, and the texture direction remains horizontal. The existing feature changes are found to conform to the rule, requiring no adjustment; texture extension is directly performed according to the existing features. After completing the fusion of sub-region 1, the preliminary fusion information of the edge neighbor pixels of this region is obtained.

[0126] Specifically, after determining that the region fusion method is texture transition fusion, the first texture information of the defect boundary pixels and the second texture information of all texture-related pixels within the corresponding boundary fusion sub-region are extracted. Next, intermediate pixel features are calculated (i.e., the texture features that transition pixels located between defect boundary pixels and texture-related pixels in the transition texture should possess, including color grayscale values, brightness values, texture direction, etc., which are determined based on the average value or gradient range of the first and second texture information; the transition texture refers to the transitional texture used to connect defect boundary pixels and texture-related pixels, and its features are between the two): the color grayscale value is taken as the average of the color grayscale values ​​in the first and second texture information, the brightness value is taken as the average of the brightness values ​​of the two, and the texture direction is adjusted to be consistent with the direction of the defect boundary pixels; if there are multiple texture-related pixels, multiple transition intervals are divided according to the sequential direction, and each interval corresponds to a set of intermediate pixel features.

[0127] Subsequently, at the gap positions between defect boundary pixels and texture-related pixels within the sub-region, transition pixels with intermediate pixel features are added; simultaneously, the inconsistent features in the original texture-related pixels are adjusted to connect with the transition pixel features. This completes the pixel-texture transition for the sub-region. By integrating the fused pixel information of all sub-regions that have undergone texture transition fusion, the corresponding preliminary fused edge neighborhood pixel information can be obtained.

[0128] Indicatively, the fusion method of the boundary fusion sub-region 2 is determined to be texture transition fusion. The texture information of this sub-region (horizontal 500 to 503, vertical 450) is extracted: the color grayscale value of the defect boundary pixel (horizontal 500, vertical 450) is 180, the texture direction is horizontal, and the brightness value is 182; the color grayscale value of the texture-related pixel (horizontal 501, vertical 450) is 173, the texture direction is horizontal, and the color grayscale value of (horizontal 502, vertical 450) is 172, the texture direction is vertical. Calculate the intermediate pixel features: The intermediate color grayscale value in the horizontal range of 500 to 501 is (180+173) / 2=176.5, rounded to 177; the brightness value is (182+175) / 2=178.5, rounded to 179; the texture direction is horizontal. The intermediate color grayscale value in the horizontal range of 501 to 502 is (173+172) / 2=172.5, rounded to 173; the brightness value is (175+174) / 2=174.5, rounded to 175; the texture direction is horizontal. Add transition pixels to the sub-region: At horizontal position 500.5 and vertical position 450, add transition pixels with a grayscale value of 177 and a brightness value of 179; adjust the texture direction at (horizontal position 502, vertical position 450) to horizontal. After completing the texture transition fusion, the preliminary fused pixel information for this sub-region is obtained.

[0129] Obtain the defect pixel information of the aforementioned defect area, as well as the background pixel information of the background area excluding the aforementioned defect area and edge neighborhood; The above defect pixel information, background pixel information, and edge neighborhood pixel information are used to perform pixel information fusion transition to obtain the above-mentioned anti-measure defect sample image.

[0130] Specifically, the process involves acquiring the edge neighborhood pixel information after initial fusion, the defect pixel information of the region defined by the defect localization mask, and the background pixel information of the target defect composite image. Secondly, the pixel coordinate ranges of the three regions are determined to clarify the coverage of each region and avoid pixel coordinate overlap or omission: the coordinate range of the defect region (i.e., the region defined by the aforementioned defect localization mask) is the coordinate range of the target pixels in the defect localization mask; the range corresponding to the edge neighborhood pixel information is the aforementioned edge neighborhood range; and the coordinate range of the background region is the remaining coordinate range of all pixel coordinates in the target defect composite image, excluding the defect region and edge neighborhood ranges.

[0131] Specifically, the pixel information of the three regions is integrated according to coordinate range: for non-edge pixels within the defect region, their original pixel information is retained as the aforementioned defect pixel information; for pixels within the edge neighborhood range, the fused edge neighborhood pixel information is used; for pixels in the background region, their original background pixel information is retained as the aforementioned background pixel information. Finally, all the integrated pixel information is combined according to the corresponding coordinate range to obtain the reverse defect sample image.

[0132] The diagram illustrates the following: The fused edge neighborhood pixel information, such as the color grayscale values ​​of 179 to 188 and brightness values ​​of 181 to 187 for pixels within the horizontal range of 297 to 503 and the vertical range of 397 to 603; the defect pixel information of the crack defect area defined by the defect localization mask, such as the color grayscale values ​​of 178 to 183 and brightness values ​​of 183 for non-edge pixels within the horizontal range of 300 to 500 and the vertical range of 400 to 600; and the background pixel information of the target defect composite image, such as the color grayscale values ​​of 180 to 190 and brightness values ​​of 185 for pixels outside the horizontal range of 0 to 296, 297 to 503, and 504 to 1023, and outside the vertical range of 0 to 396, 397 to 603, and 604 to 1023. The coordinate ranges of each region are determined as follows: the defect region is defined as horizontally 300 to 500 and vertically 400 to 600; the edge neighborhood range is horizontally 297 to 503 and vertically 397 to 603; the background region is the remaining coordinate range in the target defect composite image excluding the above two regions. Pixel information is integrated according to the coordinate range: non-edge pixels within the defect region (e.g., horizontally 320, vertically 450) retain the original crack texture pixel information; pixels within the edge neighborhood (e.g., horizontally 299, vertically 400) use the fused edge neighborhood pixel information; and background region pixels (e.g., horizontally 200, vertically 300) retain the original utility pole background pixel information, resulting in a reverse defect sample image.

[0133] Preferably, the present invention can accurately complete the smooth transition processing, and the obtained edge neighborhood pixel information after preliminary fusion has good smoothness and consistency. After being integrated with the defect area pixel information and background pixel information, it can generate a highly realistic anti-defect sample image with natural connection between the defect and the background, providing high-quality training data for the intelligent detection model, thereby improving the model's recognition performance for small targets and defects in specific locations.

[0134] Preferably, the present invention can accurately complete the edge fusion of the defect area and the background area, and the generated reverse defect sample image can realistically simulate the actual power equipment defect scenario, making up for the problem of the lack of actual defect samples, providing high-quality targeted training data for the intelligent detection model, thereby improving the accuracy of the intelligent detection model in identifying small targets and defects in specific locations.

[0135] In this preferred embodiment, edge fusion is performed on the embedded boundaries in the target defect synthetic image based on the defect description text to obtain the anti-defect defect sample image.

[0136] Based on the above method embodiments, the present invention provides corresponding apparatus embodiments.

[0137] like Figure 2 As shown, an embodiment of the present invention provides a device for generating defect samples of countermeasures based on image-text linkage, comprising: The system includes modules for image and text data acquisition, text semantic parsing, real defect location feature matching, defect spatial range localization, local scene image generation, defect texture image generation, and countermeasure defect sample image generation. The aforementioned image and text data acquisition module is used to acquire target scene images of power equipment in normal condition, as well as defect description text; The aforementioned text semantic parsing module is used to perform semantic parsing on the aforementioned defect description text to obtain the defect type, defect location, and appearance features of the defect to be generated as a countermeasure. The aforementioned real defect location feature matching module is used to match the real defect location features corresponding to the aforementioned defect type from a preset knowledge base based on the aforementioned defect type; wherein, the aforementioned preset knowledge base is constructed based on the historical defect status of each power equipment; The aforementioned defect spatial range localization module is used to locate the defect spatial range of the defect to be generated in the target scene image based on the defect location and the actual defect location features, and to generate a defect localization mask based on the defect spatial range. The aforementioned local scene image generation module is used to extract a local scene image representing the spatial range of the defect from the aforementioned target scene image based on the aforementioned defect localization mask. The aforementioned defect texture image generation module is used to extract corresponding visual features from the aforementioned local scene image, and adjust the visual parameters of the defect to be generated according to the aforementioned visual features so that the aforementioned visual parameters are consistent with the aforementioned visual features, and generate a defect texture image according to the adjusted visual parameters and appearance features. The aforementioned defect sample image generation module is used to perform image embedding and edge pixel fusion on the aforementioned target scene image based on the aforementioned defect texture image to obtain the defect sample image.

[0138] It should be noted that the device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without creative effort. The above schematic diagram is merely an example of a device for generating countermeasure defect samples based on image-text linkage and does not constitute a limitation on a device for generating countermeasure defect samples based on image-text linkage. It may include more or fewer components than shown, or combine certain components, or use different components.

[0139] Based on the above method embodiments, the present invention provides corresponding terminal device embodiments.

[0140] Another embodiment of the present invention provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the above-described method for generating countermeasure defect samples based on image-text linkage in any embodiment of the present invention.

[0141] For example, in this embodiment, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the device. The aforementioned terminal devices may be computing devices such as desktop computers, laptops, handheld computers, and cloud servers. These devices may include, but are not limited to, processors and memory. The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. This processor is the control center of the device, connecting various parts of the device via various interfaces and lines. The aforementioned memory can be used to store the aforementioned computer programs and / or modules. The aforementioned processor implements various functions of the aforementioned device by running or executing the computer programs and / or modules stored in the aforementioned memory, and by calling data stored in the memory. The aforementioned memory may mainly include a program storage area and a data storage area, wherein the program storage area may store the operating system, at least one application program required for a function, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0142] Based on the above method embodiments, the present invention provides corresponding storage medium embodiments.

[0143] Another embodiment of the present invention provides a storage medium including a stored computer program, wherein, when the computer program is running, it controls the device where the storage medium is located to execute the above-described method for generating countermeasure defect samples based on image-text linkage according to any embodiment of the present invention.

[0144] In this embodiment, the storage medium is a computer-readable storage medium, and the computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0145] The above are preferred embodiments of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for generating defect samples for countermeasures based on image-text linkage, characterized in that, include: Acquire target scene images of power equipment in normal condition, along with defect description text; Semantic parsing is performed on the defect description text to obtain the defect type, defect location, and appearance features of the defect to be generated as a countermeasure. Based on the defect type, the actual defect location features corresponding to the defect type are matched from a preset knowledge base; wherein, the preset knowledge base is constructed based on the historical defect status of each power equipment; Based on the defect location and the actual defect location features, the defect spatial range of the defect to be generated is located in the target scene image, and a defect location mask is generated based on the defect spatial range. Based on the defect localization mask, a local scene image representing the spatial range of the defect is extracted from the target scene image; The corresponding visual features are extracted from the local scene image, and the visual parameters of the defect to be generated are adjusted according to the visual features so that the visual parameters are consistent with the visual features. Based on the adjusted visual parameters and appearance features, a defect texture image is generated. Based on the defect texture image, the target scene image is embedded and edge pixel fusion is performed to obtain the anti-defect sample image.

2. The method for generating countermeasure defect samples based on image-text linkage according to claim 1, characterized in that, The step of extracting the corresponding visual features from the local scene image includes: Obtain the grayscale values ​​of the red channel, green channel, and blue channel of each pixel in the local scene image, as well as the pixel's grayscale value. Based on the red channel grayscale values, green channel grayscale values, and blue channel grayscale values ​​of all local scene pixels, the first average grayscale value, the maximum red channel grayscale value, the minimum red channel grayscale value, the second average grayscale value, the maximum green channel grayscale value, the minimum green channel grayscale value, the third average grayscale value, the maximum blue channel grayscale value, and the minimum blue channel grayscale value of the image in the local scene are calculated. The first average gray value, the maximum red channel gray value, the minimum red channel gray value, the second average gray value, the maximum green channel gray value, the minimum green channel gray value, the third average gray value, the maximum blue channel gray value, and the minimum blue channel gray value are used as color features. For each local scene pixel, the brightness value is calculated based on the gray values ​​of the red channel, green channel, and blue channel of the current local scene pixel. Calculate the average brightness value of all local scene pixels and use the average value as the brightness feature; The brightness difference is calculated based on the brightness values ​​of all local scene pixels, and the brightness difference is used as a contrast feature. Based on the sliding window method and the grayscale value of the pixel, the variance within each window in the local scene image is calculated, and the variance is used as a texture feature. The color features, brightness features, contrast features, and texture features are used as the visual features.

3. The method for generating countermeasure defect samples based on image-text linkage according to claim 2, characterized in that, The step of embedding and fusing edge pixels of the target scene image based on the defect texture image to obtain a reverse defect sample image includes: Obtain the pixel coordinates of the target pixel point in the defect localization mask used to identify the spatial location of the defect to be generated, as well as the number of the first horizontal pixels in the horizontal direction and the number of the first vertical pixels in the vertical direction in the defect texture image; Based on the pixel coordinates of all target pixels, determine the number of second horizontal pixels in the horizontal direction and the number of second vertical pixels in the vertical direction of the region to be embedded in the target scene image; Based on the first horizontal pixel count, the second horizontal pixel count, the first vertical pixel count, and the second vertical pixel count, a size alignment operation is performed on the defect texture image to obtain a target texture image with the same size as the region to be embedded. The target scene image is replaced and stitched with pixel information based on the target texture image to obtain a composite image of the target defect; Edge fusion is performed on the embedded boundaries in the target defect synthetic image based on the defect description text to obtain the anti-defect sample image.

4. The method for generating countermeasure defect samples based on image-text linkage according to claim 3, characterized in that, The step of replacing and connecting pixel information of the target scene image based on the target texture image to obtain a composite image of the target defect includes: From the target scene image, determine the target scene image pixels corresponding to each target texture pixel in the target texture image; For each pixel in the target scene image, the pixel information of the current target scene image pixel is replaced with the pixel information of the corresponding target texture pixel to obtain the initial defect synthesis image; Determine the boundary pixels between the region to be embedded and the region not to be embedded from the initial defect synthesis image; The initial defect composite image is processed by connecting the boundary pixels to obtain the target defect composite image.

5. The method for generating countermeasure defect samples based on image-text linkage according to claim 4, characterized in that, The step of performing boundary pixel joining processing on the initial defect composite image based on the boundary pixels to obtain the target defect composite image includes: Separate a number of first boundary pixels that are located within the region to be embedded from all boundary pixels, and a number of remaining second boundary pixels; For each first boundary pixel, construct a pixel pair based on the current first boundary pixel and the second boundary pixel that will be adjacent to the current first boundary pixel; Based on the coordinates of the first boundary pixel and the second boundary pixel in each pixel pair, the first coordinate difference in the horizontal direction and the second coordinate difference in the vertical direction of each pixel pair are calculated. For each pixel pair, if the first coordinate difference is 1 and the second coordinate difference is 0 in the current pixel pair; or the first coordinate difference is 0 and the second coordinate difference is 1, then the current pixel pair is determined to be in a normal connection state; otherwise, the current pixel pair is determined to be in an abnormal connection state. Anomaly repair is performed on all pixel pairs in an abnormal connection state to obtain the target defect synthetic image.

6. The method for generating countermeasure defect samples based on image-text linkage according to claim 5, characterized in that, The step of performing abnormal repair on all pixel pairs in an abnormal connection state to obtain the target defect synthetic image includes: Get the coordinates of the abnormal pixels in all pixel pairs that are in an abnormal connection state; For each pixel pair in an abnormal connection state, the first coordinate offset in the horizontal direction and the second coordinate offset in the vertical direction between the two pixels in the current pixel pair are calculated based on the abnormal pixel coordinates. Based on the abnormal coordinates, the first coordinate offset, and the second coordinate offset, the abnormal type of each pixel pair in an abnormal connection state is determined; wherein, if there are no blank coordinates between the two pixels in the current pixel pair, the abnormal pixel coordinates are not repeated, and at least one of the first coordinate offset and the second coordinate offset is greater than a preset offset threshold, the abnormal type of the pixel pair in the current abnormal connection state is determined to be pixel misalignment. If there is a blank coordinate between two pixels in the current pixel pair, the abnormal type of the pixel pair that is currently in an abnormal connection state is determined to be pixel gap. If the abnormal pixel coordinates of two pixels in the current pixel pair are the same, the abnormal type of the pixel pair currently in an abnormal connection state is determined to be pixel overlap. For all first abnormal pixels with the abnormality type of pixel misalignment, the coordinate deviation value of each first abnormal pixel is calculated according to the preset standard coordinates and the coordinates of the abnormal pixels. Then, the corresponding first abnormal pixel is translated at an equal distance according to each coordinate deviation value to generate the target defect composite image. For all second abnormal pixels with the abnormality type of pixel gap, obtain the abnormal texture features of each second abnormal pixel and the normal texture features of the normal boundary pixels adjacent to each second abnormal pixel; according to the abnormal texture features and normal texture features, perform pixel filling processing on the corresponding gap region to generate the target defect synthetic image. For all third-order anomalous pixels with pixel overlap as the anomalous type, redundant pixel removal is performed on all third-order anomalous pixels to generate the target defect synthetic image.

7. The method for generating countermeasure defect samples based on image-text linkage according to claim 6, characterized in that, The step of edge-blending the embedded boundaries in the target defect synthetic image based on the defect description text to obtain a counter-defect sample image includes: From the synthesized image of the target defect, determine the defect boundary pixels between the defect area and the non-defect area; An edge neighborhood range is obtained by extending a predetermined number of pixels outward from the defect boundary pixel; Based on the defect description text, perform edge texture analysis on the defect to be generated to determine the edge texture features of the defect to be generated. Extract the visual features of all edge neighborhood pixels within the edge neighborhood range, and compare the visual features of all edge neighborhood pixels one by one with the edge texture features to determine the coordinates of edge neighborhood pixels that match the edge texture features, thereby obtaining a texture-related pixel set; Smooth transition processing is performed on the edge neighbor pixels in the texture-associated pixel set and the edge pixels in the defect region to obtain the preliminary fused edge neighbor pixel information; Obtain defect pixel information of the defect region, and background pixel information of the background region other than the defect region and the edge neighborhood; The defective sample image is obtained by fusing pixel information based on the defective pixel information, background pixel information, and edge neighbor pixel information.

8. A device for generating defect samples for countermeasures based on image-text linkage, characterized in that, include: The system includes modules for image and text data acquisition, text semantic parsing, real defect location feature matching, defect spatial range localization, local scene image generation, defect texture image generation, and countermeasure defect sample image generation. The image and text data acquisition module is used to acquire target scene images of power equipment in normal condition, as well as defect description text; The text semantic parsing module is used to perform semantic parsing on the defect description text to obtain the defect type, defect location, and appearance features of the defect to be generated as a countermeasure. The real defect location feature matching module is used to match the real defect location feature corresponding to the defect type from a preset knowledge base according to the defect type; wherein, the preset knowledge base is constructed based on the historical defect status of each power equipment; The defect spatial range localization module is used to locate the defect spatial range of the defect to be generated in the target scene image based on the defect location and the actual defect location features, and to generate a defect localization mask based on the defect spatial range. The local scene image generation module is used to extract a local scene image representing the spatial range of the defect from the target scene image based on the defect localization mask; The defect texture image generation module is used to extract corresponding visual features from the local scene image, adjust the visual parameters of the defect to be generated according to the visual features so that the visual parameters are consistent with the visual features, and generate a defect texture image according to the adjusted visual parameters and appearance features. The anti-measure defect sample image generation module is used to perform image embedding and edge pixel fusion on the target scene image based on the defect texture image to obtain the anti-measure defect sample image.

9. A terminal device, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement a method for generating countermeasure defect samples based on image-text linkage as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device where the storage medium is located to execute a method for generating countermeasure defect samples based on image-text linkage as described in any one of claims 1 to 7.

Images