Data enhancement method and device, computer readable storage medium and electronic equipment

Abstract

The application relates to the technical field of data enhancement of computer vision, and discloses a data enhancement method, device, computer readable storage medium and electronic equipment, the method comprising the following steps: performing frequency domain disturbance enhancement on an original image to obtain a frequency domain disturbance enhanced image; determining a global scene embedding vector, a conditional vector corresponding to each detection target and a region soft mask according to a global scene prompt and target level prompts, position information and region masks corresponding to the at least one detection target; generating a virtual target domain image by performing diffusion generation on the global scene embedding vector and the conditional vector corresponding to each detection target and the region soft mask by using a pre-trained instance level conditional diffusion model; performing fusion processing on the frequency domain disturbance enhanced image and the virtual target domain image to obtain a fusion image; and performing physical enhancement processing on the fusion image to obtain a target enhanced image. The method improves the diversity of training sample distribution and improves the generalization ability of a cross-domain detection model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data enhancement, in particular to a data enhancement method, device, computer readable storage medium and electronic equipment. BACKGROUND

[0002] Target detection in power inspection tasks needs to accurately identify equipment defects in various complex environments. However, related deep learning detection models usually assume that the training data and the test data have the same distribution, which is often not true in actual application. Especially in variable weather (such as rain, fog, snow and other bad weather) and different imaging device conditions, the distribution characteristics of the image will change significantly, resulting in a serious decline in the performance of the conventional model in cross-domain scenarios, and an increase in missed detection and false detection.

[0003] In addition, real inspection data under bad weather is very scarce, which is difficult to support the learning of the model for these conditions, and poses a hidden danger to the safety and reliability of the power inspection system. SUMMARY

[0004] The present application aims to at least partially solve one of the technical problems in the related art. To this end, one object of the present application is to propose a data enhancement method that improves the diversity of training sample distribution while maintaining semantic consistency, and improves the generalization ability of the cross-domain detection model.

[0005] A second object of the present application is to propose a data enhancement device.

[0006] A third object of the present application is to propose a computer readable storage medium.

[0007] A fourth object of the present application is to propose an electronic equipment.

[0008] To achieve the above-mentioned objects, the first aspect of the present application proposes a data enhancement method, which comprises: performing frequency domain disturbance enhancement on an original image to obtain a frequency domain disturbance enhanced image, wherein the original image includes at least one detection target; determining a global scene embedding vector, a condition vector corresponding to each detection target and a region soft mask according to a global scene prompt and a target level prompt, position information and a region mask corresponding to at least one detection target; using a pre-trained instance level condition diffusion model to generate a virtual target domain image according to the global scene embedding vector, the condition vector corresponding to each detection target and the region soft mask; performing fusion processing on the frequency domain disturbance enhanced image and the virtual target domain image to obtain a fusion image; and performing physical enhancement processing on the fusion image to obtain a target enhanced image.

[0009] According to the data enhancement method provided in the embodiments of the present application, the original image is subjected to frequency domain disturbance enhancement, and instance-level conditional diffusion generation is performed based on the detection target. The frequency domain disturbance enhanced image obtained through the frequency domain disturbance enhancement is fused with the virtual target domain image obtained through the instance-level conditional diffusion generation, and the fusion result is subjected to physical enhancement to obtain a target enhanced image. The embodiments of the present application improve the diversity of training sample distribution and the generalization ability of the cross-domain detection model while keeping semantic consistency.

[0010] In addition, the data enhancement method provided in the above embodiments of the present application can also have the following additional technical features:

[0011] According to an embodiment of the present application, the frequency domain disturbance enhancement on the original image to obtain a frequency domain disturbance enhanced image comprises: performing global random disturbance enhancement on the original image in the frequency domain to obtain a global frequency domain enhanced image; performing disturbance enhancement on a local target region corresponding to each detection target in the original image to obtain a local mask enhanced image; and performing fusion processing on the global frequency domain enhanced image and the local mask enhanced image to obtain the frequency domain disturbance enhanced image.

[0012] According to an embodiment of the present application, the global random disturbance enhancement on the original image in the frequency domain to obtain a global frequency domain enhanced image comprises: performing Fourier transform on the original image to obtain a frequency domain image; performing frequency domain filtering processing and random disturbance processing on the frequency domain image according to a preset weather condition to obtain a weather disturbance image; and performing inverse Fourier transform on the weather disturbance image to obtain the global frequency domain enhanced image. The disturbance enhancement on a local target region corresponding to each detection target in the original image to obtain a local mask enhanced image comprises: extracting a region mask corresponding to each detection target in the original image according to a labeled box corresponding to each detection target in the original image by using a pre-trained image segmentation model to obtain a region mask set; performing division of a background and a target foreground on the original image according to the region mask set to obtain a background image and a target foreground image corresponding to each detection target, wherein the background image is obtained by removing all local target regions corresponding to detection targets from the original image, and the target foreground image is obtained by multiplying the original image and a target mask corresponding to the detection target pixel by pixel; performing random data enhancement processing on the background image and each target foreground image to obtain a background enhanced image and each target foreground enhanced image; and superimposing each target foreground enhanced image into the background enhanced image to obtain the local mask enhanced image.

[0013] According to one embodiment of the present application, the global scene prompt and the target level prompt are text prompts, and the determining of the global scene embedding vector, the condition vector corresponding to each detection target, and the region soft mask according to the global scene prompt and the target level prompt corresponding to at least one detection target includes: converting the global scene prompt and the target level prompt corresponding to at least one detection target into a global scene embedding vector and a target level embedding vector corresponding to each detection target respectively by using a pre-trained text encoder, wherein the global scene prompt is determined by a preset scene, and the target level prompt is determined by the attribute of the corresponding detection target; converting the position information corresponding to at least one detection target into a position vector by using a two-dimensional Fourier position encoding method to obtain a position vector corresponding to each detection target; connecting the target level embedding vector and the position vector corresponding to each detection target, and mapping each connection result by using a multilayer perception machine to obtain a condition vector corresponding to each detection target; and performing fuzzy processing on the region mask corresponding to at least one detection target to obtain a region soft mask corresponding to each detection target.

[0014] According to one embodiment of the present application, the fusion processing of the frequency domain disturbance enhanced image and the virtual target domain image to obtain a fusion image includes: performing linear weighted fusion of the frequency domain disturbance enhanced image and the virtual target domain image in a pixel space to obtain the fusion image.

[0015] According to one embodiment of the present application, the physical enhancement processing of the fusion image to obtain a target enhanced image includes: performing global ambient light disturbance on the fusion image by using a pre-trained atmospheric scattering model to obtain a degradation image; performing local directional disturbance on the fusion image to obtain a local directional disturbance image; and performing fusion processing on the degradation image and the local directional disturbance image to obtain the target enhanced image.

[0016] According to one embodiment of the present application, the global ambient light disturbance of the fusion image by using the pre-trained atmospheric scattering model comprises: performing Retinex decomposition on the fusion image by using a lightweight convolutional neural network for self-supervised learning to obtain a predicted illumination map; generating a detail reflection map according to the fusion image and the predicted illumination map; and approximately generating the degraded image according to the detail reflection map, an RGB three-channel transmittance field and an ambient light three-channel value by using the pre-trained atmospheric scattering model, wherein the RGB three-channel transmittance field is controlled by a weather parameter, and the ambient light three-channel value is generated by a preset three-channel value generation model according to the weather parameter, and the weather parameter is determined by a preset weather condition.

[0017] To achieve the above object, the second aspect of the present application provides a data enhancement device, which comprises: a frequency domain disturbance module, configured to perform frequency domain disturbance enhancement on an original image to obtain a frequency domain disturbance enhanced image, wherein the original image comprises at least one detection target; a diffusion generation module, configured to determine a global scene embedding vector and a condition vector and a region soft mask corresponding to each detection target according to a global scene prompt and a target level prompt, position information and a region mask corresponding to the at least one detection target, and generate a virtual target domain image by using a pre-trained instance level conditional diffusion model according to the global scene embedding vector and the condition vector and the region soft mask corresponding to each detection target; a fusion module, configured to perform fusion processing on the frequency domain enhanced image and the virtual target domain image to obtain a fusion image; and a physical enhancement module, configured to perform physical enhancement processing on the fusion image to obtain a target enhanced image.

[0018] To achieve the above object, the third aspect of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the data enhancement method according to the first aspect of the present application.

[0019] To achieve the above object, the fourth aspect of the present application provides an electronic device, which comprises a memory and a processor, and the memory stores a computer program, and the computer program is executed by the processor to implement the data enhancement method according to the first aspect of the present application.

[0020] Additional aspects and advantages of the present application will be made apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 is a flowchart of a data augmentation method according to an embodiment of the present application;

[0022] Figure 2 is a flowchart of obtaining a frequency domain perturbation augmented image according to an embodiment of the present application;

[0023] Figure 3 is a flowchart of obtaining a global frequency domain augmented image according to an embodiment of the present application;

[0024] Figure 4 is a flowchart of obtaining a local mask augmented image according to an embodiment of the present application;

[0025] Figure 5 is a flowchart of determining a global scene embedding vector and a conditional vector and a region soft mask corresponding to each detection target according to an embodiment of the present application;

[0026] Figure 6 is a flowchart of obtaining a target augmented image according to an embodiment of the present application;

[0027] Figure 7 is a flowchart of obtaining a degraded image according to an embodiment of the present application;

[0028] Figure 8 is a flowchart of obtaining a local direction perturbation image according to an embodiment of the present application;

[0029] Figure 9 is a schematic diagram of obtaining a target augmented image in a snowy scene according to an embodiment of the present application;

[0030] Figure 10 is a schematic diagram of obtaining a target augmented image in a thick fog scene according to an embodiment of the present application;

[0031] Figure 11 is a schematic diagram of a data augmentation device according to an embodiment of the present application;

[0032] Figure 12 is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0033] Embodiments of the present application are described in detail below with reference to the attached drawings, which show by way of example, embodiments in which like numerals indicate like elements or elements having a similar function. The embodiments described below are examples intended to explain the present application, and should not be understood as limiting the present application.

[0034] Related visual enhancement techniques mostly operate in pixel or semantic space, without explicitly suppressing the model's abnormal response to noise in specific frequency bands (high-frequency edges / textures, or low-frequency overall). In the case of noise reduction or extreme degradation, this frequency domain sensitivity directly leads to false positives, false negatives, and detection box shifts.

[0035] Related target detection methods have many shortcomings in cross-domain generalization. Deep learning detection models are highly dependent on training distribution, and their performance often drops sharply when encountering different weather or device conditions from the training domain. To improve robustness, traditional data augmentation strategies mostly perform random transformations (such as color jitter, blur, noise, etc.) at the image pixel level or convert image styles at the semantic level. Although this alleviates the differences between training and testing distributions to some extent, it does not fundamentally solve the problem. The model may still have an excessively high response to noise and changes in certain sensitive areas of the frequency spectrum (e.g., high-frequency edges and details, or low-frequency overall brightness). In extreme degradation scenarios (such as heavy fog or strong noise interference), this frequency domain sensitivity directly leads to detection box shifts and false positives / negatives. On the other hand, related cross-domain generalization detection methods mostly focus on global alignment in feature space, trying to reduce differences by matching source and target feature distributions. However, visual features are extremely sensitive to low-level weather, texture, and resolution changes. When the inspection scene spans different weather or devices, these low-level factors change significantly first, and simply aligning statistical distributions cannot maintain robustness and interpretability.

[0036] In addition, most cross-domain generalization detection methods ignore the difference enhancement of local targets and backgrounds, and lack modeling of the physical mechanisms of complex weather imaging degradation. Simple data augmentation cannot truly reproduce image features under conditions such as rain, snow, and fog.

[0037] In summary, related target detection techniques cannot simultaneously consider multiple adverse environmental factors and cannot fully address the detection reliability problem under cross-weather and cross-device conditions for power inspection.

[0038] To solve the above problems, embodiments of the present application provide a data enhancement method, device, computer readable storage medium, and electronic device. The data enhancement method, device, computer readable storage medium, and electronic device of the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0039] Figure 1 is a flowchart of the data enhancement method of an embodiment of the present application. As shown in Figure 1 the data enhancement method can include:

[0040] S101, performing frequency domain disturbance enhancement on an original image to obtain a frequency domain disturbance enhanced image, wherein the original image includes at least one detection target;

[0041] S102, determining a global scene embedding vector and a condition vector and a region soft mask corresponding to each detection target according to the global scene prompt and the target level prompt, position information and region mask corresponding to at least one detection target;

[0042] S103, generating a virtual target domain image by diffusion generation according to the global scene embedding vector and the condition vector and the region soft mask corresponding to each detection target by using a pre-trained instance level condition diffusion model;

[0043] S104, performing fusion processing on the frequency domain disturbance enhanced image and the virtual target domain image to obtain a fusion image;

[0044] S105, performing physical enhancement processing on the fusion image to obtain a target enhanced image.

[0045] To enhance the robustness of target detection in complex weather in the power inspection task, when training the target detection model, the sample image input to the target detection model can be data enhanced by using the data enhancement method of the embodiment of the present application, and the target detection model can be trained based on the enhanced sample image. The sample image in the training sample set can also be data enhanced by using the data enhancement method in the embodiment of the present application to expand the sample image, and the target detection model can be trained based on the expanded training sample set.

[0046] The data enhancement method of the embodiment of the present application includes a frequency domain disturbance branch, an instance level condition diffusion generation branch and a physical imaging simulation enhancement branch. The frequency domain disturbance branch and the instance diffusion generation branch generate their own enhanced images respectively, and then obtain an intermediate enhanced result (fusion image) by linearly weighting and fusing, and finally input the fusion image into the physical enhancement branch to simulate the effect of complex weather, and output the final enhanced result, that is, obtain the target enhanced image.

[0047] Specifically, the frequency domain disturbance branch performs frequency domain disturbance enhancement on the original image to obtain a frequency domain disturbance enhanced image The original image of the embodiment of the present application can have at least one detection target, and the detection target in the original image can be the same type of detection target or different types of detection targets.

[0048] The global scene prompt is acquired, and target-level prompts, position information and region masks corresponding to at least one detection target of the same type or different types are acquired. .

[0049] The frequency domain disturbance enhanced image and the virtual target domain image are fused, and a target enhanced image is generated in a physically guided manner.

[0050] In order to enable the target detection model to obtain good generalization detection performance even in the case of lacking real severe weather data, and to meet the actual deployment requirements, the data enhancement method containing multiple branches constructed by the embodiment of the application is used to expand the training data distribution by comprehensively utilizing frequency domain disturbance, instance-level conditional diffusion generation and physical imaging simulation enhancement, etc. in the application background of power inspection scene and power equipment as the detection target.

[0051] By using the data enhancement method provided by the embodiment of the application, realistic severe weather simulation images can be generated on the basis of limited normal weather training data such as sunny days, so as to improve the robustness and adaptability of the target detection model under severe weather and multi-source equipment conditions.

[0052] The data enhancement method provided by the embodiment of the application performs frequency domain disturbance enhancement on the original image, and performs instance-level conditional diffusion generation based on the detection target. The frequency domain disturbance enhanced image obtained by the frequency domain disturbance enhancement is fused with the virtual target domain image generated by the instance-level conditional diffusion generation. The fusion result is physically enhanced to obtain a target enhanced image. The embodiment of the application improves the diversity of training sample distribution under the premise of maintaining semantic consistency, and improves the generalization ability of the cross-domain detection model.

[0053] In an embodiment of the application, as shown in Figure 2 , the frequency domain disturbance enhancement on the original image to obtain the frequency domain disturbance enhanced image can include:

[0054] S201, performing global random disturbance enhancement on the original image in the frequency domain to obtain a global frequency domain enhanced image;

[0055] S202, the local target region corresponding to each detection target in the original image is disturbed and enhanced to obtain a local mask enhanced image;

[0056] S203, the global frequency domain enhanced image and the local mask enhanced image are fused to obtain a frequency domain disturbance enhanced image.

[0057] Specifically, the original image is transformed into a frequency domain, and the transformed frequency domain image is globally randomly disturbed and enhanced to obtain a global frequency domain enhanced image . The local target region corresponding to each detection target in the original image is extracted, and each local target region is disturbed and enhanced to obtain a local mask enhanced image . The global frequency domain enhanced image and the local mask enhanced image are fused to obtain a frequency domain disturbance enhanced image .

[0058] The global frequency domain enhanced image and the local mask enhanced image are fused according to a weight to obtain a frequency domain disturbance enhanced image output by a frequency domain disturbance branch :

[0059]

[0060] wherein, represents the frequency domain disturbance enhanced image, represents the global frequency domain enhanced image, represents the local mask enhanced image, represents a first fusion coefficient. By adjusting the first fusion coefficient , the balance of global and local disturbance can be realized (for example a larger value makes the global frequency domain change dominant, and a smaller value emphasizes the local target appearance change), which can further increase the distribution diversity of the enhanced samples.

[0061] In the frequency domain disturbance branch in the embodiment of the application, image-level fusion of global and local results is performed, rather than fusion of network features in the target detection model, so that the independent and pluggable characteristics of the data enhancement method in the embodiment of the application are ensured.

[0062] In one embodiment of the application, as shown in Figure 3 , the original image is globally randomly disturbed and enhanced in the frequency domain to obtain a global frequency domain enhanced image, which can include:

[0063] S301, Fourier transform is performed on the original image to obtain a frequency domain image;

[0064] S302, based on preset weather conditions, perform frequency domain filtering and random perturbation processing on the frequency domain image to obtain a weather perturbation image;

[0065] S303 performs an inverse Fourier transform on the weather disturbance image to obtain a global frequency domain enhanced image.

[0066] In this embodiment of the invention, when performing global random perturbation enhancement on the entire original image in the frequency domain, perturbation control based on complex weather conditions is introduced.

[0067] Specifically, for the original image Performing a Fourier transform, its two-dimensional discrete Fourier transform is expressed as: . Indicates Fourier transform, Indicates the inverse Fourier transform. The horizontal axis represents the frequency domain space. The vertical axis represents the frequency domain space.

[0068] According to preset weather conditions Introducing frequency domain filtering With disturbance function .in This refers to a bandpass filter, specifically its passband radius or range. According to preset weather conditions Dynamic settings (e.g., using a low-pass filter to emphasize low frequencies when simulating dense fog, and using a high-pass filter to preserve high-frequency details when simulating noisy raindrops). This represents a random disturbance function with zero mean, whose variance varies with preset weather conditions. Adjustments were made (e.g., adding high-frequency random perturbations to simulate noise on rainy days, and adding low-frequency offsets to simulate uneven lighting on foggy days).

[0069] It should be noted that, in the embodiments of the present invention This is a set of configuration parameters representing the target weather, used to indicate the weather type and its intensity; the weather type can be one or a combination of fog, rain, snow, dust storm, and sunshine, and the intensity can be determined by levels or percentages. Frequency domain parameters are determined based on w. Used to set the bandpass (or low / high pass) filter. passband radius / range Used to set the perturbation function The zero-mean-variance distribution in the frequency domain emphasizes low frequencies in foggy weather and enhances high-frequency random disturbances in rainy weather.

[0070] For example, the preset weather conditions in the embodiments of the present invention It can be represented as: ,in, represent weather type, such as fog, rain, snow, sand, light, etc. represent intensity level or [0, 1], represent direction angle (optional for rain, sandstorm, etc.), represent color deviation, particle correlation, etc.

[0071] Decided by The following mapping (function or lookup table): filter parameters, such as morphological ; radius / bandwidth , ; directional angle (if any) . Disturbance statistics: variance spectrum (low / middle / high frequency weight respectively); spatial correlation length / anisotropy coefficient (control texture grain or stripe). Ambient light, fog, etc. in physical enhancement.

[0072] The frequency domain filtering and random disturbance processing in the embodiment of the present application obtain , wherein the obtained . The shape and intensity of are given by .

[0073] Minimum parameter table (example):

[0074]

[0075] It should be noted that the preset weather condition in the embodiment of the present application contains common severe weather types (such as fog, rain, snow, sand, light, etc.) and severe weather intensity (such as the concentration of fog, the size of rain, snow and sand, the intensity of light) in the power inspection scene. The severe weather intensity can be divided into multiple levels or determined by percentage. Each weather type corresponds to a specific set of frequency domain parameter configurations. By dynamically adjusting the passband range of the bandpass filter and the distribution variance of the random disturbance function , the image spectrum features under different weather conditions are accurately simulated:

[0076] For example, when the preset weather condition is set to “thick fog”, the bandpass filter adopts low-pass characteristics, emphasizing low-frequency components (suppressing high-frequency details, simulating foggy image blur), while the variance of the random disturbance function increases in the low-frequency region, simulating the low-frequency offset of uneven fog light.

[0077] When the preset weather condition is “rainy day”, the bandpass filter Reserve high-frequency passband (highlight the high-frequency features of raindrop texture), random disturbance function Increase the random disturbance variance in the high-frequency region to simulate the high-frequency noise caused by rainwater reflection;

[0078] For "snowy day", "sandstorm" and other preset weather, the corresponding exclusive frequency band range and disturbance variance parameters are also set to ensure that the generated frequency domain disturbance is consistent with the spectral characteristics of the real image under this weather.

[0079] By presetting multiple weather conditions And matching the corresponding frequency domain parameters, batch-generated enhanced data covering multiple types of severe weather can expand the weather diversity of the training samples and improve the model's adaptability to different weather scenarios.

[0080] According to the preset weather conditions Determine the corresponding frequency domain filtering And the disturbance function , and according to the determined frequency domain filtering And the disturbance function The frequency domain image is subjected to frequency domain filtering and random disturbance processing, and a weather disturbance image :

[0081]

[0082] Among them, The weather disturbance image is represented by The disturbance function is represented by The band-pass filter is represented by The frequency domain image is represented by

[0083] Take the inverse Fourier transform of the disturbance image To get the global frequency domain enhanced image :

[0084]

[0085] Among them, The global frequency domain enhanced image is represented by The inverse Fourier transform is represented by The weather disturbance image is represented by

[0086] The modeling of the disturbance function in the embodiment of the application can randomly scale and disturb each frequency component according to a Gaussian distribution. By randomly disturbing the entire image in the frequency domain and controlling the frequency band, the influence of different severe weather on the image spectrum (for example, the disturbance function containing the preset weather conditions Increase texture noise in rainy conditions and attenuate high-frequency details in foggy conditions.

[0087] In one embodiment of the present invention, such as Figure 4 As shown, perturbation enhancement is applied to the local target regions corresponding to each detected target in the original image to obtain a locally enhanced mask image, which may include:

[0088] S401: Using a pre-trained image segmentation model, extract the region mask corresponding to each detection target in the original image based on the bounding box corresponding to each detection target in the original image, and obtain a set of region masks.

[0089] S402, based on the set of region masks, divide the original image into background and target foreground to obtain a background image and a target foreground image corresponding to each detected target. The background image is obtained by removing all local target regions corresponding to the detected targets from the original image, and the target foreground image is obtained by multiplying the original image with the corresponding region mask pixel by pixel.

[0090] S403, perform random data augmentation processing on the background image and each target foreground image to obtain the background enhanced image and each target foreground enhanced image;

[0091] S404: The foreground enhancement images of each target are superimposed on the background enhancement image to obtain a local mask enhancement image.

[0092] To enhance the diversity of target instances, this invention utilizes a pre-trained image segmentation model to extract the original image. The local target regions corresponding to each detected target are extracted, and the extracted local target regions are independently perturbed and enhanced.

[0093] In practice, the powerful segmentation capabilities of the Segment Anything Model (SAM, image segmentation model) can be utilized to extract local target regions from an image. It should be noted that the images input to train the object detection model, as well as the sample images in the training sample set, are pre-annotated with bounding boxes for each detected object. Therefore, given the original image... and original image The set of bounding boxes for each detection target in the middle (each) For the first The bounding box of each target. Original image The number of bounding boxes in the original image (Number of detected targets), and use SAM to extract the region mask corresponding to each detected target in the original image. : , thus obtaining the target mask set . representative pixel whether belonging to a target .

[0094] according to the region mask set , the original image is divided into two parts of target foreground and background. The original image removes all local target regions corresponding to the detected target to obtain a background image : . The target foreground image corresponding to each detected target is obtained by pixel-by-pixel multiplication of the pixels of the original image and the corresponding region mask , and the target foreground image .

[0095] Random data augmentation operations are applied to the background image and each target foreground image respectively to realize the diversification of local regions, so that different transformations can be applied to the background and the target. Let be a random enhancement operator for the background (such as random color jittering, blurring, etc.), be an enhancement operator for the target instance (such as random deformation, brightness disturbance, etc.). Random disturbance is applied to the background image to obtain a background enhanced image . Random disturbance is applied to the target foreground image to obtain a target foreground enhanced image .

[0096] The enhanced target is superimposed back into the enhanced background image to obtain a local mask enhanced image : .

[0097] It should be noted that at the local target region , the background enhanced image is zero, which can keep the background enhanced image the original mask shape of the detected target to ensure that the local mask enhanced image is the same size as the original image .

[0098] The embodiment of the application realizes decoupled enhancement of scene background and target attributes by randomly disturbing the background and the target respectively, and improves the data diversity.

[0099] In an embodiment of the application, the global scene prompt and the target-level prompt are text prompts. As shown in the following formula, the global scene embedding vector and the conditional vector and the region soft mask corresponding to each detected target are determined according to the global scene prompt and the target-level prompt, position information and region mask corresponding to at least one detected target, which can include: Figure 5 ​

[0100] S501, converting the global scene prompt and the target-level prompt corresponding to each detection target into a global scene embedding vector and a target-level embedding vector corresponding to each detection target, respectively, by using a pre-trained text encoder, wherein the global scene prompt is determined by a preset scene, and the target-level prompt is determined by the attribute of the corresponding detection target;

[0101] S502, converting the position information corresponding to each detection target into a position vector by using a two-dimensional Fourier position encoding method, to obtain a position vector corresponding to each detection target;

[0102] S503, connecting the target-level embedding vector and the position vector corresponding to each detection target, and mapping each connection result by using a multilayer perceptron, to obtain a condition vector corresponding to each detection target;

[0103] S504, performing fuzzy processing on the region mask corresponding to each detection target, to obtain a region soft mask corresponding to each detection target.

[0104] The instance-level conditional diffusion generation branch of the embodiment of the present application is based on the object-level controlled image generation mechanism of Instance Diffusion, and aims to generate virtual samples of the target domain by using an instance-level conditional diffusion model. Unlike using only a single text conditional diffusion, the embodiment of the present application adopts double prompts: an image-level prompt (a global scene prompt) describes the overall scene and environment, and a target-level prompt describes the attribute of each target to be generated, so as to ensure that the generated image not only matches the background environment under complex weather, but also accurately contains the expected target object.

[0105] The global scene prompt can be determined based on a preset scene. For example, a global scene prompt of an image level is constructed for a preset scene : a transformer station in heavy rain at night, which is used to describe the global environment.

[0106] It should be noted that the preset scene in the embodiment of the present application covers the common environment types in power inspection, including different weather (such as “sunny day”, “heavy fog”, “heavy rain”, “light snow”), time period (such as “dawn”, “noon”, “night”), device deployment scene (such as “outdoor area of transformer station”, “mountain section of power transmission line”, “inside of power distribution room”), etc. Each preset scene corresponds to an image-level text prompt For example: “outdoor transformer station in heavy rain”, “mountain power transmission tower at dawn”, “power distribution room equipment under night light”.

[0107] For each preset scene text prompt, the text encoder of CLIP (Contrastive Language-Image Pretraining) is used The corresponding global scene embedding vector is generated This vector is one of the input conditions of the instance diffusion generation branch, guiding the instance-level conditional diffusion model to generate images that conform to the characteristics of the scene environment (such as rain in a rainstorm scene and low light, halo in a night scene, etc.). By presetting multiple scenes and generating corresponding embedding vectors, the scene coverage of the generated samples can be systematically expanded, ensuring that the augmented data can match the diverse actual environments in power inspection, further improving the model's adaptability to cross-scene changes.

[0108] Practically, the target-level prompt can be constructed based on the attributes of the detected target For example, for a transformer The target-level prompt is constructed : A red transformer is smoking. The detection targets in power inspection can include transformers, insulators, towers, etc., and the preset attributes cover their physical characteristics (color, material, shape) and possible fault states (such as "smoking", "oil leakage", "damage", "aging", "snow cover", etc.). These attributes are directly related to common fault types of detection targets, for example: the attributes of a transformer can include "red / blue" (color), "smoking / oil leakage / surface rust" (fault state); the attributes of an insulator include "white / brown" (color), "damage / cracks / covered with dirt" (fault state). Each combination of different attributes of a detection target corresponds to a target-level prompt , such as: "a red transformer is running normally" (no fault), "a blue transformer is leaking oil" (fault), "a rusty transformer surface is damaged" (aging + fault).

[0109] The global scene prompt and the target-level prompt are converted into embedding vectors using a pre-trained text encoder (such as clip-ViT-B-32):

[0110]

[0111] where, represents the text embedding of the global scene, represents the text embedding of the target-level prompt, represents the operation of the text encoder to extract semantic vectors.

[0112] To guide the instance-level conditional diffusion model to generate the corresponding detection target at the specified position, a two-dimensional Fourier position encoding method is used to embed the detection target position (for example, the normalized bounding box coordinates) corresponding to each detection target. For the normalized bounding box coordinates of the i-th detection target , a set of cosine functions is used to encode the center position and size of the normalized bounding box coordinates to obtain a fixed-dimensional position vector .

[0113]

[0114] The position vector corresponding to each detection target in the embodiment of the application is connected with the semantic embedding (target-level embedding vector ), and is mapped through a multi-layer perceptron (MLP) to obtain a conditional vector that fuses semantics and positions .

[0115] Since the spatial constraint provided by the rectangular bounding box is relatively rough, in order to improve the shape adaptability of position control, the embodiment of the application proposes to use a soft mask (probability heat map) to replace the hard bounding box constraint. The segmentation mask of each detection target can be blurred to obtain a continuous-valued probability heat map (region soft mask) , the value of which reflects the prior probability that a pixel belongs to a target . The region soft mask can also be generated based on a two-dimensional Gaussian function when only the bounding box is available :

[0116]

[0117] wherein, represents the center of the bounding box of the i-th detection target, represents the width of the bounding box, represents the height of the bounding box, is a scale coefficient, and correspond to the standard deviation in the width and height directions, respectively, to control the diffusion range of the heat map in the two directions. The above-mentioned region soft mask is a continuous weight map at the pixel level, used to represent the degree to which a pixel belongs to a detection target. The higher the value is, the more it should belong to the target; the gradual edge transition allows spatial flexibility, overcoming the rigid restriction of the rectangular box, so that the model can more naturally fit the actual shape when generating the detection target.

[0118] The above-mentioned region soft mask is a continuous weight map at the pixel level, used to represent the degree to which a pixel belongs to a detection target. The higher the value is, the more it should belong to the target; the gradual edge transition allows spatial flexibility, overcoming the rigid restriction of the rectangular box, so that the model can more naturally fit the actual shape when generating the detection target.

[0119] ​​​​​​In generating the virtual target domain image, the global scene embedding vector and the conditional vector corresponding to each detection target and the region soft mask are input into the pre-trained instance-level conditional diffusion model. In the instance-level conditional diffusion model, the global scene embedding vector and the conditional vector corresponding to each detection target are injected as conditions into the cross-modal attention and / or normalization modulation module of the UNet (for example, the conditional vector corresponding to each detection target is projected as a conditional token and concatenated with the text token as the key / value of cross-attention, or the channel-level modulation of several UNet modules is performed through AdaLN / FiLM / LoRA), to achieve object-level control; at the same time, the region soft mask corresponding to each detection target guides the spatial attention in the instance-level conditional diffusion model. Specifically, any one or more of the following methods can be used:

[0120] (1) Attention bias: according to , an additive / multiplicative bias is applied to the scoring of self-attention or cross-attention, guiding the focus to the area where the target should appear;

[0121] (2) Channel concatenation: the up / down-sampled to the latent space resolution is concatenated as an additional channel with the UNet input or intermediate feature to participate in feature propagation;

[0122] (3) ControlNet control branch (optional): the is input into the ControlNet branch as a conditional graph (level-by-level with the UNet), and control features are injected into the main UNet by the control branch to enhance spatial controllability.

[0123] After step-by-step sampling of the diffusion process, the virtual target domain image output by the instance diffusion generation branch is obtained :

[0124]

[0125] wherein is the distribution of the image generated by the instance-level conditional diffusion model under given conditions , , , and is the model parameter.

[0126] Through the above mechanism, the virtual target domain image will contain each detection target In the image-level prompt depicts the background scene, according to The specified position rendering, and consistent with the semantic attributes of the text description The embodiment of the application retains global and local prompts for guidance at the same time, so that the generated virtual target domain image Not only has a real background under complex weather (controlled by And has the expected target appearance and state (controlled by And Compared with the original Instance Diffusion, the embodiment of the application introduces soft spatial constraints Enhance the flexibility and accuracy of positioning, thereby improving the precise control of the generated virtual target domain image The shape and position of the detection target.

[0127] It should be noted that the pre-trained instance-level conditional diffusion model in the embodiment of the application is obtained by training or fine-tuning the initial instance-level conditional diffusion model based on the training sample set and the loss function. The specific training process is not repeated here.

[0128] In an embodiment of the application, the frequency domain perturbation enhanced image and the virtual target domain image are fused to obtain a fused image, which can include:

[0129] The frequency domain perturbation enhanced image and the virtual target domain image are linearly weighted and fused in the pixel space to obtain a fused image.

[0130] Specifically, a linear weighting fusion strategy is adopted to fuse the frequency domain perturbation enhanced image Output by the frequency domain perturbation branch And the virtual target domain image output by the instance diffusion generation branch In the pixel space to obtain a fused image

[0131]

[0132] Wherein, Indicates the second fusion coefficient, Take a value biased towards the frequency domain branch (for example To ensure that the fused image retains more original image structure details, while moderately incorporating the target and style changes generated by diffusion. The fusion performed by the embodiment of the application only at the image level avoids complex alignment in the deep feature space, maintaining the simplicity and independence of the module. The fused image Combines the frequency domain perturbation based on real images and the content transformation generated based on instance-level conditional diffusion, enriching the diversity of detection target distribution.

[0133] In one embodiment of the present application, as shown in Figure 6 The physical enhancement processing on the fusion image to obtain the target enhanced image can include:

[0134] S601, performing global ambient light disturbance on the fusion image to obtain a degraded image;

[0135] S602, performing local directional disturbance on the fusion image to obtain a local directional disturbance image;

[0136] S603, performing fusion processing on the degraded image and the local directional disturbance image to obtain the target enhanced image.

[0137] Specifically, the fusion image is input into a physical enhancement branch, which simulates the influence of global light and transmission changes in the atmospheric physical imaging process on the image, and generates a degraded image by performing global ambient light disturbance on the fusion image . At the same time, the physical enhancement branch also simulates the local dynamic blur and noise effects caused by the weather, and generates a local directional disturbance image by performing local directional disturbance on the fusion image .

[0138] The degraded image and the local directional disturbance image are fused in the pixel space by using a linear weighted fusion strategy, thereby generating the final target enhanced image . The target enhanced image processed through the above-mentioned physical two-stage enhancement in cooperation presents the imaging characteristics under complex weather conditions in a visually realistic manner.

[0139] In one embodiment of the present application, as shown in Figure 7 The global ambient light disturbance on the fusion image to obtain the degraded image can include:

[0140] S701, performing Retinex decomposition on the fusion image by using a lightweight convolutional neural network for self-supervised learning to obtain a predicted illumination map;

[0141] S702, generating a detail reflection map according to the fusion image and the predicted illumination map;

[0142] S703, approximately generating the degraded image according to the detail reflection map, an RGB three-channel transmittance field and an ambient light three-channel value by using a pre-trained atmospheric scattering model, wherein the RGB three-channel transmittance field is controlled by weather parameters, and the ambient light three-channel value is generated by a pre-set three-channel value generation model according to the weather parameters, and wherein the weather parameters are determined by a pre-set weather condition.

[0143] It should be noted that, in simulating the impact of global illumination and transmission variations on images during atmospheric physical imaging, unlike strategies that start with random perturbations of global / local illumination in the frequency domain, this embodiment of the invention employs image decomposition and learnable control based on Retinex theory to achieve physical enhancement.

[0144] It should be noted that Retinex theory is an important theory for explaining how the human visual system perceives the color and brightness of objects. Its core lies in distinguishing the influence of "inherent properties of objects" and "lighting conditions" in an image, helping to understand why humans can stably identify the color and brightness of objects under different lighting conditions.

[0145] Specifically, lightweight CNNs (Convolutional Neural Networks) are used to process the fused images. Self-supervised Retinex decomposition, lightweight CNN outputs predicted illumination map (i.e., the slowly changing lighting components in the scene).

[0146] fused images With predicted illuminance map Perform pixel-by-pixel subtraction, i.e. , to obtain detailed reflection map The embodiments of the present invention yield detailed reflection images. This approach replaces the traditional Retinex multiplication model, making subsequent processing more convenient. (Predicted illumination map) It captures the overall light intensity distribution of the image, while the detailed reflectance map... Preserving object texture details, these two correspond to the ambient light effect and scene inherent radiance in the atmospheric scattering model, respectively.

[0147] To simulate light propagation attenuation under complex weather conditions, this embodiment of the invention introduces an RGB three-channel transmittance field. To replace the traditional single transmittance This allows different wavelengths of light to have different transmittances, illustrating phenomena such as the stronger attenuation of blue light by haze. The RGB three-channel transmittance field is determined by weather parameters. Control, weather parameters The value can be set according to preset weather conditions (such as fog, rain, dust storms). Alternatively, it can be determined using a simple model based on weather parameters. Calculate the RGB three-channel transmittance field This makes the attenuation level of each channel related to weather parameters. The corresponding physical characteristics are consistent.

[0148] This invention employs a small network. (Preset three-channel value generation model) Predicts ambient light three-channel values , instead of the fixed ambient light constant in the past. The ambient light color and intensity can be outputted reasonably according to the preset weather parameter , for example, the preset weather parameter indicates a sandstorm, The ambient light of warm color is outputted (R, G is relatively higher, B is relatively lower), and the brightness is determined by the weather intensity and the scene illumination, and the transmittance field is reduced, so that the relative contribution of the term in is increased. It should be noted that, in the parameterization of the color and brightness of the ambient light A, it can be expressed as: , , wherein is the chrominance (warm color is yellow: R, G is high, B is low), is the brightness (determined by the weather intensity and the scene illumination, not necessarily increased).

[0149] With the detailed reflectance map , the RGB three-channel transmittance field (wherein, ), and the ambient light three-channel value (wherein, ), the embodiment of the present application uses an extended atmospheric scattering model (a pre-trained atmospheric scattering model) to approximately generate a degraded image . Specifically,

[0150]

[0151] , wherein represents the pixel value of the degraded image in the channel. The above formula performs exponential decay on the reflectance in each color channel (controlled by ) and superimposes the atmospheric ambient light . By adjusting the distribution and value through the preset weather parameter , the visibility and color shift under different weather conditions can be flexibly simulated, so that the degraded image meets the expected physical degradation effect.

[0152] In an embodiment of the present application, as shown in Figure 8 , the local direction perturbation is performed on the fused image to obtain a local direction perturbation image, which can include:

[0153] S801, extracting gradient information of the fused image from the fused image;

[0154] S802, estimates the optical flow information of the fused image;

[0155] S803 estimates the dominant orientation of the fused image based on gradient and optical flow information;

[0156] S804 applies a dominant-direction convolutional blur to the fused image and adds noise stripes in the dominant direction to obtain a locally oriented perturbation image.

[0157] Specifically, a preset gradient extraction algorithm is used to extract the fused image. Gradient information is used to capture the main edge directions of the fused image.

[0158] It should be noted that in video sequence scenarios, optical flow can be used to estimate scene motion trends. Therefore, if the original image has a corresponding video sequence, the optical flow information of the original image can be calculated based on the video sequence, and then associated with this optical flow information in the fused image generated from the original image. This allows for the estimation of the optical flow information in the fused image based on the optical flow information of the original image. If the original image does not have a corresponding video sequence, the optical flow information of the fused image can be estimated using a single-frame optical flow estimation network or a single-frame motion prior algorithm.

[0159] Integrated fusion image The gradient and optical flow information can be used to estimate the fused image. Dominant movement / wind direction angle (Dominant direction). Then proceed along the dominant direction. For fused images Apply dominant direction Convolutional blur (simulating motion blur or windblown rain streaks along this direction) and adding blur along the dominant direction Consistent noise stripes yield a locally oriented perturbation image. It should be noted that directional processing can simulate elongated stripes caused by rain or snow falling at high speeds at specific angles, as well as motion blur effects caused by camera shake or object movement. The final image shows a locally oriented perturbation. It can be represented as:

[0160]

[0161] in, Indicates along the dominant direction Fuzzy transform operator, Indicates along the dominant direction Noise mapping of distribution.

[0162] Based on the original image acquired on a sunny day, the data augmentation method of this invention is used to generate an enhanced target image for a snowy scene. See [link to relevant documentation].Figure 9 The original image collected in a sunny day is shown in Fig. Figure 9 (a) of Fig. Figure 9 (b) of Fig. Figure 9 (c) of Fig.

[0163] Based on the original image collected in a sunny day, the data enhancement method of the embodiment of the present application is used to generate the target enhanced image of the thick fog and snow day scene, as shown in Fig. Figure 10 The original image collected in a sunny day is shown in Fig. Figure 10 (a) of Fig. Figure 10 (b) of Fig. Figure 10 (c) of Fig.

[0164] The data enhancement method of the embodiment of the present application adopts multi-mechanism fusion data enhancement, which significantly improves the robustness and accuracy of the power inspection target detection model in complex and variable environments. On the one hand, combining frequency domain transformation disturbance enhancement, controlled instance diffusion generation and physical simulation enhancement and other means, the imaging degradation effect of rain, fog, snow and other bad weather is accurately simulated, the distribution diversity of the training data is effectively expanded, and the problem of lack of real bad weather data is solved. On the other hand, based on the training sample set expanded by the embodiment of the present application, the target detection model trained has a significantly improved detection accuracy under extreme weather conditions, and the false detection rate and the missed detection rate are significantly reduced. For example, under thick fog and heavy rain conditions, the previous model is prone to miss detection of equipment due to reduced contrast or increased noise, and the data enhancement method of the embodiment of the present application can generate realistic fog / rain training samples to enable the target detection model to learn relevant features in advance, thereby maintaining a high detection rate in actual fog / rain weather.

[0165] Since data enhancement covers the imaging differences of multiple imaging devices and sensors (such as the resolution, noise level, spectral response of different cameras, etc.), the adaptability of the target detection model to multiple source device conditions is also greatly improved. In other words, even if deployed on different types of inspection terminals (unmanned aerial vehicle aerial photography, fixed monitoring, etc.), the target detection model can still maintain stable detection performance on images collected by new devices, reducing the need to train a model for each device separately. It is worth emphasizing that the data enhancement method of the embodiment of the present application adopts an independent and pluggable design, has low computational overhead, and can be executed in real time on edge computing devices, without affecting the real-time requirements of the inspection task.

[0166] The data enhancement method of the embodiment of the present application effectively makes up for the shortcomings of existing training data in bad environments, making the target detection system more reliable and stable in actual power inspection applications.

[0167] The application provides a data enhancement device.

[0168] Figure 11 is a schematic diagram of the data enhancement device of an embodiment of the application. As shown in Figure 11 The data enhancement device 100 can include a frequency domain disturbance module 10, a diffusion generation module 20, a fusion module 30, and a physical enhancement module 40.

[0169] The frequency domain disturbance module 10 is configured to perform frequency domain disturbance enhancement on an original image to obtain a frequency domain disturbance enhanced image, wherein the original image includes at least one detection target; the diffusion generation module 20 is configured to determine a global scene embedding vector, a condition vector corresponding to each detection target, and a region soft mask according to a global scene hint and a target level hint, position information, and a region mask corresponding to the at least one detection target, and perform diffusion generation according to the global scene embedding vector, the condition vector corresponding to each detection target, and the region soft mask by using a pre-trained instance level conditional diffusion model to obtain a virtual target domain image; the fusion module 30 is configured to perform fusion processing on the frequency domain enhanced image and the virtual target domain image to obtain a fusion image; and the physical enhancement module 40 is configured to perform physical enhancement processing on the fusion image to obtain a target enhanced image.

[0170] The data enhancement device of the embodiment of the application is a pluggable data enhancement module composed of three branches. Specifically, the data enhancement device includes a frequency domain disturbance branch, an instance diffusion generation branch, and a physical imaging simulation enhancement branch. The frequency domain disturbance branch and the instance diffusion generation branch generate respective enhanced images, and then obtain an intermediate enhancement result (fusion image) through linear weighted fusion. Finally, the fusion image is input into the physical enhancement branch to simulate complex weather effects, and an ultimate enhancement result, i.e., a target enhanced image, is output.

[0171] The embodiment of the application generates a target enhanced image by using a frequency domain transformation, instance diffusion generation, and physical simulation guidance mode for an original image.

[0172] In order to enable a target detection model to obtain good generalization detection performance even in the case of lacking real harsh weather data and meet actual deployment requirements, the data enhancement method including multiple branches constructed by using the embodiment of the application is used to expand the training data distribution by comprehensively using frequency domain disturbance, instance diffusion generation, and physical imaging simulation enhancement means, taking the power inspection scene as an application background and taking power equipment as a detection target.

[0173] It should be noted that other specific implementation manners of the data enhancement device provided by the embodiment of the application can refer to other specific implementation manners of the data enhancement method of the above-mentioned embodiments of the application.

[0174] The data enhancement device of the embodiment of the present application can accurately simulate imaging degradation effects of adverse weather such as rain, fog and snow by combining frequency domain transformation disturbance enhancement, controlled instance diffusion generation and physical simulation enhancement, and effectively expand distribution diversity of training data, thereby solving the problem of lack of real adverse weather data.

[0175] The present application provides a computer readable storage medium.

[0176] In this embodiment, the computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the data enhancement method as described above.

[0177] The present application provides an electronic device.

[0178] In this embodiment, the electronic device can include a memory and a processor, and the memory stores a computer program, wherein the computer program is executed by the processor to implement the data enhancement method as described above.

[0179] Figure 12 The present application provides an electronic device.

[0180] As shown in Figure 12 , the electronic device 500 includes a processor 501 and a memory 503. The processor 501 and the memory 503 are connected, such as through a bus 502. Optionally, the electronic device 500 can also include a transceiver 504. It should be noted that in actual application, the transceiver 504 is not limited to one, and the structure of the electronic device 500 does not constitute a limitation on the embodiments of the present application.

[0181] The processor 501 can be a CPU (Central Processing Unit, central processor), a general-purpose processor, a DSP (Digital Signal Processor, digital signal processor), an ASIC (Application Specific Integrated Circuit, application specific integrated circuit), an FPGA (Field Programmable Gate Array, field programmable gate array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It can implement or execute various exemplary logical blocks, modules and circuits described in combination with the present disclosure. The processor 501 can also be a combination of computing functions, such as one or more microprocessor combinations, combinations of DSP and microprocessor, etc.

[0182] The bus 502 can include a path for transmitting information between the above components. The bus 502 can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, or the like. The bus 502 can be divided into an address bus, a data bus, a control bus, and the like. For convenience of representation, Figure 12 Only one thick line is used to represent the bus in the middle, but it does not mean that there is only one bus or only one type of bus.

[0183] The memory 503 is used to store a computer program corresponding to the data enhancement method of the above-mentioned embodiments of the present application, which is controlled and executed by the processor 501. The processor 501 is used to execute the computer program stored in the memory 503 to realize the content shown in the above-mentioned method embodiments.

[0184] The electronic device 500 includes, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (Personal Digital Assistant), a PAD (Tablet Personal Computer), a PMP (Portable Multimedia Player), a vehicle-mounted terminal (such as a vehicle-mounted navigation terminal), and the like, and a fixed terminal such as a digital TV, a desktop computer, and the like. Figure 12 The electronic device 500 shown is only an example, and should not bring any limitation to the function and use range of the embodiments of the present application.

[0185] The computer readable storage medium and the electronic device of the embodiments of the present application can accurately simulate the imaging degradation effect of bad weather such as rain, fog, and snow based on the above-mentioned data enhancement method, effectively expand the distribution diversity of the training data, and solve the problem of lack of real bad weather data.

[0186] It is to be appreciated that the above description and the examples that follow are intended to be illustrative only and that changes can be made to the description and examples without departing from the scope of the application. Note also that the use of particular brand names in the description is solely for illustration and should not be construed as an endorsement of such brands.

[0187] It should be understood that aspects of the application can be implemented in hardware, software, firmware or a combination thereof. In the above embodiments, various steps or methods can be implemented in software or firmware that is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any of the following technologies, known in the art, can be used: a hybrid of the technologies mentioned above, discrete logic circuitry having logic gates for implementing logic functions upon an application of data signals, application specific integrated circuits having appropriate combinational logic gates, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0188] In the description of the present application, reference has been made to the use of terms such as "one embodiment", "some embodiments", "an example", "a specific example" or "some examples" means that a particular feature, structure, material or characteristic is included in at least one embodiment or example of the present application. The illustrative examples given are not necessarily to be construed as preferred or advantageous or with the exclusion of other equally valid examples that can be particularly adapted to a given application. Moreover, such illustrative examples are not necessarily mutually exclusive as the various aspects of the application can be combined in any suitable manner.

[0189] In the description of the application, it should be understood that the orientation or positional relationship indicated by the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the application and simplifying the description, and does not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the application.

[0190] In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the application, the meaning of "a plurality of" is at least two, such as two, three, etc., unless otherwise explicitly specified and limited.

[0191] In the present application, unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connecting", "fixing" and the like should be broadly understood, for example, it can be fixedly connected, or it can be detachably connected, or it can be integrated; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the internal communication of two elements or the interaction relationship between two elements, unless otherwise explicitly limited. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0192] In the present application, unless otherwise explicitly specified and limited, the first feature is "on" or "under" the second feature, which can be direct contact between the first and second features, or indirect contact between the first and second features through an intermediate medium. Moreover, the first feature "above", "above" and "above" the second feature can be directly above or obliquely above the first feature, or only indicate that the horizontal height of the first feature is higher than that of the second feature. The first feature "below", "below" and "below" the second feature can be directly below or obliquely below the first feature, or only indicate that the horizontal height of the first feature is less than that of the second feature.

[0193] Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and cannot be understood as a limitation on the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A data augmentation method, characterized in that, The method includes: The original image is enhanced by frequency domain perturbation to obtain a frequency domain perturbation enhanced image, wherein the original image includes at least one detection target; Based on the global scene cues and the target-level cues, location information and region mask corresponding to at least one of the detected targets, determine the global scene embedding vector and the condition vector and region soft mask corresponding to each of the detected targets; Using a pre-trained instance-level conditional diffusion model, a virtual target domain image is generated by diffusion based on the global scene embedding vector, the conditional vectors corresponding to each of the detected targets, and the region soft mask. The frequency domain perturbation enhanced image and the virtual target domain image are fused to obtain a fused image; The fused image is then subjected to physical enhancement processing to obtain the target enhanced image; The step of performing frequency domain perturbation enhancement on the original image to obtain a frequency domain perturbation enhanced image includes: The original image is enhanced by global random perturbation in the frequency domain to obtain a globally frequency-domain enhanced image. The local target regions corresponding to each detected target in the original image are perturbed and enhanced to obtain a local mask-enhanced image; The global frequency domain enhanced image and the local mask enhanced image are fused to obtain a frequency domain perturbation enhanced image; The physical enhancement process performed on the fused image to obtain the target enhanced image includes: Using a pre-trained atmospheric scattering model, the fused image is subjected to global ambient light perturbation to obtain a degraded image; The fused image is subjected to local directional perturbation to obtain a locally directional perturbation image; The degraded image and the locally oriented perturbation image are fused together to obtain the target enhancement image.

2. The data augmentation method according to claim 1, characterized in that, The step of performing global random perturbation enhancement on the original image in the frequency domain to obtain a globally frequency-domain enhanced image includes: Perform a Fourier transform on the original image to obtain a frequency domain image; Based on preset weather conditions, the frequency domain image is subjected to frequency domain filtering and random perturbation processing to obtain a weather perturbation image; Perform an inverse Fourier transform on the weather disturbance image to obtain a global frequency domain enhanced image; The step of perturbing and enhancing the local target regions corresponding to each detected target in the original image to obtain a locally masked enhanced image includes: Using a pre-trained image segmentation model, region masks corresponding to each detection target in the original image are extracted based on the bounding boxes corresponding to each detection target in the original image, resulting in a set of region masks. Based on the set of region masks, the original image is divided into background and target foreground to obtain a background image and a target foreground image corresponding to each detected target. The background image is obtained by removing all local target regions corresponding to the detected targets from the original image, and the target foreground image is obtained by multiplying the original image with the target mask of the corresponding detected target pixel by pixel. Random data augmentation processing is performed on the background image and each of the target foreground images to obtain a background enhanced image and each target foreground enhanced image; The enhanced foreground images of each target are superimposed on the enhanced background image to obtain a local mask enhancement image.

3. The data augmentation method according to claim 1, characterized in that, The global scene cues and the target-level cues are text cues. The step of determining the global scene embedding vector and the conditional vector and soft mask corresponding to each of the detected targets based on the global scene cues, the target-level cues corresponding to at least one of the detected targets, the location information, and the region mask includes: Using a pre-trained text encoder, the global scene cues and the target-level cues corresponding to at least one of the detected targets are converted into global scene embedding vectors and target-level embedding vectors corresponding to each of the detected targets, respectively. The global scene cues are determined by a preset scene, and the target-level cues are determined by the attributes of the corresponding detected targets. A two-dimensional Fourier position coding method is used to convert the position information corresponding to at least one of the detected targets into a position vector, thereby obtaining the position vector corresponding to each of the detected targets; The target-level embedding vector and position vector corresponding to each of the detection targets are connected, and the connection results are mapped using a multilayer perceptron to obtain the condition vector corresponding to each of the detection targets. The region mask corresponding to at least one of the detection targets is blurred to obtain a soft mask for each of the detection targets.

4. The data augmentation method according to claim 1, characterized in that, The process of fusing the frequency domain perturbation enhanced image and the virtual target domain image to obtain a fused image includes: The frequency domain perturbation enhanced image and the virtual target domain image are linearly weighted and fused in pixel space to obtain the fused image.

5. The data augmentation method according to claim 1, characterized in that, The process of using a pre-trained atmospheric scattering model to perform global ambient light perturbation on the fused image to obtain a degraded image includes: A lightweight convolutional neural network is used to perform self-supervised Retinex decomposition on the fused image to obtain a predicted illumination map. A detailed reflectance map is generated based on the fused image and the predicted illumination map; Using a pre-trained atmospheric scattering model, the degraded image is approximately generated based on the detailed reflectance map, the RGB three-channel transmittance field, and the ambient light three-channel values. The RGB three-channel transmittance field is controlled by weather parameters, and the ambient light three-channel values ​​are generated by a preset three-channel value generation model based on weather parameters, wherein the weather parameters are determined by preset weather conditions. The step of performing local directional perturbation on the fused image to obtain a locally directional perturbation image includes: Extract gradient information from the fused image; Estimate the optical flow information of the fused image; The dominant orientation of the fused image is estimated based on the gradient information and the optical flow information; The fused image is subjected to convolutional blur in the dominant direction and noise stripes in the dominant direction are added to obtain the local directional perturbation image.

6. A data augmentation device, characterized in that, The device includes: A frequency domain perturbation module is used to perform frequency domain perturbation enhancement on the original image to obtain a frequency domain perturbation enhanced image, wherein the original image includes at least one detection target; The diffusion generation module is used to determine the global scene embedding vector and the condition vector and region soft mask corresponding to each of the detection targets based on the global scene cues and the target-level cues, location information and region mask corresponding to at least one of the detection targets. It then uses a pre-trained instance-level conditional diffusion model to perform diffusion generation based on the global scene embedding vector and the condition vector and region soft mask corresponding to each of the detection targets to obtain a virtual target domain image. The fusion module is used to fuse the frequency domain perturbation enhancement image and the virtual target domain image to obtain a fused image; The physical enhancement module is used to perform physical enhancement processing on the fused image to obtain the target enhanced image; The frequency domain perturbation module is used to perform global random perturbation enhancement on the original image in the frequency domain to obtain a global frequency domain enhanced image; to perform perturbation enhancement on the local target regions corresponding to each detected target in the original image to obtain a local mask enhanced image; and to perform fusion processing on the global frequency domain enhanced image and the local mask enhanced image to obtain a frequency domain perturbation enhanced image. The physical enhancement module is used to perform global ambient light perturbation on the fused image using a pre-trained atmospheric scattering model to obtain a degraded image; perform local directional perturbation on the fused image to obtain a local directional perturbation image; and perform fusion processing on the degraded image and the local directional perturbation image to obtain the target enhancement image.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the data augmentation method as described in any one of claims 1-5.

8. An electronic device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the computer program is executed by the processor, it implements the data augmentation method as described in any one of claims 1-5.

Images