Image expansion method, system and device, storage medium and product

Abstract

The invention discloses an image expansion method, system and device, a storage medium and a product, and relates to the technical field of image enhancement, and the method comprises the steps: inputting an original image into a pre-constructed generative adversarial network, and obtaining a virtual image consistent with the features of the original image; and according to a randomly generated pixel grid, fusing the original image and the virtual image to obtain a fused image. According to the embodiment of the invention, new feature information can be introduced on the basis of keeping the original features of the real image, and the available image set is greatly enriched.

Description

Technical Field

[0001] This invention relates to the field of image enhancement technology, and in particular to an image augmentation method, system, device, storage medium, and product. Background Technology

[0002] More and more business platforms are starting to use more complex image verification codes for human-machine verification, thereby effectively intercepting machine behavior and ensuring the first line of defense for business security.

[0003] Relying solely on manual collection of image verification codes is time-consuming and labor-intensive. Existing image augmentation solutions rely on artificial intelligence models, which aim to generate virtual images that are consistent with real images. However, the number of virtual images generated is limited, and it is difficult to guarantee that all virtual images have high-quality usability. Summary of the Invention

[0004] The embodiments of the present invention aim to provide an image augmentation method, system, device, storage medium and product that can introduce new feature information while preserving the original features of the real image, thus greatly enriching the available image set.

[0005] In a first aspect, embodiments of the present invention provide an image augmentation method, comprising: The original image is input into a pre-constructed generative adversarial network to obtain a virtual image with features consistent with the original image; The original image and the virtual image are fused together based on a randomly generated pixel grid to obtain a fused image.

[0006] As an improvement to the above scheme, the generative adversarial network is a deep convolutional generative adversarial network; the deep convolutional generative adversarial network constructs a generator through transposed convolutional layers and a discriminator through convolutional layers.

[0007] As an improvement to the above scheme, the improved method for the convolution kernel in the generator includes: The input features of the transposed convolutional layer are subjected to global average pooling to obtain a compressed feature vector; The compressed feature vector is mapped to a low-dimensional space to obtain a low-dimensional feature vector; The attention weights of the low-dimensional feature vector in the position dimension, channel dimension, filter dimension and kernel dimension are calculated in parallel through multi-head attention branches. The attention weights are progressively applied to the convolution kernel of the transposed convolutional layer to adjust the convolution kernel, resulting in an adjusted convolution kernel. The adjusted convolution kernel is convolved with the input features to obtain the output features of the convolution kernel.

[0008] As an improvement to the above scheme, the step of inputting the original image into a pre-constructed generative adversarial network to obtain a virtual image with features consistent with the original image includes: Obtain the original image and annotate the original image; Based on the original image and its annotations, supervised data augmentation is performed on the original image to obtain augmented training samples; The generative adversarial network is trained using the original image and the enhanced training samples, so that the generative adversarial network can generate virtual images with features consistent with the original image based on random noise.

[0009] As an improvement to the above scheme, the step of fusing the original image and the virtual image based on a randomly generated pixel grid to obtain a fused image includes: Based on the detection target, a pixel grid is randomly generated; Based on the pixel grid, set a filter with the same size as the original image; Calculate the first product of the filter and the pixel values ​​of the original image, and the second product of the negative mask of the filter and the pixel values ​​of the virtual image; Add the first product to the second product to obtain the first fused image; Calculate the third product of the filter and the pixel values ​​of the virtual image, and the fourth product of the negative mask of the filter and the pixel values ​​of the original image; The third product is added to the fourth product to obtain the second fused image.

[0010] As an improvement to the above solution, the step of setting a filter with the same size as the original image based on the pixel grid includes: An initial filter with the same size as the original image is set; the weight coefficient of each pixel in the initial filter is 1; Based on the pixel grid, the weight coefficients of the corresponding pixels in the initial filter are set to 0 to obtain the filter for image fusion.

[0011] Secondly, embodiments of the present invention provide an image augmentation system, comprising: A virtual image generation module is used to input the original image into a pre-constructed generative adversarial network to obtain a virtual image with features consistent with the original image; The image fusion module is used to fuse the original image and the virtual image according to a randomly generated pixel grid to obtain a fused image.

[0012] Thirdly, embodiments of the present invention provide an image augmentation device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the image augmentation method as described above.

[0013] Fourthly, embodiments of the present invention provide a computer-readable storage medium, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the image augmentation method as described above.

[0014] Fifthly, embodiments of the present invention provide a computer program product, the computer program product including a computer program or computer instructions, wherein when the computer program or computer instructions are executed by a processor, the image augmentation method described above is performed.

[0015] Compared with existing technologies, the image augmentation method, system, device, storage medium, and product disclosed in this invention obtain a virtual image with features consistent with the original image by inputting the original image into a pre-constructed generative adversarial network; the original image and the virtual image are then fused according to a randomly generated pixel grid to obtain a fused image. By employing embodiments of this invention, new feature information can be introduced while preserving the original features of the real image, greatly enriching the available image set. Attached Figure Description

[0016] Figure 1 This is a schematic flowchart of the steps of an image augmentation method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of an improved architecture for convolutional kernels in a generator provided by an embodiment of the present invention; Figure 3 This invention provides a supervised data augmentation method applied to original images. Figure 4 This is another supervised data augmentation method for original images provided in this embodiment of the invention. Figure 5 This is a schematic diagram illustrating the steps and results of fusing an original image with a virtual image according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of an image augmentation system provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of an image augmentation device provided in an embodiment of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] In the description and claims, it should be understood that the terms "first," "second," etc., used in the description and claims are only for the purpose of distinguishing the description of the same technical features, and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated, nor necessarily the order of description or chronological order. The terms are interchangeable where appropriate. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature.

[0019] This invention provides an image augmentation method. Please refer to [link to relevant documentation]. Figure 1 In this embodiment, the image augmentation method is specifically executed through steps S1 to S2: S1. Input the original image into a pre-constructed generative adversarial network to obtain a virtual image with features consistent with the original image; S2. Based on the randomly generated pixel grid, the original image and the virtual image are fused to obtain a fused image.

[0020] Image enhancement has diverse applications. For example, with the rapid development of artificial intelligence technology, more and more hacking techniques are being used for security breaches, posing a serious challenge to business security. Therefore, many business platforms set up complex image CAPTCHAs for human-machine verification to effectively intercept machine behavior.

[0021] These image-based CAPTCHA verifications typically require users to manually select a specific target. Therefore, business platforms need to determine whether the CAPTCHA contains the specific target and its location within the image to achieve verification. Manually collecting and labeling CAPTCHAs would be extremely time-consuming; image augmentation is therefore highly valuable for expanding the scope of image CAPTCHAs. Furthermore, image augmentation can be widely applied in fields such as deep learning.

[0022] Generative Adversarial Networks (GANs) are an unsupervised data augmentation method that learns the distribution of the original data and then randomly generates samples with the same feature distribution as the original data, thus augmenting unlabeled data.

[0023] Generative Adversarial Networks (GANs) primarily consist of two neural networks: a generator and a discriminator. The discriminator aims to distinguish between real and fake samples; the generator takes a random noise vector as input and creates fake samples that appear to have the same distribution as the real data and can pass through the discriminator. Through the competition between the generator and discriminator in the GAN, the generated virtual images can learn the features and distribution characteristics of the original image.

[0024] In some preferred embodiments, the generative adversarial network is pre-trained using a public dataset that matches the target domain. During training, the discriminator determines whether the generated samples produced by the generator are real based on real training samples. In this process, the generator can learn how to output images that are consistent with real training samples in the feature space based on noise signals.

[0025] Generative Adversarial Networks (GANs) trained on public datasets can be applied to data augmentation for all their corresponding targets, forming a general-purpose GAN. However, considering that the original images may differ from those in the public dataset, the general-purpose GAN can be further fine-tuned using the original images to ensure that it can generate virtual images with features consistent with the original images.

[0026] In some other preferred embodiments, the original image is directly used to train the generative adversarial network to obtain a targeted generative adversarial network, which is then used specifically to generate virtual images of the same type as the original image.

[0027] It is understood that the specific training method of the generative adversarial network does not affect the beneficial effects produced by the embodiments of the present invention. It can be fine-tuning the training of a general GAN ​​or directly using the original image to train the pre-constructed generative adversarial network.

[0028] It should be noted that while generative adversarial networks (GANs) can generate virtual images with features identical to the original image, the number of generated virtual images is limited, and directly applying virtual images carries certain quality risks. Therefore, in this embodiment of the invention, a pixel grid is introduced to fuse the original and virtual images, further expanding the number of images while improving image quality.

[0029] The pixel grid is a binary mask matrix with a customizable size, the size and shape of which are determined by the characteristics of the detected target. By randomly generating the pixel grid, it is possible to fuse the pixels at the same locations in the original and virtual images, thereby introducing perturbations in local regions of potential targets in the image. This can simulate diversity while preserving the overall style of the image.

[0030] For example, when the target in the original image is small, the number of pixel grids should be increased while the grid size is decreased; conversely, when the target is large, the size of the pixel grids should be increased while the number of grids is decreased, thereby reducing the possibility of the complete target being segmented in the image. This pixel grid generation scheme is highly flexible and adaptable, and can dynamically adjust the size and number of pixel grids according to different image types.

[0031] The above scheme can introduce new feature information while preserving the original features of the real image, which greatly enriches the available image set.

[0032] In a preferred embodiment, the generative adversarial network is a deep convolutional generative adversarial network; the deep convolutional generative adversarial network constructs a generator through transposed convolutional layers and a discriminator through convolutional layers.

[0033] Deep Convolution Generative Adversarial Network (DDCGAN) is a generative adversarial network that uses a convolutional neural network architecture. Its generator and discriminator are mainly composed of convolution operations.

[0034] The transposed convolutional layer in the generator progressively upsamples the low-dimensional features of random noise in space, and finally outputs a virtual image that matches the size of the original image; the convolutional layer in the discriminator progressively downsamples the input original image or virtual image in space, and finally outputs the probability that the image is a real image.

[0035] In the above scheme, the deep convolutional generative adversarial network uses a convolutional neural network instead of a multilayer perceptron in a regular generative adversarial network. At the same time, by leveraging the integration of convolutional layers, batch normalization, and ReLU activation functions in the deep convolutional generative adversarial network, the generated virtual images have higher quality.

[0036] Furthermore, in this embodiment of the invention, the convolution kernel in the generator is also improved. Preferably, the method for improving the convolution kernel in the generator includes: The input features of the transposed convolutional layer are subjected to global average pooling to obtain a compressed feature vector; The compressed feature vector is mapped to a low-dimensional space to obtain a low-dimensional feature vector; The attention weights of the low-dimensional feature vector in the position dimension, channel dimension, filter dimension and kernel dimension are calculated in parallel through multi-head attention branches. The attention weights are progressively applied to the convolution kernel of the transposed convolutional layer to adjust the convolution kernel, resulting in an adjusted convolution kernel. The adjusted convolution kernel is convolved with the input features to obtain the output features of the convolution kernel.

[0037] It should be noted that the input features of the transposed convolutional layer are the feature vectors of the input noise, which are upsampled through the transposed convolutional layer.

[0038] By performing global average pooling on the input features, the average value of the input features across multiple dimensions can be obtained to produce a compressed feature vector with the same number of channels as the input features, providing global guidance for subsequent dynamic adjustment of the convolution kernel.

[0039] Directly using compressed feature vectors for multi-head attention computation may be too large and redundant. Therefore, in this embodiment of the invention, they are further mapped to a low-dimensional space, which can reduce the feature dimension and filter out redundant information as much as possible.

[0040] The multi-head attention branch used in this embodiment of the invention is a four-head mechanism. The four heads are responsible for calculating the attention weights of the low-dimensional feature vector in the position dimension, channel dimension, filter dimension and kernel dimension, respectively, and then generating a highly content-adaptive convolutional kernel.

[0041] It should also be noted that in the process of gradually applying attention weights to the convolution kernel of the transposed convolutional layer, the weights of the convolution kernel will be dynamically generated or adjusted according to the current input features, ultimately forming a dynamic convolutional structure.

[0042] For example, please see Figure 2 , Figure 2 A schematic diagram of an improved architecture for convolutional kernels in a generator is provided.

[0043] In this embodiment of the invention, the input features of the transposed convolutional layer are compressed using global average pooling (GAP). Subsequently, a fully connected (FC) layer maps the compressed feature vector to a reduced low-dimensional space of size r. For the four attention branches, each branch contains a fully connected (FC) layer and a sigmoid function, generating normalized features respectively. , , and Finally, the four types of attention scores are multiplied sequentially by the convolution kernel. This is how the output of the convolution kernel is obtained.

[0044] In the above scheme, by introducing a dynamic convolution kernel algorithm, the generative adversarial network can fully leverage the advantages of the convolution kernel to extract more complex background information from multiple dimensions, thereby generating higher quality images.

[0045] As a preferred implementation, step S1, inputting the original image into a pre-constructed generative adversarial network to obtain a virtual image with features consistent with the original image, includes: Obtain the original image and annotate the original image; Based on the original image and its annotations, supervised data augmentation is performed on the original image to obtain augmented training samples; The generative adversarial network is trained using the original image and the enhanced training samples, so that the generative adversarial network can generate virtual images with features consistent with the original image based on random noise.

[0046] The virtual images generated by generative adversarial networks depend on the quantity and quality of the original dataset. Limited training data may cause the model to fail to learn the true distribution of the data, resulting in overfitting.

[0047] Therefore, in this embodiment of the invention, the original image is augmented through supervised data training to improve the original samples that the generative adversarial network can learn. Then, the augmented original image and the enhanced training samples are used to train the generative adversarial network together to improve the feature extraction capability of the generative adversarial network, thereby improving the image quality.

[0048] Please see Figure 3 , Figure 3 This paper presents a supervised data augmentation method for original images. It first flips the original image horizontally, then vertically, followed by a 90° rotation and cropping, and finally scales the image to obtain augmented training samples. Figure 3 The main method used in this study is to augment the original image through geometric transformation.

[0049] Please see Figure 4 , Figure 4 Another supervised data augmentation method for original images is provided. It first adds salt-and-pepper noise and Gaussian noise sequentially to the original image, then performs sample filling and mixing through Cutout and Cutmix, and finally randomly discards image regions through Coarse Dropout to form augmented training samples.

[0050] Preferably, the supervised data augmentation method for the original image includes any one or a combination of color transformation, horizontal flip, vertical flip, rotation, cropping, noise enhancement, scaling, Cutout, Cutmix, and Coarse Dropout.

[0051] In the above scheme, the positive sample pool of the discriminator is effectively expanded by performing supervised enhancement on the original image, which provides richer supervision information for the training of the generative adversarial network, helps to stabilize training and improve the generation quality.

[0052] In a preferred embodiment, step S2, fusing the original image and the virtual image according to the randomly generated pixel grid to obtain a fused image, is performed through steps S21-S26: S21. Randomly generate a pixel grid based on the detection target; S22. Based on the pixel grid, set a filter with the same size as the original image; S23. Calculate the first product of the filter and the pixel values ​​of the original image, and the second product of the negative mask of the filter and the pixel values ​​of the virtual image; S24. Add the first product to the second product to obtain the first fused image; S25. Calculate the third product of the filter and the pixel values ​​of the virtual image, and the fourth product of the negative mask of the filter and the pixel values ​​of the original image; S26. Add the third product to the fourth product to obtain the second fused image.

[0053] By randomly generating pixel grids, the position, shape, and size of the fusion region may differ in each operation, thus introducing rich spatial variations. At the same time, the size and shape of the randomly generated pixel grids are adapted to the detection target, ensuring that the fusion region can effectively cover the possible positions and ranges of the target in the image, making the data augmentation operation more targeted.

[0054] In this embodiment of the invention, pixel grids of different sizes and quantities are preferably used to filter the original image and the virtual image. The size and quantity of the pixel grids can be dynamically adjusted according to different scenes and tasks to adapt to datasets with different characteristics. Furthermore, the pixel grids filling the fused image are all pixel values ​​of the real scene.

[0055] The size of the filter is the same as that of the original image, and of course, it is also the same as that of the virtual image. By materializing the pixel grid into a numerical filter that can be applied to the entire image space, the local fusion intention can be transformed into a global operator, enabling efficient image fusion through matrix operations of the filter.

[0056] In some preferred embodiments, step S22, setting a filter with the same size as the original image based on the pixel grid, includes: An initial filter with the same size as the original image is set; the weight coefficient of each pixel in the initial filter is 1; Based on the pixel grid, the weight coefficients of the corresponding pixels in the initial filter are set to 0 to obtain the filter for image fusion.

[0057] It is understood that the image fusion process in this embodiment of the invention is a symmetrical bidirectional design. Therefore, in practical applications, the weight coefficients of each pixel in the initial filter can be set to 0, and the weight coefficients of the corresponding pixels in the pixel grid can be set to 1. This process is to ensure the correctness of the deletion and padding logic.

[0058] In the above scheme, by first creating an initial filter of the same size as the image and then making local modifications, it is fundamentally guaranteed that the generated filter corresponds one-to-one with the pixel positions of the original image and the virtual image in the spatial dimension. This is a prerequisite for achieving accurate pixel-level fusion.

[0059] The negative mask of a filter refers to the mask matrix that is complementary to the filter, and its calculation is represented as 1 - filter. That is, where the value in the filter is 1, the corresponding negative mask is 0; conversely, where the value in the filter is 0, the corresponding negative mask is 1.

[0060] Steps S23-S26 involve two sets of symmetrical product and summation operations. In practice, one original image and one virtual image of the same type are selected sequentially.

[0061] Specifically, when generating the first fused image, some pixel grids are randomly deleted from the original image, and then pixel grids of the same position and size from the virtual image are used to fill the original image.

[0062] Correspondingly, when generating the second fused image, some pixel grids are randomly deleted from the virtual image, and then pixel grids of the same position and size from the original image are used to fill the virtual image.

[0063] In one embodiment, the original image and the virtual image are fused using the present invention, and the resulting fused image is as follows: Figure 5 As shown.

[0064] The two image fusion methods are inverse operations of each other, resulting in a complementary fused image. On the one hand, this enriches the diversity of generated samples, and on the other hand, it can better distinguish the essential features of the target from its accidental background.

[0065] In some preferred embodiments, the pixel values ​​of the first fused image at the (x,y) coordinates are represented as follows: ; The pixel values ​​of the second fused image at the (x,y) coordinates are represented as follows: ; in, The pixel value at (x, y) coordinates in the original image. This represents the filter value at the (x, y) coordinates; The pixel value at the (x,y) coordinates in the virtual image; y is the value of the negative mask of the filter at the (x,y) coordinates. This means that image pixels will be preserved. This indicates that the pixel should be discarded.

[0066] In this embodiment of the invention, for each original image-virtual image pair, two fused images can be generated, meaning the number of usable images increases from one to two or three. When there are m original images, and m virtual images of the same type as the original images are generated through a generative adversarial network, all generated images of the same type are sequentially selected and fused with the original images using a random real-fake data fusion method, ultimately resulting in m×n×2 fused images of the same type. Finally, the new image dataset formed by mixing the real images, generated images, and fused images is expanded by hundreds of times compared to the original images.

[0067] In the above scheme, the original dataset can be easily expanded by hundreds of times by fusing random real and fake data. The fused image not only retains the original features of the real image, but also introduces new feature information, which greatly enriches the original data sample.

[0068] The image augmentation method provided in this embodiment of the invention can introduce new feature information while preserving the original features of the real image, thus greatly enriching the available image set.

[0069] This invention provides an image augmentation system. Please refer to [link / reference]. Figure 6 The image augmentation system includes a virtual image generation module 11 and an image fusion module 12, wherein: The virtual image generation module 11 is used to input the original image into a pre-constructed generative adversarial network to obtain a virtual image with features consistent with the original image; The image fusion module 12 is used to fuse the original image and the virtual image according to the randomly generated pixel grid to obtain a fused image.

[0070] In a preferred embodiment, the generative adversarial network is a deep convolutional generative adversarial network; the deep convolutional generative adversarial network constructs a generator through transposed convolutional layers and a discriminator through convolutional layers.

[0071] Furthermore, preferably, the convolution kernel in the generator is improved, and the improvement method includes: The input features of the transposed convolutional layer are subjected to global average pooling to obtain a compressed feature vector; The compressed feature vector is mapped to a low-dimensional space to obtain a low-dimensional feature vector; The attention weights of the low-dimensional feature vector in the position dimension, channel dimension, filter dimension and kernel dimension are calculated in parallel through multi-head attention branches. The attention weights are progressively applied to the convolution kernel of the transposed convolutional layer to adjust the convolution kernel, resulting in an adjusted convolution kernel. The adjusted convolution kernel is convolved with the input features to obtain the output features of the convolution kernel.

[0072] In a preferred embodiment, the virtual image generation module 11 is specifically used for: Obtain the original image and annotate the original image; Based on the original image and its annotations, supervised data augmentation is performed on the original image to obtain augmented training samples; The generative adversarial network is trained using the original image and the enhanced training samples, so that the generative adversarial network can generate virtual images with features consistent with the original image based on random noise.

[0073] In a preferred embodiment, the image fusion module 12 is used for: Based on the detection target, a pixel grid is randomly generated; Based on the pixel grid, set a filter with the same size as the original image; Calculate the first product of the filter and the pixel values ​​of the original image, and the second product of the negative mask of the filter and the pixel values ​​of the virtual image; Add the first product to the second product to obtain the first fused image; Calculate the third product of the filter and the pixel values ​​of the virtual image, and the fourth product of the negative mask of the filter and the pixel values ​​of the original image; The third product is added to the fourth product to obtain the second fused image.

[0074] Further, preferably, setting a filter with the same size as the original image based on the pixel grid includes: An initial filter with the same size as the original image is set; the weight coefficient of each pixel in the initial filter is 1; Based on the pixel grid, the weight coefficients of the corresponding pixels in the initial filter are set to 0 to obtain the filter for image fusion.

[0075] The image augmentation system provided by this invention can introduce new feature information while preserving the original features of the real image, greatly enriching the available image set.

[0076] Please seeFigure 7 , Figure 7 This is a structural block diagram of an image augmentation device provided in an embodiment of the present invention. The image augmentation device includes a processor 31, a memory 32, and a computer program stored in the memory 32 and executable on the processor 31. When the processor 31 executes the computer program, it implements the steps in the above-described embodiments of the image augmentation methods, such as steps S1 to S2.

[0077] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 32 and executed by the processor 31 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the image augmentation device.

[0078] The image expansion device may include, but is not limited to, a processor 31 and a memory 32. Those skilled in the art will understand that the schematic diagram is merely an example of an image expansion device and does not constitute a limitation on the image expansion device. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the image expansion device may also include input / output devices, network access devices, buses, etc.

[0079] The processor 31 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 31 is the control center of the image expansion device, connecting various parts of the entire image expansion device through various interfaces and lines.

[0080] The memory 32 can be used to store the computer program and / or modules. The processor 31 implements various functions of the image expansion device by running or executing the computer program and / or modules stored in the memory 32 and calling the data stored in the memory 32. The memory 32 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 32 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital card (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0081] If the modules / units integrated into the image augmentation device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by the processor 31, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0082] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. An image augmentation method, characterized in that, include: The original image is input into a pre-constructed generative adversarial network to obtain a virtual image with features consistent with the original image; The original image and the virtual image are fused together based on a randomly generated pixel grid to obtain a fused image.

2. The image augmentation method as described in claim 1, characterized in that, The generative adversarial network is a deep convolutional generative adversarial network; the deep convolutional generative adversarial network constructs a generator through transposed convolutional layers and a discriminator through convolutional layers.

3. The image augmentation method as described in claim 2, characterized in that, The improved method for the convolution kernel in the generator includes: The input features of the transposed convolutional layer are subjected to global average pooling to obtain a compressed feature vector; The compressed feature vector is mapped to a low-dimensional space to obtain a low-dimensional feature vector; The attention weights of the low-dimensional feature vector in the position dimension, channel dimension, filter dimension and kernel dimension are calculated in parallel through multi-head attention branches. The attention weights are progressively applied to the convolution kernel of the transposed convolutional layer to adjust the convolution kernel, resulting in an adjusted convolution kernel. The adjusted convolution kernel is convolved with the input features to obtain the output features of the convolution kernel.

4. The image augmentation method as described in claim 1, characterized in that, The step of inputting the original image into a pre-constructed generative adversarial network to obtain a virtual image with features consistent with the original image includes: Obtain the original image and annotate the original image; Based on the original image and its annotations, supervised data augmentation is performed on the original image to obtain augmented training samples; The generative adversarial network is trained using the original image and the enhanced training samples, so that the generative adversarial network can generate virtual images with features consistent with the original image based on random noise.

5. The image augmentation method as described in claim 1, characterized in that, The step of fusing the original image and the virtual image based on a randomly generated pixel grid to obtain a fused image includes: Based on the detection target, a pixel grid is randomly generated; Based on the pixel grid, set a filter with the same size as the original image; Calculate the first product of the filter and the pixel values ​​of the original image, and the second product of the negative mask of the filter and the pixel values ​​of the virtual image; Add the first product to the second product to obtain the first fused image; Calculate the third product of the filter and the pixel values ​​of the virtual image, and the fourth product of the negative mask of the filter and the pixel values ​​of the original image; The third product is added to the fourth product to obtain the second fused image.

6. The image augmentation method as described in claim 5, characterized in that, The step of setting a filter with the same size as the original image based on the pixel grid includes: An initial filter with the same size as the original image is set; the weight coefficient of each pixel in the initial filter is 1; Based on the pixel grid, the weight coefficients of the corresponding pixels in the initial filter are set to 0 to obtain the filter for image fusion.

7. An image augmentation system, characterized in that, include: A virtual image generation module is used to input the original image into a pre-constructed generative adversarial network to obtain a virtual image with features consistent with the original image; The image fusion module is used to fuse the original image and the virtual image according to a randomly generated pixel grid to obtain a fused image.

8. An image augmentation device, characterized in that, The device includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the image augmentation method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the image augmentation method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program or computer instructions, which, when executed by a processor, perform the image augmentation method as described in any one of claims 1 to 6.

Images