Small sample image generation method based on semantic mask reconstruction and category semantic learning

Abstract

The invention proposes a small sample image generation method based on semantic mask reconstruction and category semantic learning, and relates to the technical field of image processing, and the method comprises the steps: S1, constructing a network model comprising a generator and a discriminator based on a LoFGAN network; s2, extracting depth features of the input image by using a generator, and fusing the depth features of the input image into depth semantic features through local feature fusion; s3, semantic mask reconstruction is carried out on the depth semantic features, and a corresponding image is generated through a decoder; s4, sending a real image and a corresponding image generated by the decoder into a discriminator to train the network model; s5, inputting the small sample image into the trained network model to obtain a generated image similar to the input image; according to the method, the extraction precision of the semantic information can be effectively enhanced, the understanding ability of the generation model for a complex semantic scene is improved, and the fidelity and definition of the image are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a small sample image generation method based on semantic mask reconstruction and category semantic learning. Background Art

[0002] With the continuous development of machine learning related research, the application of related technologies in computer vision, natural language processing and other fields has become increasingly widespread, and methods and models with excellent performance have been continuously proposed. In recent years, due to the continuous leap in the computing performance of hardware devices such as GPUs and CPUs, and the emergence of advanced models and algorithms such as ResNet and BERT, machine learning methods have gradually moved from theory to practical application, enabling a large number of tasks in real-life scenarios and effectively changing everyone's life. However, the successful application of traditional machine learning methods depends on the detailed study of large-scale data, and the model training of machine learning methods requires a large amount of data. In practical applications, such as industrial product defect images, a large number of image samples that meet the requirements are often difficult to obtain, and due to the difficulty of data collection and the high cost of image annotation, the scale of image data available for training is often small, and there are even only a few training samples for each category. Therefore, the small sample problem has become one of the important research directions.

[0003] Although the current small sample image generation methods have made significant progress, they still face the problem of poor quality and insufficient diversity of generated images. Unlike models based on large-scale datasets, which can learn rich enough semantic information through training to generate images, in the case of small samples, the generated images often lack details and diversity because the model has difficulty in fully understanding the semantic information of the image and capturing rich enough semantic information, and lack the ability to extract high-quality semantic representations from the image, especially in the case of small samples. Summary of the invention

[0004] The purpose of the present invention is to provide a small sample image generation method based on semantic mask reconstruction and category semantic learning, which can effectively enhance the extraction accuracy of semantic information, improve the ability of the generation model to understand complex semantic scenes, and improve the fidelity and clarity of the image.

[0005] The technical solution of the present invention is:

[0006] In a first aspect, the present application provides a small sample image generation method based on semantic mask reconstruction and category semantic learning, which comprises the following steps:

[0007] S1. Build a network model including generator and discriminator based on LoFGAN network;

[0008] S2. Use the generator to extract the deep features of the input image, and fuse the deep features of the input image into deep semantic features through local feature fusion;

[0009] S3, reconstruct the semantic mask of the deep semantic features and generate the corresponding image through the decoder;

[0010] S4, sending the real image and the corresponding image generated by the decoder to the discriminator to train the network model;

[0011] S5. Input the small sample image into the trained network model to obtain a generated image similar to the input image.

[0012] Further, step S3 includes:

[0013] Generate random noise based on deep semantic features;

[0014] sorting the indices in each mask block based on random noise to determine pixels to be retained and pixels requiring mask operation;

[0015] Initialize the mask tensor with the mask rate to create a binary mask based on the pixels to be retained and required masking operation;

[0016] The generated binary mask is resized to the same size as the deep semantic feature using the nearest neighbor interpolation method;

[0017] The deep semantic features are multiplied element-wise with the resized mask and passed through the decoder to generate a corresponding image consistent with the deep semantic features.

[0018] Furthermore, in step S4, the process of training the network model by the discriminator includes training with maximizing the discriminative ability of the real image and the image generated by the decoder as the overall objective function, and the loss function includes:

[0019] Category semantic loss:

[0020]

[0021] Local reconstruction loss:

[0022]

[0023] Fighting Loss:

[0024]

[0025] Classification loss:

[0026]

[0027] In the formula, is the category semantic loss, is the square of the Euclidean distance, f(G) is the generated image feature, f(P) is the positive sample feature, f(N) is the negative sample feature, margin is the threshold, is the local reconstruction loss, To generate an image, α is a random coefficient vector, LFM(X,α) is the semantically aligned fusion feature obtained by the local fusion module LFM, is the adversarial loss of the generator, is the adversarial loss of the discriminator, D(X) is the discriminator’s judgment result on the real image X, Generate images for the discriminator pair The judgment result of , max(·) means maximization, All are classification losses, log(·) is the logarithmic loss function, P(·) is the probability, and c(X) is the true category of the sample.

[0028] Furthermore, the calculation formula of the above total objective function is:

[0029]

[0030] In the formula, is the objective optimization function of the generator, All are against losses, λ local , and λ csl are all hyperparameters for model optimization. are classification losses, is the local reconstruction loss, is the category semantic loss, is the objective optimization function of the discriminator, To combat losses.

[0031] Furthermore, step S4 also includes: optimizing network parameters by alternately training the generator and the discriminator to generate high-quality images, and using multiple indicators to evaluate the quality of the generated images.

[0032] In a second aspect, the present application provides an electronic device, including:

[0033] A memory for storing one or more programs;

[0034] processor;

[0035] When the one or more programs are executed by the processor, a small sample image generation method based on semantic mask reconstruction and category semantic learning as described in any one of the first aspects is implemented.

[0036] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a small sample image generation method based on semantic mask reconstruction and category semantic learning as described in any one of the first aspects above.

[0037] Compared with the prior art, the present invention has at least the following advantages or beneficial effects:

[0038] (1) The present invention provides a small sample image generation method based on semantic mask reconstruction and category semantic learning. The semantic-level mask helps the model learn high-quality semantic information. At the same time, a dynamic mask rate adjustment mechanism is introduced to allow the model to adaptively adjust the mask rate according to the characteristics of the input image during the training process, so that the masking process can more flexibly highlight the key semantic areas, thereby realizing adaptive learning for different scenes, and enabling the model to perform targeted masking according to the complexity of the semantic features, effectively enhancing the extraction accuracy of the semantic information and improving the ability of the generated model to understand complex semantic scenes;

[0039] (2) The present invention introduces the category semantic learning technology CSL to enhance the model's ability to capture subtle differences between categories, and adjusts the distance between the generated samples and the positive samples and negative samples, so that the distance between samples within a class is shortened and the distance between samples between classes is increased, thereby achieving effective separation of category semantics and improving the category consistency of the generated images. The model has a better recognition effect on fine-grained category differences in small sample generation, and significantly reduces category ambiguity during the generation process, thereby improving the fidelity and clarity of the image, making the generated image more realistic and having obvious category differentiation characteristics. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 A step diagram of a small sample image generation method based on semantic mask reconstruction and category semantic learning of the present invention;

[0042] Figure 2 This is the overall framework diagram of the small sample image generation model SMR-CSL based on semantic mask reconstruction and category semantic learning;

[0043] Figure 3 The figure is a schematic structural block diagram of an electronic device according to an embodiment of the present invention.

[0044] Icon: 101, memory; 102, processor; 103, communication interface. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0046] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0047] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0048] It should be noted that, in this article, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprises..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0049] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0050] Example 1

[0051] See also Figure 1 , Figure 1 Shown is a step diagram of a small sample image generation method based on semantic mask reconstruction and category semantic learning provided in an embodiment of the present application.

[0052] In a first aspect, the present application provides a small sample image generation method based on semantic mask reconstruction and category semantic learning, which comprises the following steps:

[0053] S1. Build a network model including generator and discriminator based on LoFGAN network;

[0054] S2. Use the generator to extract the deep features of the input image, and fuse the deep features of the input image into deep semantic features through local feature fusion;

[0055] S3, reconstruct the semantic mask of the deep semantic features and generate the corresponding image through the decoder;

[0056] S4, sending the real image and the corresponding image generated by the decoder to the discriminator to train the network model;

[0057] S5. Input the small sample image into the trained network model to obtain a generated image similar to the input image.

[0058] As a preferred implementation, step S3 includes:

[0059] Generate random noise based on deep semantic features;

[0060] sorting the indices in each mask block based on random noise to determine pixels to be retained and pixels requiring mask operation;

[0061] Initialize the mask tensor with the mask rate to create a binary mask based on the pixels to be retained and required masking operation;

[0062] The generated binary mask is resized to the same size as the deep semantic feature using the nearest neighbor interpolation method;

[0063] The deep semantic features are multiplied element-wise with the resized mask and passed through the decoder to generate a corresponding image consistent with the deep semantic features.

[0064] As a preferred implementation, in step S4, the process of training the network model by the discriminator includes training with maximizing the discriminative ability of the real image and the image generated by the decoder as the overall objective function, and the loss function includes:

[0065] Category semantic loss, by defining a triplet, selecting K images of the same category as positive samples, randomly selecting K images of other categories as negative samples, and obtaining the generated image features f(G), positive sample features f(P) and negative sample features f(N) through the feature extraction network, and then calculating it using the following formula:

[0066]

[0067] The local reconstruction loss reproduces the local fusion process at the feature level at the image level, helping to reduce artifacts in the generated image. It is described as:

[0068]

[0069] Adversarial loss, the hinge-type adversarial loss is used to optimize the generator and discriminator, making the generated images difficult to distinguish from the real images. The adversarial losses of the generator and discriminator are and

[0070]

[0071] Classification loss is used by the discriminator to correctly classify real images and encourage the generator to generate images of the same category as the input image. Based on the ACGAN framework, its formula is:

[0072]

[0073] In the formula, is the category semantic loss, is the square of the Euclidean distance, f(G) is the generated image feature, f(P) is the positive sample feature, f(N) is the negative sample feature, margin is the threshold, is the local reconstruction loss, To generate an image, α is a random coefficient vector, LFM(X,α) is the semantically aligned fusion feature obtained by the local fusion module LFM, is the adversarial loss of the generator, is the adversarial loss of the discriminator, D(X) is the discriminator’s judgment result on the real image X, Generate images for the discriminator pair The judgment result of , max(·) means maximization, All are classification losses, log(·) is the logarithmic loss function, P(·) is the probability, and c(X) is the true category of the sample.

[0074] As a preferred implementation, the calculation formula of the overall objective function is:

[0075]

[0076] In the formula, is the objective optimization function of the generator, All are against losses, λ local , and λ csl are all hyperparameters for model optimization. are classification losses, is the local reconstruction loss, is the category semantic loss, is the objective optimization function of the discriminator, To combat losses.

[0077] As a preferred implementation, step S4 also includes: optimizing network parameters by alternately training the generator and the discriminator to generate high-quality images, and using multiple indicators to evaluate the quality of the generated images.

[0078] Among them, the evaluation indicators include image clarity and image similarity.

[0079] Example 2

[0080] See also Figure 2 , Figure 2 The figure shows the overall framework of the small sample image generation model SMR-CSL based on semantic mask reconstruction and category semantic learning provided by this embodiment, which is composed of a generator and a discriminator. The generator mainly includes an encoder, a local fusion module (LFM) and a semantic mask reconstruction module SMR. The encoder extracts the input image X = {x1,...,x k}, then the LFM module fuses the semantically aligned features by local feature fusion, then SMR reconstructs the deep semantic features obtained by the encoder and the local fusion module LFM, and reconstructs the masked features back to the image through a decoder, and obtains the corresponding generated image Finally, the real image X and the generated image The images are sent to the discriminator (Discriminator, D) to train the entire model. The proposed SMR and CSL learn semantics from different angles and improve them through targeted means. SMR reconstructs high-level semantic features with masks and combines the adjustment of dynamic mask rate to enhance semantic learning to help the discriminator to better update. CSL uses a category semantic loss to shorten the distance between images that belong to the same category as the generated image, and at the same time shorten the distance between images that do not belong to the same category as the generated image, so that the network can be better updated by learning category semantics.

[0081] Specifically, the encoder of the generator has one input convolution block and four downsampling convolution blocks. Each convolution block has a convolution layer, followed by Leaky-ReLU (LReLU) activation and batch normalization (BatchNormalization, BN); the decoder structure is symmetrical to that of the encoder, with four upsampling convolution blocks and one output convolution block. The feature size obtained after encoding is 8×8; the discriminator adopts a network architecture similar to that in FUNIT, in which there are four residual blocks as feature extractors and two fully connected layers for evaluating authenticity and classification respectively. The network is trained using the Adam optimizer with a fixed learning rate of 0.0001. In each iteration, 8 K-shot image generation tasks are randomly sampled as a mini-batch task to update the model, and a total of 100,000 iterations are performed. The four hyperparameters involved are: λ local = 0.5 and λ csl =1.

[0082] Example 3

[0083] See also Figure 3 , Figure 3 A schematic structural block diagram of an electronic device provided in an embodiment of the present application.

[0084] An electronic device includes a memory 101, a processor 102 and a communication interface 103, wherein the memory 101, the processor 102 and the communication interface 103 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines. The memory 101 can be used to store software programs and modules, and the processor 102 executes various functional applications and data processing by executing the software programs and modules stored in the memory 101. The communication interface 103 can be used to communicate signaling or data with other node devices.

[0085] Among them, the memory 101 can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc.

[0086] The processor 102 may be an integrated circuit chip with signal processing capability. The processor 102 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0087] It can be understood that the structure shown in the figure is only for illustration, and a method for generating small sample images based on semantic mask reconstruction and category semantic learning may also include more or fewer components than those shown in the figure, or have a different configuration than those shown in the figure. Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0088] In the embodiments provided in the present application, it should be understood that the disclosed method can also be implemented in other ways. The embodiments described above are merely schematic, for example, the flowchart or block diagram in the accompanying drawings shows the possible implementation architecture, functions and operations of the method and computer program product according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0089] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0090] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0091] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0092] It will be apparent to those skilled in the art that the present application is not limited to the details of the exemplary embodiments described above, and that the present application can be implemented in other specific forms without departing from the spirit or essential features of the present application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the present application is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims be included in the present application. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A small sample image generation method based on semantic mask reconstruction and category semantic learning, characterized in that: The following steps are involved: S1. Build a network model including generator and discriminator based on LoFGAN network; S2. Use the generator to extract the deep features of the input image, and fuse the deep features of the input image into deep semantic features through local feature fusion; S3, reconstruct the semantic mask of the deep semantic features and generate the corresponding image through the decoder; S4, sending the real image and the corresponding image generated by the decoder to the discriminator to train the network model; S5. Input the small sample image into the trained network model to obtain a generated image similar to the input image.

2. The method for generating small sample images based on semantic mask reconstruction and category semantic learning according to claim 1, characterized in that: Step S3 includes: Generate random noise based on deep semantic features; sorting the indices in each mask block based on random noise to determine pixels to be retained and pixels requiring mask operation; Initialize the mask tensor with the mask rate to create a binary mask based on the pixels to be retained and required masking operation; The generated binary mask is resized to the same size as the deep semantic feature using the nearest neighbor interpolation method; The deep semantic features are multiplied element-wise with the resized mask and passed through the decoder to generate a corresponding image consistent with the deep semantic features.

3. The method for generating small sample images based on semantic mask reconstruction and category semantic learning according to claim 1, characterized in that: In step S4, the process of training the network model through the discriminator includes training with maximizing the discriminative ability of the real image and the image generated by the decoder as the overall objective function, and the loss function includes: Category semantic loss: Local reconstruction loss: Fighting Loss: Classification loss: In the formula, is the category semantic loss, is the square of the Euclidean distance, f(G) is the generated image feature, f(P) is the positive sample feature, f(N) is the negative sample feature, margin is the threshold, is the local reconstruction loss, To generate an image, α is a random coefficient vector, LFM(X,α) is the semantically aligned fusion feature obtained by the local fusion module LFM, is the adversarial loss of the generator, is the adversarial loss of the discriminator, D(X) is the discriminator’s judgment result on the real image X, Generate images for the discriminator pair The judgment result of , max(·) means maximization, All are classification losses, log(·) is the logarithmic loss function, P(·) is the probability, and c(X) is the true category of the sample.

4. The method for generating small sample images based on semantic mask reconstruction and category semantic learning according to claim 3, characterized in that: The calculation formula of the overall objective function is: In the formula, is the objective optimization function of the generator, All are against losses, λ local , and λ csl are all hyperparameters for model optimization. are classification losses, is the local reconstruction loss, is the category semantic loss, is the objective optimization function of the discriminator, To combat losses.

5. The method for generating small sample images based on semantic mask reconstruction and category semantic learning according to claim 1, characterized in that: Step S4 also includes: optimizing network parameters by alternately training the generator and the discriminator to generate high-quality images, and using multiple indicators to evaluate the quality of the generated images.

6. An electronic device, characterized in that: include: A memory for storing one or more programs; processor; When the one or more programs are executed by the processor, the small sample image generation method based on semantic mask reconstruction and category semantic learning as described in any one of claims 1 to 5 is implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for generating small sample images based on semantic mask reconstruction and category semantic learning as described in any one of claims 1 to 5 is implemented.

Images