A method and device for training a generative adversarial network

Abstract

This application relates to the field of image processing and provides a method and apparatus for training a generative adversarial network. The method includes: acquiring a mosaic image synthesized from an original multi-channel spectral image; constructing a generator and processing the mosaic image using the generator to generate a de-mosaic image; constructing a discriminator and processing the original multi-channel spectral image and the de-mosaic image using the discriminator to generate a discrimination matrix, wherein N elements in the discrimination matrix are used to determine the authenticity of N blocks in the original multi-channel spectral image and the de-mosaic image, where N is a positive integer greater than 1; determining loss functions for the generator and the discriminator based on the discrimination matrix; and alternately training the generator and the discriminator based on the loss functions. This technical solution recovers the lost spectral information in the multispectral mosaic image, obtaining a high-resolution multispectral image.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a method and device for training a generative adversarial network. Background Art

[0002] With the development of multispectral imaging technology, the application market of multispectral cameras is becoming wider and wider. For example, multispectral cameras are integrated into various small and medium-sized multi-rotor, helicopter, and fixed-wing UAV platforms for agricultural monitoring, resource survey, disaster assessment, target identification, etc. Since the multispectral images taken by multispectral cameras lack spectral information and image data is seriously lost, it is necessary to restore the lost spectral information in the multispectral images to obtain high-resolution images. Among them, the multispectral image is a mosaic image, and the restoration process of the multispectral image is the demosaicing process. The result of the demosaicing process is the demosaiced image.

[0003] In recent years, Generative Adversarial Networks (GAN) have been widely used in the field of image processing. For example, the generator of GAN is used to generate an image, and the discriminator of GAN is used to discriminate the image generated by the generator to determine whether the image is the target image; then the generator is trained with the output of the discriminator, so that the generator can finally generate the correct target image. This unsupervised learning feature of GAN can be used for image restoration, but the output of the discriminator of GAN is the probability value of the overall true input data, which cannot reflect the local features of the image and is not suitable for high-resolution image restoration. Therefore, how to use GAN to restore the lost spectral information in multispectral images is a problem that needs to be solved at present. Summary of the invention

[0004] The present application provides a method and device for training a generative adversarial network, which can restore the lost spectral information in a multispectral mosaic image to obtain a high-resolution multispectral image.

[0005] In a first aspect, a method for training a generative adversarial network comprises:

[0006] Obtain a mosaic image synthesized from the original multi-channel spectral image;

[0007] Constructing a generator and using the generator to process the mosaic image to generate a demosaiced image;

[0008] Constructing a discriminator and using the discriminator to process the original multi-channel spectral image and the demosaiced image to generate a discriminant matrix, wherein N elements in the discriminant matrix are used to discriminate the authenticity of N blocks in the original multi-channel spectral image and the demosaiced image, where N is a positive integer greater than 1;

[0009] Determine the loss function of the generator and the discriminator according to the discriminant matrix;

[0010] The generator and the discriminator are alternately trained according to the loss function.

[0011] The above method can be executed by an electronic device or a chip in an electronic device. Among them, the generator can be used to restore the input mosaic image to generate a demosaiced image. Before application, it is necessary to construct a generator and a discriminator; use the generator to generate a demosaiced image; use the discriminator to process the original multi-channel spectral image and the demosaiced image to generate a discriminant matrix, and the N elements in the discriminant matrix are used to discriminate the authenticity of the original multi-channel spectral image and the N blocks in the demosaiced image, and N is a positive integer greater than 1; determine the loss function according to the discriminant matrix and alternately train the generator and the discriminator according to the loss function to improve the accuracy of the generator in generating the demosaiced image. Specifically, first fix the parameters of the generator to train the discriminator, and then fix the parameters of the discriminator to train the generator, and complete a complete training by alternating in this way. During each training process, the discriminator judges the N blocks of the input image separately and outputs the judgment results of each block. By judging each block, the extraction and characterization of local image features are realized, which helps to train the generator more efficiently and enables the generator to generate higher resolution multispectral image data.

[0012] Optionally, the generator includes a downsampling module, an upsampling module, an attention module, a convolution module and a sigmoid function module.

[0013] The downsampling module is used to perform downsampling processing on the mosaic image.

[0014] The up-sampling module is used to perform up-sampling processing on the result of the down-sampling processing,

[0015] The attention module is used to determine the weight of the output result of the upsampling module,

[0016] The convolution module is used to fuse the output results of the upsampling module.

[0017] The sigmoid function module is used to generate the demosaiced image based on the output result of the convolution layer.

[0018] Optionally, the downsampling module includes downsampling layers of M scales, the upsampling module includes upsampling layers of M scales, the attention module includes attention submodules of M scales, M is a positive integer greater than 1, and the M scales include a first scale and a second scale, wherein,

[0019] The upsampling layer of the first scale in the upsampling module is used to: output an upsampling result of the second scale according to an output result of the upsampling layer of the second scale in the upsampling module and an output result of the downsampling layer of the first scale in the downsampling module;

[0020] The first-scale attention submodule is used to determine the weight of the output result of the first-scale upsampling layer in the upsampling module according to the output result of the first-scale downsampling layer in the downsampling module.

[0021] The generator includes a downsampling module, an upsampling module, an attention module, a convolutional layer and a sigmoid function. The downsampling module includes downsampling layers of M scales, where M is a positive integer greater than 1, and the M scales include a first scale and a second scale. The downsampling layer of each scale is used to downsample the mosaic image. The upsampling module includes upsampling layers of M scales, and the upsampling layer of each scale is used to upsample the result of the downsampling process. The attention module is added between the convolutional layers of the downsampling module and the upsampling module. The attention module includes attention submodules of M scales, and the attention submodule of each scale is used to transfer the weight of the output result of the downsampling module to the upsampling module to retain the detail information of each level. The convolutional layer fuses the output results of the upsampling module to obtain images of each channel. The sigmoid function is used to process the output results of the convolutional layer to generate demosaiced images of each channel.

[0022] Optionally, the loss function of the generator is positively correlated with the square difference function of the original multi-channel spectral image and the demosaiced image, and,

[0023] The loss function of the generator is positively correlated with a cross entropy function, and the cross entropy function is a function determined based on a discrimination result of the discriminator on the demosaiced image.

[0024] Optionally, the loss function of the discriminator is positively correlated with a first cross entropy function, and the first cross entropy function is a function determined based on a discrimination result of the discriminator on the demosaiced image;

[0025] The loss function of the discriminator is positively correlated with a second cross entropy function, and the second cross entropy function is a function determined based on a discrimination result of the discriminator on the original multi-channel spectral image.

[0026] In a second aspect, a device for training a generative adversarial network is provided, comprising a module for executing any one of the methods in the first aspect.

[0027] In a third aspect, a device for training a generative adversarial network is provided, comprising a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the device executes any one of the methods in the first aspect.

[0028] According to a fourth aspect, a computer-readable medium is provided, wherein the computer-readable medium stores a program code, wherein the program code includes instructions for executing any one of the methods according to the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0030] Figure 1 is a schematic diagram of an image processing system provided by an embodiment of the present application;

[0031] Figure 2 It is a flowchart of a method for training a generative adversarial network provided in an embodiment of the present application;

[0032] Figure 3 It is a schematic diagram of the generator structure provided by an embodiment of the present application;

[0033] Figure 4 is a schematic diagram of a residual block structure provided by another embodiment of the present application;

[0034] Figure 5 is a schematic diagram of the structure of an attention submodule provided in another embodiment of the present application;

[0035] Figure 6 Schematic diagram of a device for training a generative adversarial network provided in one embodiment of the present application;

[0036] Figure 7 It is a schematic diagram of a device for training a generative adversarial network provided in one embodiment of the present application. DETAILED DESCRIPTION

[0037] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0038] It should be understood that when used in the present specification and the appended claims, the term "comprising" identifies the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0039] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0040] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as judging or implying relative importance.

[0041] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0042] The method for training a generative adversarial network provided in the embodiment of the present application can be applied to electronic devices such as servers, laptops, and ultra-mobile personal computers (UMPCs). The embodiment of the present application does not impose any restrictions on the specific type of electronic devices.

[0043] The method for training a generative adversarial network provided by the present application will be described in detail below in conjunction with the accompanying drawings. The following is an exemplary description using a multispectral image taken by a multispectral camera as an example, which does not constitute any limitation on the application scenario of the present application.

[0044] Figure 1 is a schematic diagram of an image processing system suitable for the present application.

[0045] The image processing system (i.e., GAN) includes a generator and a discriminator. The generator is used to generate an image based on input information. For example, after a mosaic image is input into the generator, the generator downsamples and upsamples the mosaic image, and then generates a demosaiced image, wherein the mosaic image is an image generated after resampling the original image (such as a multispectral image).

[0046] The purpose of the generator is to generate an image that is as realistic as possible, and the purpose of the discriminator is to determine whether the input image is real or fake. The generator and the discriminator are trained according to the judgment results of the discriminator until the generator can generate an accurate demosaiced image, that is, until the similarity between the demosaiced image generated by the generator and the original image meets the requirements.

[0047] Normally, the result output by the discriminator of GAN is the probability value of the input data as a whole being true, which cannot reflect the local features of the image and is not suitable for high-resolution image restoration. Multispectral images are often high-resolution images. The result obtained by training the generative adversarial network model based on a probability value is difficult to ensure that the generator generates high-resolution multispectral images, which is prone to cause the loss of local data. Below, a generative adversarial network training method suitable for multispectral images provided by this application will be introduced in detail.

[0048] like Figure 2 As shown, the method for training a generative adversarial network can be performed by an electronic device or a chip in the electronic device, and the method includes:

[0049] S201, obtaining a mosaic image synthesized from original multi-channel spectral images.

[0050] Optionally, an original multi-channel spectral image may be generated according to the spectral image data and a multi-spectral filter array (MSFA) camera spectral response function, which is recorded as an original image.

[0051] For example, given the spectral image data I(x, y, λ), the spectral response function of the MSFA camera is S K (λ), k = 1, 2, ..., K, k represents the number of spectral bands contained in the MSFA mode, then the spectral image of the k-channel is generated as:

[0052] I k (x, y) = ∫I(x, y, λ)S K (λ)dλ,k=1,2,…,K

[0053] Then, the spectral image data of the k channel (i.e., the original multi-channel spectral image data) is resampled according to the MSFA arrangement pattern, and the sampling results are synthesized into a mosaic image, wherein each original multi-channel spectral image (i.e., the original image) corresponds to a mosaic image. The original image and the mosaic image corresponding to the original image are used as a set of network training data to train the generative adversarial network model. During the training process, multiple sets of network training data can be used to train the generative adversarial network model, that is, multiple original images and mosaic images corresponding to the multiple original images can be used to train the generator and the discriminator. During the training process, the electronic device can control the generative adversarial network model to perform the following steps.

[0054] S202, constructing a generator and using the generator to process the mosaic image to generate a demosaiced image (ie, a multi-channel spectral image).

[0055] As an optional implementation, Figure 3 As shown, a generator 300 capable of de-mosaicing an image is constructed, and the electronic device inputs the mosaic image into the generator 300. The generator 300 de-mosaices the mosaic image and generates a de-mosaiced image (i.e., a multi-channel spectral image). The generator 300 includes a downsampling module 301 (i.e., an encoder), an upsampling module 303 (i.e., a decoder), an attention module 302, a convolutional layer 304, and a sigmoid function 305. The downsampling module 301 includes downsampling layers of M scales, where M is a positive integer greater than 1. For example, the downsampling module 301 includes downsampling layers of 4 scales, namely, downsampling layer 3011, downsampling layer 3012, downsampling layer 3013, and downsampling layer 3014. Each downsampling layer may include a residual block, which includes a convolutional layer, a batch normalization pooling (BatchNorm, BN) and a rectified linear unit (Rectified Linear Unit, Relu), such as Figure 4 shown.

[0056] The output result of the downsampling module 301 is divided into three parts. The first part is used as the input of the splicing layer 306, the second part is used as the input of the attention module 302, and the third part is used as the input of the upsampling module 303.

[0057] The upsampling module 303 includes upsampling layers of M scales. For example, the upsampling module 303 includes upsampling layers of 4 scales, and the upsampling layers of the 4 scales are upsampling layers 3031, upsampling layers 3032, upsampling layers 3033 and upsampling layers 3034, respectively. Each upsampling layer includes a bilinear upsampling layer and a residual block. The structure of the residual block is as follows: Figure 4 shown.

[0058] The attention module 302 includes M scales of attention submodules. For example, the attention module 302 includes 4 scales of attention submodules, namely, attention submodule 3021, attention submodule 3022, attention submodule 3023, and attention submodule 3024. Each attention submodule includes a convolution layer, a batch normalization pooling (BN) layer, and a Tanh activation function, such as Figure 5 The attention module 302 is used to apply weights to the output features of each downsampling layer in the downsampling module 301 and multiply them with the features output by each upsampling module 303 to retain the detailed information of each level and to strengthen the data editing capability in a targeted manner, that is, when each convolutional layer in the generator processes the mosaic image, the same weight is applied to the spectral data of the same channel, and different weights are applied to the spectral data of different channels to improve the accuracy of the generated multispectral image.

[0059] Exemplarily, the output result of the concatenation layer 306 is used as the first input of the upsampling layer 3031, and the output result of the downsampling layer 3014 is used as the second input of the upsampling layer 3031. The upsampling layer 3031 processes the first input and the second input to obtain a first output result. The first output result is multiplied by the output result of the attention submodule 3024 to obtain a second output result. The second output result is used as the first input of the upsampling layer 3032, and the output result of the downsampling layer 3013 is used as the second input of the upsampling layer 3032. The upsampling layer 3032 processes the first input and the second input to obtain a third output result. Wherein, when the upsampling layer 3032 is an upsampling layer of the first scale, the upsampling layer 3031 is an upsampling layer of the second scale, the attention submodule 3024 is an attention submodule of the second scale, the downsampling layer 3013 is a downsampling layer of the first scale, and the downsampling layer 3014 is a downsampling layer of the second scale.

[0060] By analogy, the input data processed by the convolution layer 304 is the result of convolution operation on the output result of the upsampling layer 3034 and the output result of the attention submodule 3021. The convolution layer 304 fuses the input data to obtain features corresponding to the number of channels (for example, images of each channel). The sigmoid function 305 processes the output result of the convolution layer 304 so that the pixel grayscale value is between 0 and 1, thereby obtaining the demosaiced image of each channel.

[0061] S203, constructing a discriminator and using the discriminator to process the original multi-channel spectral image and the demosaiced image to generate a discriminant matrix, wherein the N elements in the discriminant matrix are used to discriminate the authenticity of the N blocks in the original multi-channel spectral image and the demosaiced image, and N is a positive integer greater than 1.

[0062] Optionally, a discriminator is constructed to discriminate the demosaiced image and the original image using the discriminator, and the discriminator and the generator are trained according to the discrimination result. The discriminator can use a classification network model, for example, a Visual Geometry Group (VGG) model, a Residual Neural Network (ResNet) model, an Inception model, etc. The discriminator is used to discriminate the authenticity of the demosaiced image, and the discriminator can judge the authenticity of N blocks in the demosaiced image.

[0063] Exemplarily, the discriminator is a VGG model, and the VGG model includes multiple sampling convolutional layers. The convolutional layer includes a convolution (Conv), a batch normalization pooling (BN) layer, a Sigmoid activation function, and a Maxpool maximization operation. The structure of the VGG model is shown in Table 1.

[0064] Table 1

[0065] Name parameter Input size / channel Output size / channel Conv11 k3c64 256×256 / 9 256×256 / 64 Conv12 k3c64 256×256 / 64 256×256 / 64 Maxpool1 k2s2 256×256 / 64 128×128 / 64 Conv21 k3c128 128×128 / 64 128×128 / 128 Conv22 k3c128 128×128 / 128 128×128 / 128 Maxpool2 k2s2 128×128 / 128 64×64 / 128 Conv31 k3c256 64×64 / 128 64×64 / 256 Conv32 k3c256 64×64 / 256 64×64 / 256 Conv33 k3c256 64×64 / 256 64×64 / 256 Maxpool3 k2s2 64×64 / 256 32×32 / 256 Conv41 k3c512 32×32 / 512 32×32 / 512 Conv42 k3c512 32×32 / 512 32×32 / 512 Conv43 k3c512 32×32 / 512 32×32 / 512 Maxpool4 k2s2 32×32 / 512 16×16 / 512 Conv51 k3c512 16×16 / 512 16×16 / 512 Conv52 k3c512 16×16 / 512 16×16 / 512 Conv53 k3c512 16×16 / 512 16×16 / 512 Maxpool5 k2s2 16×16 / 512 8×8 / 512 Conv6 k1c1 8×8 / 512 8×8 / 1 Sigmoid 8×8 / 1 8×8 / 1

[0066] The processing of the VGG model is as follows:

[0067] The input is a 256×256×9 demosaiced image, where 256×256 is the size of the demosaiced image and 9 is the number of channels of the demosaiced image. The 256×256×9 demosaiced image is processed by the convolution layer Conv11 to output image data of size 256×256×64, which is processed by the convolution layer Conv12 to output image data of size 256×256×64, which is processed by the Maxpool1 operation to output image data of size 128×128×64, where 64 is the number of channels.

[0068] Subsequently, the 64-channel image data is processed into 128-channel image data, 256-channel image data, and 512-channel image data in turn, where the size of the 512-channel image data is 8×8. The image data does not use full convolution operation to output the discrimination result of the whole image at the end, but generates an 8×8 tensor (i.e., discrimination matrix) after ordinary convolution (Conv6), and then processes the tensor through the convolution layer of the sigmoid function to obtain the probability value of the image area corresponding to each element in the tensor being true. The probability value range is [0,1]. The closer the probability value of the element is to 0, the smaller the probability that the image block corresponding to the element is true; the closer the probability value of the element is to 1, the greater the probability that the image block corresponding to the element is true. The above method is equivalent to dividing the demosaiced image into 64 image blocks, calculating the probability of each image block being true separately and outputting the discrimination result.

[0069] After determining the discriminant matrix, the electronic device can perform the following steps to facilitate training the generative adversarial network.

[0070] S204: Determine the loss function of the generator and the discriminator according to the discriminant matrix.

[0071] As mentioned above, the mosaic image synthesized from the original image gt is input into the generator to output the demosaiced image pred. The original image gt and the demosaiced image pred are respectively input into the discriminator to output the discriminant results (discriminant matrices) σ(D(gt) and σ(D(pred). Based on σ(D(gt) and σ(D(pred), the cross entropy function L can be constructed. critical .

[0072] First, use the cross entropy function to construct the generator's loss function lossG as follows:

[0073] lossG=L2(gt,pred)+λL critical (σ(D(pred)),1)

[0074] Among them, L2(gt,pred) is the square difference function between the de-mosaiced image pred and the original image gt, and λ is used to balance the matching loss weight between the de-mosaiced image and the original image and the adversarial constraint loss weight after the discriminator. From the above formula, we can see that lossG is positively correlated with the square difference function, and lossG is positively correlated with the cross entropy function, that is, the smaller the square difference function and the cross entropy function, the smaller the lossG, and the better the performance of the generator.

[0075] Based on the cross entropy function L critical The loss function lossD of the constructed discriminator is as follows:

[0076] lossD=L critical (σ(D(pred)),0)+L critical (σ(D(gt)),1)

[0077] L critical (σ(D(pred)),0) is the first cross entropy function, which is a function determined based on the discriminator's discrimination result on the demosaiced image; when L critical (σ(D(gt)),1) is the second cross entropy function, which is a function determined based on the discrimination result of the discriminator on the original multi-channel spectral image. lossD is positively correlated with the first cross entropy function and the second cross entropy function. When the first cross entropy function and the second cross entropy function are smaller, the performance of the discriminator is better.

[0078] S205, alternately training the generator and the discriminator according to the loss function.

[0079] Randomly initialize the network parameters of the generator and discriminator. Set the parameters in the generator and discriminator models to random numbers. Here, the loss function includes the loss function of the generator (generation loss function, lossG) and the loss function of the discriminator (discrimination loss function, lossG). The generator and discriminator are alternately trained according to the generation loss function and the discrimination loss function to minimize lossG and lossD.

[0080] In the alternating training process, the discriminator is trained first. At this time, the generator parameters are fixed. That is, a mosaic image is selected and input into the generator to generate a demosaiced image. The original image corresponding to the mosaic image and the generated demosaiced image are respectively input into the discriminator to obtain the discrimination result. The discrimination loss function value is calculated according to the discrimination result, and the stochastic gradient descent method is used to perform back propagation according to the discrimination loss function value to update the parameters of the discriminator. Next, the generator is trained. At this time, the discriminator parameters are fixed, and another mosaic image is selected and input into the generator to generate a demosaiced image. The demosaiced image is input into the discriminator to obtain the discrimination result. The generation loss function is calculated according to the discrimination result, and the stochastic gradient descent method is used to perform back propagation according to the generation loss function value to update the parameters of the generator. In this way, a complete training is completed by alternating. Then the above training process is repeated to complete the next complete training, and it is repeated continuously until the training is completed.

[0081] After each complete training is completed, the generator can be fed with a test image (mosaic image) and the generated result can be output. Then, the output result (demosaiced image) of the generator can be verified to be the same as the original image (multispectral image) based on structural similarity, peak signal-to-noise ratio or other methods. If they are the same, it means that the generative adversarial network model has been successfully trained and no further training is required. If they are different, the generative adversarial network model needs to be trained further and the network parameters updated until the generative adversarial network model parameters with good performance are obtained. It is understandable that it is also possible to choose to verify once after completing a certain number of trainings, and there is no specific limit on the number of times.

[0082] The above describes in detail an example of a method for training a generative adversarial network provided by the present application. It is understandable that, in order to implement the above functions, the corresponding device includes a hardware structure and / or software module corresponding to each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.

[0083] The present application can divide the functional units of the device for training a generative adversarial network according to the above method example. For example, each function can be divided into each functional unit, or two or more functions can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of units in the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0084] Figure 6 600 is a schematic diagram of a device for training a generative adversarial network provided by the present application. The device 600 includes an input module 601 and a processing module 602.

[0085] The input module 601 is used to obtain a mosaic image synthesized from an original multi-channel spectral image.

[0086] The processing module 602 is used to: construct a generator and use the generator to process the mosaic image to generate a demosaiced image; construct a discriminator and use the discriminator to process the original multi-channel spectral image and the demosaiced image to generate a discriminant matrix, wherein the N elements in the discriminant matrix are used to discriminate the authenticity of N blocks in the original multi-channel spectral image and the demosaiced image, and N is a positive integer greater than 1; determine the loss function of the generator and the discriminator according to the discriminant matrix; and alternately train the generator and the discriminator according to the loss function.

[0087] The specific manner in which the device 600 executes the method for training a generative adversarial network and the beneficial effects produced can be found in the relevant description in the method embodiment.

[0088] Figure 7 A schematic diagram of the device structure of a method for training a generative adversarial network is shown in the present application. Figure 7The dotted line in the figure indicates that the unit or the module is optional. The device 700 can be used to implement the method described in the above method embodiment. The device 700 can be an electronic device, such as a server, a chip, etc.

[0089] The device 700 includes one or more processors 701, which can support the device 700 to implement Figure 2 The method in the corresponding method embodiment. The processor 701 can be a general-purpose processor or a special-purpose processor. For example, the processor 701 can be a central processing unit (CPU). The CPU can be used to control the device 700, execute software programs, and process data of software programs. The device 700 can also include a communication unit 705 to implement signal input (reception) and output (transmission).

[0090] For example, the device 700 may be a chip, and the communication unit 705 may be an input and / or output circuit of the chip, or the communication unit 705 may be a communication interface of the chip, and the chip may be a component of an electronic device.

[0091] For another example, the device 700 may be an electronic device, and the communication unit 705 may be a transceiver of the electronic device, or the communication unit 705 may be a transceiver circuit of the electronic device.

[0092] The device 700 may include one or more memories 702, on which a program 704 is stored. The program 704 can be executed by the processor 701 to generate an instruction 703, so that the processor 701 performs the method described in the above method embodiment according to the instruction 703. Optionally, data (such as the ID of the chip to be tested) can also be stored in the memory 702. Optionally, the processor 701 can also read the data stored in the memory 702, and the data can be stored at the same storage address as the program 704, or the data can be stored at a different storage address from the program 704.

[0093] The processor 701 and the memory 702 may be provided separately or integrated together, for example, integrated on a system on chip (SOC) of the electronic device.

[0094] The specific manner in which the processor 701 executes the method for training a generative adversarial network can refer to the relevant description in the method embodiment.

[0095] It should be understood that each step of the above method embodiment can be completed by a hardware-based logic circuit or a software-based instruction in the processor 701. The processor 701 can be a CPU, a digital signal processor (DSP), a field programmable gate array (FPGA), or other programmable logic devices, such as discrete gates, transistor logic devices, or discrete hardware components.

[0096] The present application also provides a computer program product, which, when executed by the processor 701, implements the method described in any method embodiment of the present application.

[0097] The computer program product may be stored in the memory 702 , for example, a program 704 , which is converted into an executable target file that can be executed by the processor 701 after preprocessing, compiling, assembling, and linking.

[0098] The present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a computer, the method described in any method embodiment of the present application is implemented. The computer program can be a high-level language program or an executable target program.

[0099] The computer-readable storage medium is, for example, a memory 702. The memory 702 may be a volatile memory or a nonvolatile memory, or the memory 702 may include both a volatile memory and a nonvolatile memory. Among them, the nonvolatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DRRAM).

[0100] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described devices and equipment and the technical effects produced can refer to the corresponding processes and technical effects in the aforementioned method embodiments, and will not be repeated here.

[0101] In several embodiments provided in the present application, the disclosed systems, devices and methods can be implemented in other ways. For example, some features of the method embodiments described above can be ignored or not performed. The device embodiments described above are merely schematic, and the division of units is only a logical function division. There may be other division methods in actual implementation, and multiple units or components may be combined or integrated into another system. In addition, the coupling between the units or the coupling between the components may be direct coupling or indirect coupling, and the above coupling includes electrical, mechanical or other forms of connection.

[0102] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application is described in detail with reference to the above-mentioned embodiments, a person skilled in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features can be replaced by equivalents, and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for generating an adversarial network, characterized in that: include: Acquiring a mosaic image synthesized from an original multi-channel spectral image, comprising: generating the original multi-channel spectral image according to spectral image data and a spectral response function of a multi-spectral filter array camera, resampling the original multi-channel spectral image data, and synthesizing the sampling results into the mosaic image; Constructing a generator and using the generator to process the mosaic image to generate a demosaiced image; the generator includes a downsampling module, an upsampling module, an attention module, a convolution layer and a sigmoid function, the downsampling module is used to downsample the mosaic image, the upsampling module is used to upsample the result of the downsampling process, the attention module is used to determine the weight of the output result of the upsampling module, the weight of the spectral data of the same channel is the same, and the weight of the spectral data of different channels is different, the convolution layer is used to fuse the output result of the upsampling module, and the sigmoid function is used to generate the demosaiced image based on the output result of the convolution layer; Constructing a discriminator and using the discriminator to process the original multi-channel spectral image and the demosaiced image to generate a discriminant matrix, wherein N elements in the discriminant matrix are used to discriminate the authenticity of N blocks in the original multi-channel spectral image and the demosaiced image, where N is a positive integer greater than 1; Determine the loss function of the generator and the discriminator according to the discriminant matrix; The generator and the discriminator are alternately trained according to the loss function.

2. The method according to claim 1, characterized in that: The downsampling module includes downsampling layers of M scales, the upsampling module includes upsampling layers of M scales, the attention module includes attention submodules of M scales, M is a positive integer greater than 1, and the M scales include a first scale and a second scale, wherein, The upsampling layer of the first scale in the upsampling module is used to: output the upsampling result of the first scale according to the output result of the upsampling layer of the second scale in the upsampling module and the output result of the downsampling layer of the first scale in the downsampling module; The first-scale attention submodule is used to determine the weight of the output result of the first-scale upsampling layer in the upsampling module according to the output result of the first-scale downsampling layer in the downsampling module.

3. The method according to claim 1 or 2, characterized in that: The loss function of the generator is positively correlated with the square difference function of the original multi-channel spectral image and the demosaiced image, and, The loss function of the generator is positively correlated with a cross entropy function, and the cross entropy function is a function determined based on a discrimination result of the discriminator on the demosaiced image.

4. The method according to claim 1 or 2, characterized in that: The loss function of the discriminator is positively correlated with the first cross entropy function, Furthermore, the first cross entropy function is a function determined based on a discrimination result of the discriminator on the demosaiced image; The loss function of the discriminator is positively correlated with the second cross entropy function, Furthermore, the second cross entropy function is a function determined based on a discrimination result of the discriminator on the original multi-channel spectral image.

5. A device for training a generative adversarial network, characterized in that: Including input module and processing module, The input module is used to obtain a mosaic image synthesized from an original multi-channel spectral image, including: generating the original multi-channel spectral image according to spectral image data and a spectral response function of a multi-spectral filter array camera, resampling the original multi-channel spectral image data, and synthesizing the sampling results into the mosaic image; The processing module is used to: construct a generator and use the generator to process the mosaic image to generate a demosaiced image; construct a discriminator and use the discriminator to process the original multi-channel spectral image and the demosaiced image to generate a discriminant matrix, wherein N elements in the discriminant matrix are used to discriminate the authenticity of N blocks in the original multi-channel spectral image and the demosaiced image, and N is a positive integer greater than 1; determine the loss function of the generator and the discriminator according to the discriminant matrix; and alternately train the generator and the discriminator according to the loss function; The processing module also includes a downsampling module, an upsampling module, an attention module, a convolution module and a sigmoid function module. The downsampling module is used to perform downsampling processing on the mosaic image. The up-sampling module is used to perform up-sampling processing on the result of the down-sampling processing, The attention module is used to determine the weight of the output result of the upsampling module. The spectral data of the same channel have the same weight, while the spectral data of different channels have different weights. The convolution module is used to fuse the output results of the upsampling module. The sigmoid function module is used to generate the demosaiced image based on the output result of the convolution layer.

6. The device according to claim 5, characterized in that The downsampling module includes downsampling layers of M scales, the upsampling module includes upsampling layers of M scales, the attention module includes attention submodules of M scales, M is a positive integer greater than 1, and the M scales include a first scale and a second scale, wherein, The upsampling layer of the first scale in the upsampling module is used to: output the upsampling result of the first scale according to the output result of the upsampling layer of the second scale in the upsampling module and the output result of the downsampling layer of the first scale in the downsampling module; The first-scale attention submodule is used to determine the weight of the output result of the first-scale upsampling layer in the upsampling module according to the output result of the first-scale downsampling layer in the downsampling module.

7. The device according to claim 5 or 6, characterized in that The loss function of the generator is positively correlated with the square difference function of the original multi-channel spectral image and the demosaiced image, and, The loss function of the generator is positively correlated with a cross entropy function, and the cross entropy function is a function determined based on a discrimination result of the discriminator on the demosaiced image.

8. The device according to claim 5 or 6, characterized in that The loss function of the discriminator is positively correlated with a first cross entropy function, where the first cross entropy function is a function determined based on a discrimination result of the discriminator on the demosaiced image; The loss function of the discriminator is positively correlated with a second cross entropy function, and the second cross entropy function is a function determined based on a discrimination result of the discriminator on the original multi-channel spectral image.

9. A device for training a generative adversarial network, characterized in that The device comprises a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the device executes the method according to any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the method according to any one of claims 1 to 4.

Images