Method and device for generating multi-exposure image samples, computer equipment and medium

Abstract

The embodiment of the present invention discloses a method and device for generating multi-exposure image samples, a computer device and a medium. In a specific implementation, the method includes: inputting a first image into a feature extractor that has been trained through a cyclic generative network architecture to obtain an exposure feature vector; performing multiplication operations on the exposure feature vector and multiple weight values ​​within a set value range respectively; performing addition operations on the result of the multiplication operation and the first image to obtain multiple image samples with different exposures corresponding to the first image. This implementation can effectively realize the expansion of multi-exposure image samples, and the exposure range and exposure difference of the expanded image can be adjusted.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and more specifically, to a method and device for generating multi-exposure image samples, a computer device, and a medium. Background Art

[0002] Models such as neural networks used in the fields of target detection and target recognition need to be trained with a large number of image samples of different exposures in the early stage to ensure the generalization ability of the model, so that the model can achieve target detection, target recognition and other functions for images of different exposures. The inventors found that it is difficult to obtain a large number of image samples of different exposures, and even if collected from multiple channels, it is difficult to meet the demand for thousands or even tens of thousands of image samples. Summary of the invention

[0003] The object of the present invention is to provide a method and apparatus for generating multi-exposure image samples, a computer device and a medium to solve at least one of the problems existing in the prior art.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] A first aspect of the present invention provides a method for generating multi-exposure image samples, comprising:

[0006] Inputting the first image into a feature extractor trained by a recurrent generative network architecture to obtain an exposure feature vector;

[0007] performing multiplication operations on the exposure feature vector and a plurality of weight values ​​within a set value range respectively; and

[0008] An addition operation is performed on the result of the multiplication operation and the first image to obtain a plurality of image samples with different exposure levels corresponding to the first image.

[0009] Optionally, the feature extractor comprises a coding layer, a conversion layer, a decoding layer and a feature extraction layer connected in series;

[0010] Inputting the first image into a feature extractor that has been trained through a cyclic generation network architecture to obtain an exposure feature vector of the first image includes:

[0011] Inputting the first image into the encoding layer to obtain a first feature vector;

[0012] Inputting the first feature vector into a conversion layer to obtain a second feature vector converted from the exposure feature vector;

[0013] Inputting the second feature vector into a decoding layer to obtain a third feature vector; and

[0014] The third feature vector is input into the feature extraction layer to obtain an exposure feature vector.

[0015] Optionally, the feature extraction layer is a residual network.

[0016] Optionally, the multiple weight values ​​are taken at equal intervals within the set value range.

[0017] Optionally, the method further includes: adjusting the set value range.

[0018] Optionally, before inputting the first image into the feature extractor that has been trained through the cyclic generative network architecture, the method also includes: obtaining the feature extractor through the cyclic generative network architecture training.

[0019] Optionally, the training of a feature extractor by cyclically generating a network architecture comprises:

[0020] Acquire a plurality of first training images and a plurality of second training images, wherein one of the first training image and the second training image is an underexposed image, and the other is an overexposed image;

[0021] Constructing a cyclic generative network architecture, the cyclic generative network architecture comprising a first generator, a first discriminator, a second generator and a second discriminator, wherein the first generator comprises a feature extractor and an addition operation layer, the addition operation layer is used to perform an addition operation on the input and output of the feature extractor to obtain the output of the first generator, the output of the first generator is respectively used as the input of the second discriminator and the second generator, and the output of the second generator is respectively used as the input of the first discriminator and the first generator;

[0022] The plurality of first training images are respectively input into the first generator and the first discriminator, and the plurality of second training images are respectively input into the second generator and the second discriminator, so as to train the recurrent generative network architecture and obtain a trained feature extractor.

[0023] A second aspect of the present invention provides a device for generating multi-exposure image samples, comprising: a feature extraction module, a multiplication operation module and an addition operation module;

[0024] The feature extraction module is used to input the first image into a feature extractor that has been trained through a cyclic generation network architecture to obtain an exposure feature vector;

[0025] The multiplication operation module is used to perform multiplication operations on the exposure feature vector and multiple weight values ​​within a set value range respectively;

[0026] The addition operation module is used to perform an addition operation on the result of the multiplication operation and the first image to obtain a plurality of image samples with different exposure levels corresponding to the first image.

[0027] Optionally, the feature extractor comprises a coding layer, a conversion layer, a decoding layer and a feature extraction layer connected in series;

[0028] The encoding layer is used to obtain a first feature vector according to the input first image;

[0029] The conversion layer is used to obtain a second feature vector after the exposure feature vector is converted according to the input first feature vector;

[0030] The decoding layer is used to obtain a third feature vector according to the input second feature vector; and

[0031] The feature extraction layer is used to obtain an exposure feature vector based on the input third feature vector.

[0032] Optionally, the feature extraction layer is a residual network.

[0033] The third aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for generating multi-exposure image samples provided in the first aspect of the present invention is implemented.

[0034] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements a method for generating multi-exposure image samples when the program is executed by a processor.

[0035] The beneficial effects of the present invention are as follows:

[0036] The technical solution of the present invention can effectively realize the augmentation of multi-exposure image samples, and the exposure range and exposure difference of the augmented image can be adjusted. Among them, the feature extractor is trained by cyclically generating a network architecture, which can solve the problem of difficulty in obtaining paired training samples when training the feature extractor. Therefore, the technical solution of the present invention can greatly increase the data volume of image samples with different exposures, and increase the generalization ability of the model obtained by training image samples with different exposures and applied to the fields of target detection and target recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The specific implementation modes of the present invention are further described in detail below in conjunction with the accompanying drawings.

[0038] Figure 1 A diagram showing an exemplary system architecture in which one embodiment of the present invention may be applied.

[0039] Figure 2 A flow chart of a method for generating multi-exposure image samples provided by an embodiment of the present invention is shown.

[0040] Figure 3 A schematic diagram showing a cyclic generation network architecture adopted in an embodiment of the present invention.

[0041] Figure 4 Show Figure 3 Schematic diagram of the first generator in the recurrent generative network architecture shown.

[0042] Figure 5 A data trend diagram of the data augmentation stage of the method for generating multi-exposure image samples provided by an embodiment of the present invention is shown.

[0043] Figure 6 A schematic diagram showing a number of image samples at different exposure levels.

[0044] Figure 7 A schematic diagram showing the structure of a computer system that implements the device for generating multi-exposure image samples provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to explain the present invention more clearly, the present invention is further described below in conjunction with embodiments and drawings. Similar components in the drawings are represented by the same reference numerals. It should be understood by those skilled in the art that the content specifically described below is illustrative rather than restrictive, and the protection scope of the present invention should not be limited thereto.

[0046] Models such as neural networks used in the fields of target detection and target recognition need to be trained with a large number of image samples of different exposures in the early stage to ensure the generalization ability of the model, so that the model can achieve target detection, target recognition and other functions for images of different exposures. The inventors found that it is difficult to obtain a large number of image samples of different exposures, and even if collected from multiple channels, it is difficult to meet the demand for thousands or even tens of thousands of image samples.

[0047] In view of this, an embodiment of the present invention provides a method for generating multi-exposure image samples, comprising the following steps:

[0048] The feature extractor is obtained by training the cyclic generative network architecture;

[0049] Inputting the first image into a feature extractor trained by a recurrent generative network architecture to obtain an exposure feature vector;

[0050] performing multiplication operations on the exposure feature vector and a plurality of weight values ​​within a set value range respectively; and

[0051] An addition operation is performed on the result of the multiplication operation and the first image to obtain a plurality of image samples with different exposure levels corresponding to the first image.

[0052] Based on the method for generating multi-exposure image samples provided by this embodiment, the multi-exposure image samples can be effectively expanded, and the exposure range and exposure difference of the expanded image can be adjusted. Among them, the feature extractor is trained by cyclic generation network architecture to solve the problem that paired training samples are difficult to obtain when training the feature extractor. Therefore, the method for generating multi-exposure image samples provided by this embodiment can greatly increase the data volume of image samples with different exposures, and increase the generalization ability of the model obtained by training image samples with different exposures and applied to the fields of target detection and target recognition.

[0053] The method for generating multi-exposure image samples provided in this embodiment does not limit the objects contained in the input first image, and may be a landscape image, a person image, etc. The parameters such as the size, clarity, and exposure of the input first image are also not limited, wherein generally speaking, the input first image is an image with normal exposure.

[0054] The method for generating multi-exposure image samples provided in this embodiment can be implemented by a computer device with data processing capabilities. Specifically, the computer device can be a computer with data processing capabilities, including a personal computer (PC), a minicomputer or a mainframe, or a server or server cluster with data processing capabilities, etc. This embodiment does not limit this.

[0055] In order to facilitate understanding of the technical solution of this embodiment, Figure 1 The practical scenario of the above method provided in this embodiment is described. Figure 1 , the scenario includes a training server 10 and a generation server 20. In this embodiment, the training server 10 first uses the training image to train a feature extractor through a cyclic generation network architecture. Subsequently, the generation server 20 can use the feature extractor trained by the training server 10 to generate multi-exposure image samples. The first image is input into the generation server 20, and multiple image samples with different exposures corresponding to the first image can be obtained.

[0056] It should be noted that Figure 1 In actual applications, the training server 10 and the generation server 20 may be two independent servers or a server integrating the model training function and the image generation function. When the two servers are two independent servers, the two servers may communicate with each other through a network, and the network may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0057] Next, from the perspective of a processing device with data processing capabilities, a method for generating multi-exposure image samples provided by this embodiment is described.

[0058] An embodiment of the present invention provides a method for generating multi-exposure image samples, such as Figure 2 As shown, the following steps are included:

[0059] S201, obtain a feature extractor by cyclically generating a network architecture training.

[0060] Next, we will introduce the network structure of the cycle generation network.

[0061] Cycle GAN is essentially two mirror-symmetric generative adversarial networks (GANs), which form a ring network. The two GANs share two generators and each has a discriminator, that is, there are two discriminators and two generators. A typical advantage of Cycle GAN is that it can be trained using two unpaired training image sets. For two data domains X and Y: On the one hand, the generator G X→Y The task is to convert the image Real_X belonging to the data domain X into the image Fake_Y of the data domain Y. In Cycle GAN, this newly generated image Fake_Y is then passed to the generator G Y→X To transform back to the image Rec_X of the data domain X, this Rec_X needs to be similar to Real_X to define a meaningful mapping that does not exist in the unpaired dataset, that is, the generator can retain the characteristic target of the original image; on the other hand, the generator G Y→X The task is to convert the image Real_Y belonging to the data domain Y into the image Fake_X of the data domain X. In Cycle GAN, this newly generated image Fake_X is then passed to the generator G X→Y To convert back to the image Rec_Y in data domain Y.

[0062] The training of the cycle generation network can use the loss function of mean square error loss:

[0063]

[0064] The generator in the recurrent generative network may include an encoder, a transformer, and a decoder connected in series to generate G Y→XFor example, the encoder is used to extract features from the input image using a convolutional neural network, such as compressing the image into 256 64*64 feature vectors; the converter is used to convert the feature vector of the image in the data domain Y into the feature vector in the data domain X by combining the dissimilar features of the image. For example, the converter can use a 6-layer Resnet (residual network) module, each Resnet module is a neural network layer composed of two convolutional layers, which can achieve the goal of retaining the original image features during conversion; the encoder is used to use the deconvolution layer (Deconvolution) to complete the task of restoring low-level features from the feature vector, and finally obtain the generated image. Generator G X→Y The structure and generator G Y→X Similar, but the generator G X→Y The converter is used to convert the feature vector of the image in the data domain X into the feature vector in the data domain Y.

[0065] The task of the discriminator in the recurrent generative network is to try to predict whether the input image is the original image or the image generated by the generator. The discriminator itself belongs to a convolutional network, which needs to extract features from the image and then determine whether the extracted features belong to a specific category by adding a convolutional layer that produces a one-dimensional output.

[0066] In a possible implementation, in the method for generating multi-exposure image samples provided in this embodiment, the training image set used by the feature extractor obtained by cyclically generating a network architecture training includes a first training image set and a second training image set, the first training image set includes a plurality of underexposed first training images, the second training image set includes a plurality of overexposed second training images, and it is assumed that the underexposed first training images belong to data domain X, and the overexposed second training images belong to data domain Y. It should be noted that the first training image and the second training image do not need to be paired, and the targets contained in the two may be different.

[0067] In one possible implementation, Figure 3 As shown, in the method for generating multi-exposure image samples provided in this embodiment, the cyclic generation network architecture includes: a first generator G X→Y , the first discriminator D X , the second generator G Y→X and the second discriminator D Y , the first generator G X→Y The output Fake_Y is used as the second discriminator D Y and the second generator G Y→X The input of the second generator G Y→X The output Fake_X is used as the first discriminator D X and the first generator G X→Y Input.

[0068] The loss function for training the cycle generation network architecture can use the mean square error loss similar to the aforementioned cycle generation network.

[0069] During the training process, for two data domains X and Y:

[0070] The first underexposed training image Real_X belonging to the data domain X is input into the first generator G X→Y , the first generator G X→Y The task is to convert the underexposed first training image Real_X belonging to the data domain X into an overexposed image Fake_Y in the data domain Y. The purpose of this process in training is to make the first generator G X→Y Learning data domain Y (overexposure) features, after training, the first generator G X→Y What is learned is the ability to transform the exposure feature vector;

[0071] The overexposed second training image Real_Y belonging to the data domain Y is input into the second generator G Y→X , the second generator G Y→X The task is to convert the overexposed second training image Real_Y belonging to the data domain Y into the underexposed image Fake_X of the data domain X. The purpose of this process in training is to make the second generator G Y→X Learning data domain X (underexposure) features, after training, the second generator G Y→X What is learned is the ability to transform the exposure feature vector;

[0072] The underexposed first training image Real_X belonging to the data domain X is also input into the first discriminator D X , the second generator G Y→X The generated image Fake_X is also input into the first discriminator D X , the first discriminator D X The task is to identify the input image as the second generator G Y→X The generated image Fake_X is still the underexposed first training image Real_X belonging to the data domain X, and the formula is the loss of 0,1 binary classification;

[0073] The overexposed second training image Real_Y belonging to the data domain Y is also input into the second discriminator D Y , the first generator G X→Y The generated image Fake_Y is also input into the second discriminator D Y , the second discriminator D Y The task is to identify the input image as the first generator G X→Y The generated image Fake_Y is still the overexposed second training image Real_Y belonging to the data domain Y, and the formula is the loss of 0,1 binary classification;

[0074] In order to make the first generator G X→Y and the second generator G Y→X Can retain the characteristic target of the original image. During the training process, the first generator G X→Y The generated image Fake_Y is also passed to the generator G Y→X To convert the image Rec_X back to the data domain X, the second generator G Y→X The generated image Fake_X is then passed to the generator G X→Y Take the image Rec_Y converted back to the data domain Y and train the first generator G by calculating the loss between Rec_Y and Real_Y and the loss between Rec_X and Real_X respectively. X→Y and the second generator G Y→X .

[0075] In one possible implementation, Figure 4 As shown, the first generator G X→Y The first generator G includes a feature extractor 410 and an addition operation layer 420, wherein the addition operation layer 420 is used to perform an addition operation on the input and output of the feature extractor 410 to obtain a first generator G X→Y The output of Fake_Y (or Rec_Y).

[0076] Since during the training process, the first generator G X→Y The task is to convert the underexposed first training image Real_X belonging to the data domain X into an overexposed image Fake_Y in the data domain Y (in addition, the first generator G X→Y The task also includes the second generator G Y→X The generated image Fake_X is converted back to the image Rec_Y in the data domain Y), the first generator G X→Y It will learn the data domain Y (overexposure) features, and the first generator G after training X→Y What is learned is the ability to convert the exposure feature vector. Therefore, under the setting of adding the extracted feature vector to the original image (underexposure) to obtain an image in another data domain (overexposure), the feature extractor 410 learns to convert and extract the exposure feature vector of the image. After training, the feature extractor 410 learns the ability to convert and extract the exposure feature vector.

[0077] In one possible implementation, Figure 4 As shown, the first generator G X→Y The feature extractor 410 includes an encoding layer 411, a conversion layer 412, a decoding layer 413 and a feature extraction layer 414 connected in series.

[0078] During training:

[0079] The encoding layer 411 is used to generate the first underexposed training image Real_X (or the second generator G Y→X The generated image Fake_X) is obtained to obtain a first feature vector, that is, the encoding layer 411 extracts the first feature vector from the input image through a convolution operation;

[0080] The conversion layer 412 is used to obtain a second feature vector after the exposure feature vector is converted according to the input first feature vector. For example, the conversion layer 412 converts the feature vector of the image in the data domain X (the exposure feature vector representing underexposure) into the feature vector in the data domain Y (the exposure feature vector representing overexposure) by combining the dissimilar features of the image.

[0081] The decoding layer 413 is used to obtain a third feature vector according to the second feature vector input. For example, the decoding layer 413 obtains the third feature vector by performing a deconvolution operation on the second feature vector; and

[0082] The feature extraction layer 414 is used to obtain an exposure feature vector according to the input third feature vector. For example, the feature extraction layer 414 extracts the exposure feature vector from the third feature vector output by the decoding layer 413 through a convolution operation.

[0083] In this implementation, the structures of the encoding layer 411, the conversion layer 412 and the decoding layer 413 are similar to the encoder, the converter and the decoder connected in series included in the generator in the aforementioned recurrent generation network. The difference is that the output of the decoder of the generator in the aforementioned recurrent generation network is the generated image, while the output of the decoding layer 413 in this implementation is a feature vector. For example, the number of channels of the first training image Real_X (or image Fake_X) is usually 3, then the number of channels of the third feature vector output by the decoding layer 413 can be set to 128, 256, etc., so that after the feature extraction layer 414 performs feature extraction based on convolution operation, the number of channels of the output exposure feature vector is 3, which can be compared with the underexposed first training image Real_X (or the second generator G_X) belonging to the data domain X of the input feature extractor 410. Y→X The generated image Fake_X) implements the addition operation.

[0084] In a possible implementation, the feature extraction layer 414 is a residual network (Residual Network, ResNet). The residual network has the advantages of being easy to optimize and being able to improve accuracy by increasing considerable depth. The residual block inside it uses skip connections and solves the degradation problem of deep networks through residual learning.

[0085] It should be noted that in the first generator G X→Y In the case of including the feature extractor 410 and the addition operation layer 420, the second generator G Y→X The structure of can be directly adopted as the structure of the generator including the encoder, converter and decoder connected in series in the aforementioned recurrent generation network.

[0086] In addition, the second generator G Y→X Set to a structure including a feature extractor and an addition operation layer, or the first generator G X→Y and the second generator G Y→X All of them are set to include a structure of a feature extractor and an addition operation layer. These methods can achieve the feature extractor through training of a cyclic generation network architecture.

[0087] S202, inputting the first image into a feature extractor that has been trained through a cyclic generation network architecture to obtain an exposure feature vector.

[0088] In one possible implementation, the feature extractor includes a serially connected encoding layer, a conversion layer, a decoding layer, and a feature extraction layer;

[0089] Step S202 includes:

[0090] Inputting the first image into the encoding layer to obtain a first feature vector;

[0091] Inputting the first feature vector into a conversion layer to obtain a second feature vector converted from the exposure feature vector;

[0092] Inputting the second feature vector into a decoding layer to obtain a third feature vector; and

[0093] The third feature vector is input into the feature extraction layer to obtain an exposure feature vector.

[0094] Here, referring to the description of the training process in step S201, the feature extractor learns the ability to convert and extract exposure feature vectors through cyclic generation of network architecture training.

[0095] S203: performing multiplication operations on the exposure feature vector and a plurality of weight values ​​within a set value range respectively.

[0096] S204: performing an addition operation on the result of the multiplication operation and the first image to obtain a plurality of image samples with different exposure levels corresponding to the first image.

[0097] In a possible implementation, the plurality of weight values ​​are taken at equal intervals within the set value range. In this way, a series of image samples with uniform exposure distribution can be obtained.

[0098] In a possible implementation manner, the method for generating multi-exposure image samples provided in this embodiment further includes: adjusting the set value range.

[0099] For example, when the exposure information or exposure level (underexposure, normal exposure, overexposure) of the first image is known, the set value range can be adjusted in advance; or after obtaining a series of image samples with different exposures, the set value range can be adjusted according to the exposure conditions of multiple image samples to re-perform multiplication operations and subsequent addition operations, so as to re-obtain a series of image samples with different exposures.

[0100] The method for generating multi-exposure image samples provided in this embodiment includes two stages. Step S201 can be regarded as belonging to the training stage, and steps S202-S204 can be regarded as belonging to the data augmentation stage.

[0101] In the data augmentation phase, for the recurrent generative network architecture, only the feature extractor is used. Figure 5 As shown, the input first image of unlimited target type is respectively input into an adder and a feature extractor trained by a cyclic generative network architecture, and the feature extractor trained by the cyclic generative network architecture extracts the exposure feature vector A of the first image and outputs it to a multiplier, and the multiplier performs a multiplication on the exposure feature vector A of the first image and N weight values ​​w1, w2, ..., w in a range of, for example, [-2, 2]. N Perform multiplication operations to obtain exposure feature vectors A1, A2, ..., A N And output to the adder, the adder performs the exposure feature vector A1, A2, ..., A N An addition operation is performed with the first image respectively to obtain N image samples with different exposure degrees corresponding to the first image.

[0102] In a specific example, the first input image is such as Figure 6 The fourth image from the left of the seven images shown is an image with normal exposure. The weight value range is set to [-2, 2]. Seven weight values ​​are taken within this range, namely w1 = -2, w2 = -1.5, w3 = -1, w4 = 0, w5 = 1, w6 = 1.5, w7 = 2. The result of multiplying the exposure feature vector A of the first image with different weight values ​​is added to the original image of the first image to obtain 7 image samples with different exposures. The image samples corresponding to the weight values ​​w1 to w7 are shown in the figure. Figure 6The seven images are arranged from left to right. When the weight value is negative, the corresponding image sample is underexposed compared to the original image; when the weight value is zero, the corresponding image sample is the original image; when the weight value is positive, the corresponding image sample is overexposed compared to the original image.

[0103] For example, if the first input image is Figure 6 The second image from the left of the seven images shown (i.e., the underexposed image) can be adjusted by shifting the value range to the right, for example, setting the weight value range to [-1,3], [-1,4], etc., to obtain an exposure range of the image samples.

[0104] Another embodiment of the present invention provides a device for generating multi-exposure image samples, comprising:

[0105] A feature extraction module, configured to input the first image into a feature extractor trained by a cyclic generative network architecture to obtain an exposure feature vector;

[0106] a multiplication operation module, used to perform multiplication operations on the exposure feature vector and a plurality of weight values ​​within a set value range respectively; and

[0107] The addition operation module is used to perform an addition operation on the result of the multiplication operation and the first image to obtain a plurality of image samples with different exposure levels corresponding to the first image.

[0108] In a possible implementation, the feature extractor includes a coding layer, a conversion layer, a decoding layer, and a feature extraction layer connected in series;

[0109] Inputting the first image into a feature extractor that has been trained through a cyclic generation network architecture to obtain an exposure feature vector of the first image includes:

[0110] The encoding layer is used to obtain a first feature vector according to the input first image;

[0111] The conversion layer is used to obtain a second feature vector after the exposure feature vector is converted according to the input first feature vector;

[0112] The decoding layer is used to obtain a third feature vector according to the input second feature vector; and

[0113] The feature extraction layer is used to obtain an exposure feature vector based on the input third feature vector.

[0114] In a possible implementation, the feature extraction layer is a residual network.

[0115] It should be noted that the principle and workflow of the device for generating multi-exposure image samples provided in this embodiment are similar to the above-mentioned method for generating multi-exposure image samples. The relevant parts can be referred to the above description and will not be repeated here.

[0116] like Figure 7 As shown, a computer system or computer device suitable for implementing the device for generating multi-exposure image samples provided by the above embodiment includes a central processing module (CPU), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage part into a random access memory (RAM). Various programs and data required for the operation of the computer system are also stored in the RAM. The CPU, ROM and RAM are connected to each other through a bus. An input / output (I / O) interface is also connected to the bus.

[0117] The following components are connected to the I / O interface: an input part including a keyboard, a mouse, etc.; an output part including a liquid crystal display (LCD), etc. and a speaker, etc.; a storage part including a hard disk, etc.; and a communication part including a network interface card such as a LAN card, a modem, etc. The communication part performs communication processing via a network such as the Internet. A drive is also connected to the I / O interface as needed. Removable media, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., are installed on the drive as needed so that the computer program read therefrom is installed into the storage part as needed.

[0118] In particular, according to this embodiment, the process described in the flowchart above can be implemented as a computer software program. For example, this embodiment includes a computer program product, which includes a computer program tangibly contained on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part, and / or installed from a removable medium.

[0119] The flow chart and schematic diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product of the present embodiment. In this regard, each square box in the flow chart or schematic diagram can represent a part of a module, program segment or code, and a part of the above-mentioned module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the schematic diagram and / or flow chart, and the combination of the square boxes in the schematic diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0120] The modules involved in the description of this embodiment may be implemented by software or hardware. The modules described may also be arranged in a processor, for example, may be described as: a processor including a feature extraction module, a multiplication module and an addition module. The names of these modules do not constitute a limitation on the modules themselves in certain circumstances. For example, the multiplication module may also be described as a "multiplication module".

[0121] For example, Figure 1 The structure and function of the generation server 20 shown can be referred to Figure 7 The computer system or computer device shown is suitable for implementing the device for generating multiple exposure image samples provided in the above-mentioned embodiment, and the computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the program, it implements: obtaining a feature extractor through cyclic generative network architecture training; inputting a first image into the feature extractor that has been trained through cyclic generative network architecture to obtain an exposure feature vector; performing multiplication operations on the exposure feature vector and a plurality of weight values ​​within a set value range respectively; performing addition operations on the result of the multiplication operation and the first image to obtain a plurality of image samples with different exposures corresponding to the first image.

[0122] For example, Figure 1 The structure and function of the training server 10 shown can also be referred to Figure 7 The computer system or computer device shown is suitable for implementing the device for generating multi-exposure image samples provided in the above embodiment, and the computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the program, it implements: obtaining a feature extractor by training a cyclic generative network architecture.

[0123] As another aspect, the present embodiment further provides a non-volatile computer storage medium, which may be the non-volatile computer storage medium included in the above-mentioned device in the above-mentioned embodiment, or may be a non-volatile computer storage medium that exists independently and is not installed in the terminal. The above-mentioned non-volatile computer storage medium stores one or more programs, and when the above-mentioned one or more programs are executed by a device, the above-mentioned device: obtains a feature extractor through cyclic generative network architecture training; inputs the first image into the feature extractor that has been trained through the cyclic generative network architecture to obtain an exposure feature vector; multiplies the exposure feature vector with multiple weight values ​​within a set value range respectively; and adds the result of the multiplication operation to the first image to obtain multiple image samples with different exposures corresponding to the first image.

[0124] In the description of the present invention, it should be noted that the terms "upper", "lower", etc. indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific position, be constructed and operated in a specific position, and therefore cannot be understood as a limitation on the present invention. Unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be a connection between the two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.

[0125] It should also be noted that, in the description of the present invention, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0126] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not limitations on the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the embodiments here. All obvious changes or modifications derived from the technical solution of the present invention are still within the protection scope of the present invention.

Claims

1. A method for generating multi-exposure image samples, characterized in that: include: Inputting the first image into a feature extractor trained by a recurrent generative network architecture to obtain an exposure feature vector; Performing multiplication operations on the exposure feature vector and a plurality of weight values ​​within a set value range respectively; as well as performing an addition operation on a result of the multiplication operation and the first image to obtain a plurality of image samples with different exposure levels corresponding to the first image; Before inputting the first image into the feature extractor trained by the cyclic generative network architecture, the method further comprises: obtaining the feature extractor by training the cyclic generative network architecture; The feature extractor obtained by training the cyclic generation network architecture includes: Acquire a plurality of first training images and a plurality of second training images, wherein one of the first training image and the second training image is an underexposed image, and the other is an overexposed image; Constructing a cyclic generative network architecture, the cyclic generative network architecture comprising a first generator, a first discriminator, a second generator and a second discriminator, wherein the first generator comprises a feature extractor and an addition operation layer, the addition operation layer is used to perform an addition operation on the input and output of the feature extractor to obtain the output of the first generator, the output of the first generator is respectively used as the input of the second discriminator and the second generator, and the output of the second generator is respectively used as the input of the first discriminator and the first generator; Inputting the plurality of first training images into the first generator and the first discriminator respectively, and inputting the plurality of second training images into the second generator and the second discriminator, so as to train the recurrent generative network architecture and obtain a trained feature extractor; The training image set used by the feature extractor obtained by training the cyclic generative network architecture includes a first training image set and a second training image set, the first training image set includes a plurality of underexposed first training images, the second training image set includes a plurality of overexposed second training images, and the underexposed first training images belong to the data domain X and the overexposed second training images belong to the data domain Y. The cyclic generative network architecture includes: a first generator G X→Y , the first discriminator D X , the second generator G Y→X and the second discriminator D Y , the first generator G X→Y The output Fake_Y is used as the second discriminator D Y and the second generator G Y→X The input of the second generator G Y→X The output Fake_X is used as the first discriminator D X and the first generator G X→Y Input; During the training process, for two data domains X and Y: The first underexposed training image Real_X belonging to the data domain X is input into the first generator G X→Y , the first generator G X→Y The task is to convert the underexposed first training image Real_X belonging to the data domain X into an overexposed image Fake_Y in the data domain Y. The purpose of this process in training is to make the first generator G X→Y Learning the features of the data domain Y, after training, the first generator G X→Y What is learned is the ability to transform the exposure feature vector; The overexposed second training image Real_Y belonging to the data domain Y is input into the second generator G Y→X , the second generator G Y→X The task is to convert the overexposed second training image Real_Y belonging to the data domain Y into the underexposed image Fake_X of the data domain X. The purpose of this process in training is to make the second generator G Y→X Learning the features of data domain X, after training, the second generator G Y→X What is learned is the ability to transform the exposure feature vector; The underexposed first training image Real_X belonging to the data domain X is also input into the first discriminator D X , the second generator G Y→X The generated image Fake_X is also input into the first discriminator D X , the first discriminator D X The task is to identify the input image as the second generator G Y→X The generated image Fake_X is still the underexposed first training image Real_X belonging to the data domain X, and the formula is the loss of 0,1 binary classification; The overexposed second training image Real_Y belonging to the data domain Y is also input into the second discriminator D Y , the first generator G X→Y The generated image Fake_Y is also input into the second discriminator D Y , the second discriminator D Y The task is to identify the input image as the first generator G X→Y The generated image Fake_Y is still the overexposed second training image Real_Y belonging to the data domain Y, and the formula is the loss of 0,1 binary classification.

2. The method according to claim 1, characterized in that: The feature extractor comprises a coding layer, a conversion layer, a decoding layer and a feature extraction layer connected in series; Inputting the first image into a feature extractor that has been trained through a cyclic generation network architecture to obtain an exposure feature vector of the first image includes: Inputting the first image into the encoding layer to obtain a first feature vector; Inputting the first feature vector into a conversion layer to obtain a second feature vector converted from the exposure feature vector; Inputting the second feature vector into a decoding layer to obtain a third feature vector; and The third feature vector is input into the feature extraction layer to obtain an exposure feature vector.

3. The method according to claim 2, characterized in that The feature extraction layer is a residual network.

4. The method according to claim 1, characterized in that: The multiple weight values ​​are taken at equal intervals within the set value range.

5. The method according to claim 1, characterized in that: The method also includes: adjusting the set value range.

6. A device for generating multi-exposure image samples, characterized in that: include: A feature extraction module, configured to input the first image into a feature extractor trained by a cyclic generative network architecture to obtain an exposure feature vector; A multiplication operation module, used for performing multiplication operations on the exposure feature vector and a plurality of weight values ​​within a set value range respectively; as well as an addition operation module, configured to perform an addition operation on a result of the multiplication operation and the first image to obtain a plurality of image samples of different exposure levels corresponding to the first image; The device also includes a training module for obtaining a feature extractor by training a cyclically generated network architecture; The training module is further used to: Acquire a plurality of first training images and a plurality of second training images, wherein one of the first training image and the second training image is an underexposed image, and the other is an overexposed image; Constructing a cyclic generative network architecture, the cyclic generative network architecture comprising a first generator, a first discriminator, a second generator and a second discriminator, wherein the first generator comprises a feature extractor and an addition operation layer, the addition operation layer is used to perform an addition operation on the input and output of the feature extractor to obtain the output of the first generator, the output of the first generator is respectively used as the input of the second discriminator and the second generator, and the output of the second generator is respectively used as the input of the first discriminator and the first generator; Inputting the plurality of first training images into the first generator and the first discriminator respectively, and inputting the plurality of second training images into the second generator and the second discriminator, so as to train the recurrent generative network architecture and obtain a trained feature extractor; The training image set used by the feature extractor obtained by training the cyclic generative network architecture includes a first training image set and a second training image set, the first training image set includes a plurality of underexposed first training images, the second training image set includes a plurality of overexposed second training images, and the underexposed first training images belong to the data domain X and the overexposed second training images belong to the data domain Y. The cyclic generative network architecture includes: a first generator G X→Y , the first discriminator D X , the second generator G Y→X and the second discriminator D Y , the first generator G X→Y The output Fake_Y is used as the second discriminator D Y and the second generator G Y→X The input of the second generator G Y→X The output Fake_X is used as the first discriminator D X and the first generator G X→Y Input; During the training process, for two data domains X and Y: The first underexposed training image Real_X belonging to the data domain X is input into the first generator G X→Y , the first generator G X→Y The task is to convert the underexposed first training image Real_X belonging to the data domain X into an overexposed image Fake_Y in the data domain Y. The purpose of this process in training is to make the first generator G X→Y Learning the features of the data domain Y, after training, the first generator G X→Y What is learned is the ability to transform the exposure feature vector; The overexposed second training image Real_Y belonging to the data domain Y is input into the second generator G Y→X , the second generator G Y→X The task is to convert the overexposed second training image Real_Y belonging to the data domain Y into the underexposed image Fake_X of the data domain X. The purpose of this process in training is to make the second generator G Y→X Learning the features of data domain X, after training, the second generator G Y→X What is learned is the ability to transform the exposure feature vector; The underexposed first training image Real_X belonging to the data domain X is also input into the first discriminator D X , the second generator G Y→X The generated image Fake_X is also input into the first discriminator D X , the first discriminator D X The task is to identify the input image as the second generator G Y→X The generated image Fake_X is still the underexposed first training image Real_X belonging to the data domain X, and the formula is the loss of 0,1 binary classification; The overexposed second training image Real_Y belonging to the data domain Y is also input into the second discriminator D Y , the first generator G X→Y The generated image Fake_Y is also input into the second discriminator D Y , the second discriminator D Y The task is to identify the input image as the first generator G X→Y The generated image Fake_Y is still the overexposed second training image Real_Y belonging to the data domain Y, and the formula is the loss of 0,1 binary classification.

7. The device according to claim 6, characterized in that The feature extractor comprises a coding layer, a conversion layer, a decoding layer and a feature extraction layer connected in series; The encoding layer is used to obtain a first feature vector according to the input first image; The conversion layer is used to obtain a second feature vector after the exposure feature vector is converted according to the input first feature vector; The decoding layer is used to obtain a third feature vector according to the input second feature vector; and The feature extraction layer is used to obtain an exposure feature vector based on the input third feature vector.

8. The device according to claim 7, characterized in that The feature extraction layer is a residual network.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 5 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images