A method and system for generating perspective-varying images using a generative adversarial network

Abstract

The present invention discloses a method and system for generating variable-viewpoint images based on a generative adversarial network. The method utilizes a cascaded generative network to generate images with more refined target perspectives. A spatial feature conversion module is added to the first-level generative network to convert common semantic features, allowing the generator to obtain more semantic information about the target perspective. A multi-scale channel attention module is added to the second-level generative network to obtain attention weight maps for different channels, generating target perspective images with more detailed textures. The method provided by the present invention can generate sample data from multiple perspectives using a small amount of real sample data. The present invention can increase the number of training samples for the target detection and recognition network, thereby improving the detection rate of the entire target detection and recognition task.

Description

Technical Field

[0001] The present invention belongs to the field of image generation, and more specifically, relates to a method and system for generating perspective-varying images based on a generative adversarial network. Background Art

[0002] Image generation technology, a core task in computer vision, can generate numerous realistic images. This provides more samples for target detection, semantic segmentation, and other fields, supporting improved algorithm model performance. Military applications, such as aircraft guidance, require algorithms to accurately identify targets. However, due to the difficulty of acquiring images of specific targets and the uncooperative state of the targets, the number of image samples containing the target is often small and the angles are relatively limited. Image generation technology, however, can generate more samples to train target detection models, thereby improving recognition accuracy. Therefore, image generation technology has broad application prospects.

[0003] However, the images generated by existing multi-view image methods based on generative adversarial networks lack certain texture detail information, and the generated images are not realistic enough, which leads to a decrease in the recognition accuracy of subsequent tasks such as target detection and recognition. Summary of the Invention

[0004] In response to the above-mentioned defects or improvement needs of the prior art, the present invention provides a method and system for generating variable-viewpoint images based on a generative adversarial network, thereby solving the problem that the images generated by the existing multi-viewpoint image method lack texture detail information, thereby reducing the recognition accuracy of subsequent target detection and recognition tasks.

[0005] To achieve the above objectives, according to a first aspect of the present invention, a method for generating a variable-viewing-angle image based on a generative adversarial network is provided, comprising:

[0006] S1, including the original view image I a Perform semantic segmentation on A in the multi-view image pair of the target perspective real image A to obtain the corresponding semantic segmentation map S g ;

[0007] S2, constructing a generative network; the generative network includes a cascade of:

[0008] The first-level generation network includes multiple cascaded spatial feature residual modules RSB; each RSB includes multiple cascaded combinations, each combination includes interconnected spatial feature transformation modules SFT and convolutional layers; the first RSB is used to extract target visual features based on the input splicing features and public semantic features, and the other RSBs are used to use the target visual features output by the previous RSB as input features H, and extract target visual features based on H and the public semantic features; the last RSB outputs the final target visual feature map I g `; The SFT is used to perform convolution processing on the common semantic features to obtain the scaling feature matrix T and the translation feature matrix S, and perform addition operation on the corresponding elements of the intermediate result obtained by performing point multiplication of the splicing feature and T and S to obtain the output target visual feature; the common semantic feature is the sum of the S g Feature extraction is performed to obtain the splicing feature. a With S g Splicing obtained;

[0009] The secondary generation network includes a mutually cascaded hole convolution multi-scale feature extraction module AMS and a multi-channel attention MCA; the AMS includes a combination of N hole convolutions with different expansion rates, which are used to extract the feature according to I g ` and S g Generate N corresponding feature maps with different receptive fields; the MCA is used to process the multi-scale feature map H' obtained by splicing the N feature maps with different receptive fields to obtain the target perspective generated image I g ``;

[0010] S3, with I g ``The goal is to train the generative network with the highest similarity to A;

[0011] S4, inputting the original perspective image to be converted and the semantic segmentation map of the target perspective image into the trained generation network to obtain the converted target perspective generated image.

[0012] According to a second aspect of the present invention, a device for generating a variable-viewing-angle image based on a generative adversarial network is provided, comprising:

[0013] The first processing module is used to process the original view image I a Perform semantic segmentation on A in the multi-view image pair of the target perspective real image A to obtain the corresponding semantic segmentation map S g ;

[0014] The second processing module is used to construct a generation network; the generation network includes a cascade of:

[0015] The first-level generation network includes multiple cascaded spatial feature residual modules RSB; each RSB includes multiple cascaded combinations, each combination includes interconnected spatial feature transformation modules SFT and convolutional layers; the first RSB is used to extract target visual features based on the input splicing features and public semantic features, and the other RSBs are used to use the target visual features output by the previous RSB as input features H, and extract target visual features based on H and the public semantic features; the last RSB outputs the final target visual feature map I g `; The SFT is used to perform convolution processing on the common semantic features to obtain the scaling feature matrix T and the translation feature matrix S, and perform addition operation on the corresponding elements of the intermediate result obtained by performing point multiplication of the splicing feature and T and S to obtain the output target visual feature; the common semantic feature is the sum of the S g Feature extraction is performed to obtain the splicing feature. a With S g Splicing obtained;

[0016] The secondary generation network includes a mutually cascaded hole convolution multi-scale feature extraction module AMS and a multi-channel attention MCA; the AMS includes a combination of N hole convolutions with different expansion rates, which are used to extract the feature according to I g ` and S g Generate N corresponding feature maps with different receptive fields; the MCA is used to process the multi-scale feature map H' obtained by splicing the N feature maps with different receptive fields to obtain a generated image I under the target perspective g `` and enter;

[0017] The third processing module is used to g ``The goal is to train the generative network with the highest similarity to A;

[0018] The fourth processing module is used to input the original perspective image to be converted and the semantic segmentation map of the target perspective image into the trained generation network to obtain the generated image under the converted target perspective.

[0019] According to a third aspect of the present invention, there is provided a system for generating a perspective-varying image based on a generative adversarial network, comprising: a computer-readable storage medium and a processor;

[0020] The computer-readable storage medium is used to store executable instructions;

[0021] The processor is configured to read the executable instructions stored in the computer-readable storage medium and execute the method according to the first aspect.

[0022] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to execute the method described in the first aspect.

[0023] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0024] The present invention provides a method for generating variable-viewpoint images based on a generative adversarial network. This method utilizes a cascaded generative network to generate a more refined image from the target perspective. A spatial feature conversion module is added to the first-level generative network to convert common semantic features, allowing the generator to obtain more semantic information about the target perspective. A multi-scale channel attention module is added to the second-level generative network to obtain attention weight maps for different channels, generating a target perspective image with more detailed textures. Specifically, the following are embodied:

[0025] 1. The present invention adds a dilated convolution multi-scale feature extraction module to extract features of multiple scales of the image without reducing the image resolution, thereby retaining the detailed information on the original image to the greatest extent and enhancing the realism of the generated image.

[0026] 2. The present invention adds a spatial feature conversion module, which makes fuller use of the semantic information of the semantic segmentation map of the target perspective in the input network, which helps the generator generate images with more realistic target perspectives.

[0027] 3. The present invention adopts the form of a cascade generative network. After the first-level generative network generates an image, it is input into a second-level generative network composed of a multi-scale multi-channel attention module to obtain the attention weight map of each channel, which is multiplied with the intermediate result to obtain the final generated image.

[0028] In summary, the method provided by the present invention can generate sample data from multiple perspectives through a small amount of real sample data, increase the number of training samples of the target detection and recognition network, and improve the detection rate of the entire target detection and recognition task. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A flow chart of a method for generating a variable-viewing angle image based on a generative adversarial network according to an embodiment of the present invention;

[0030] Figure 2 A schematic diagram of the structure of a generation network provided in an embodiment of the present invention;

[0031] Figure 3 A schematic diagram of the structure of a spatial feature residual module provided by an embodiment of the present invention;

[0032] Figure 4A schematic diagram of the structure of a spatial feature conversion module provided in an embodiment of the present invention;

[0033] Figure 5 A schematic diagram of the structure of a multi-scale channel attention module provided by an embodiment of the present invention;

[0034] Figure 6 (a), (b), (c), (d), and (f) are respectively the original perspective image, the first-level generated image, the final generated image, the real image, and the semantic segmentation map of the target visual real image provided by the embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0036] The embodiment of the present invention provides a method for generating a variable-viewing angle image based on a generative adversarial network. Figure 1 As shown, it includes: training stages S1-S3 and application stage S4;

[0037] Training phase:

[0038] S1, including the original view image I a Perform semantic segmentation on A in the multi-view image pair of the target perspective real image A to obtain the corresponding semantic segmentation map S g ;

[0039] Specifically, multi-view image pairs in a public dataset are selected; or, a sufficient number of image sequences are acquired, and then the image sequences are subjected to frame extraction processing, and then processed into image pairs with one-to-one correspondence between multiple viewpoints.

[0040] Furthermore, before step S1, the method further includes:

[0041] The multi-view image pairs are pre-processed to adjust their resolution and aspect ratio.

[0042] For example, the image pairs in the multi-view image pairs are pre-processed, and the rectangular image (ie, the original view image I a The center of the real image A) is cropped to the target perspective to make it an image with an aspect ratio of 1:1 and a resolution of 256x 256.

[0043] Then, semantic segmentation is performed on the images in the image conversion pair to obtain their corresponding segmentation maps, and the four images obtained are combined into the "original view image Ia -Target perspective real image -A original perspective image semantic segmentation map -Target perspective image semantic segmentation map S g ” image group.

[0044] S2, builds a generation network; Figure 2 As shown, the generation network includes a cascaded primary generation network and a secondary generation network;

[0045] The first-level generation network includes multiple cascaded spatial feature residual modules RSB; each RSB includes multiple cascaded combinations, each combination includes interconnected spatial feature transformation modules SFT and convolutional layers; the first RSB is used to extract target visual features based on the input splicing features and public semantic features, and the other RSBs are used to use the target visual features output by the previous RSB as input features H, and extract target visual features based on H and the public semantic features; the last RSB outputs the final target visual feature map I g `; The SFT is used to perform convolution processing on the common semantic features to obtain the scaling feature matrix T and the translation feature matrix S, and perform addition operation on the corresponding elements of the intermediate result obtained by performing point multiplication of the splicing feature and T and S to obtain the output target visual feature; the common semantic feature is the sum of the S g Feature extraction is performed to obtain the splicing feature. a With S g Splicing obtained;

[0046] The secondary generation network includes a cascaded hole convolution multi-scale feature extraction module AMS and a multi-channel attention MCA; the AMS includes a combination of N hole convolutions with different expansion rates, which are used to extract the feature according to I g ` and S g Generate N corresponding feature maps with different receptive fields; the MCA is used to process the multi-scale feature map H' obtained by splicing the N feature maps with different receptive fields to obtain the target perspective generated image I g ``.

[0047] Specifically, the first-level generation network extracts the features of the semantic segmentation map of the target view image as the common semantic features through the spatial feature conversion module, and combines them with the original features (i.e., a and S g The second-level generation uses a multi-scale channel attention module to extract the attention weight map of each channel and generate a more refined image.

[0048] Furthermore, the MCA is used to perform a two-way convolution process on H' to map the multi-scale features of H' to an intermediate generated image of M dimensions and an attention weight of M dimensions respectively; and perform a one-to-one multiplication of the attention weights of the M dimensions with the intermediate generated image of the M dimensions, and then perform an addition operation on the corresponding elements to obtain I g ``.

[0049] Specifically, a spatial feature transformation module replaces the convolutional module in the traditional backbone network. The advantage of the spatial transformation module is that it can transform common semantic features in various directions and scales, allowing the generative network to have more semantic information from the target perspective, thereby generating images with more spatial texture. In addition, a cascaded generative network is used. Based on the generated image from the first-level generative network, the image is input into a second-level generative network composed of multi-scale channel attention modules. This produces a more refined attention weight map and intermediate results, and finally a target perspective image with more detailed information.

[0050] 1) Spatial feature conversion module

[0051] The network structure of the RSB module is as follows Figure 3 As shown, the network structure of the SFT module is as follows Figure 4 As shown in the figure, the common semantic features are learned through two convolutional modules consisting of three convolutional layers, respectively, to obtain the scaling feature matrix (S) and the translation feature matrix (T). The input feature H is dot-multiplied with the scaling feature obtained by the SFT module to obtain the corresponding intermediate result. The intermediate result is then added to the translation feature obtained by the SFT module by the corresponding element-wise addition operation. After the scaling and translation operations, the input feature matrix will obtain the features of the target perspective at the same scale. In the subsequent feature extraction process, the semantic information input by the SFT module can be effectively utilized to generate a more realistic image from the target perspective.

[0052] Assume that the entire public semantic feature is κ, and operate ω on κ to obtain the scaling feature matrix S and the translation feature matrix T. The input feature is x and the output feature is y. The corresponding conversion relationship is shown as follows:

[0053] (S,T)=ω(κ)

[0054] y=S⊙x+T

[0055] The input of the SFT module is the common semantic feature κ and the input feature x. The above SFT module is combined with the residual connection concept of ResNet, and the convolution module in the Res Block is replaced by the SFT module to obtain the ResSFT Block. After the input image passes through the generator composed of multiple ResSFT blocks, the scene from the original perspective and the foreground and background information from the target perspective are extracted. After passing through a decoder-encoder structure, the image from the target perspective is generated.

[0056] 2) Atrous Multi-Scale Channel Attention Module AMSMCA

[0057] AMSMCA consists of two parts: dilated convolution multi-scale feature extraction and multi-channel attention generation. That is, AMSMCA includes AMS module and MCA module. The structures of AMS and MCA are as follows: Figure 5 shown.

[0058] (a) Dilated convolution multi-scale feature extraction module

[0059] The Atrous Multi Scale (AMS) feature extraction module is a feature map structure composed of multiple atrous convolutions with different dilation rates. This structure concatenates feature maps with different receptive fields to obtain a feature map with multiple receptive fields.

[0060] The output feature map consists of five parts. The first part is the original feature dimension, with a receptive field of 1x1. The second part is a normal convolution obtained through a 3x3 convolution kernel with a dilation rate of r=1, with a receptive field of 3x3. The third part is the feature map obtained through two groups of 3x3 normal convolutions. The fourth part is the feature map obtained through a mixed group of dilated convolutions of 3x3, r=1 and 3x3, r=2, with a receptive field of 7x7. The fifth part is the feature map obtained through a mixed group of dilated convolutions of 3x3, r=1 and 3x3, r=3, with a receptive field of 9x9. After the input feature map passes through the AMS module, a combination of five feature maps with different receptive field sizes is obtained. This facilitates the integration of information at different scales, thereby obtaining complete, real-time, and accurate object and scene information. It can effectively complete multi-view image generation tasks in complex backgrounds, improving image generation quality.

[0061] Let the input feature map be F in , the output feature map is F out , then the relationship is as follows:

[0062] F out =concat(F in ,F 3x3 ,F 5x5,F 7x7 ,F 9x9 )

[0063] (b) Multi-channel attention generation

[0064] In previous multi-view image conversion tasks, the target view image is often generated in a three-channel or single-channel space. The feature selection generation of these channel numbers cannot model the complex scene of the entire multi-view conversion. In the multi-channel attention generation module, the multi-scale features extracted and processed by the multi-scale feature extraction module are passed through two networks and convolution operations with corresponding M convolution kernels to map the multi-scale features to an intermediate generated image of M dimensions. and the attention weights of M dimensions Generate images in the middle of a build When the multi-scale input feature H' is passed through M convolution kernels Then it is followed by a tanh(·) nonlinear activation function. When H' is passed through M convolution kernels Then, a Softmax(·) normalization operation is performed on the attention weight of each channel. Its mathematical expression is as follows:

[0065]

[0066]

[0067] S3, the discriminant network is based on I g ``The goal is to obtain the highest similarity with A and train the generative network.

[0068] Specifically, if Figure 2 As shown in the figure, through the discriminant network D, the training set images are used to train the constructed generative network to obtain the corresponding generative network parameters, and the weight file with better generation effect is saved.

[0069] Application stage:

[0070] S4, inputting the original perspective image to be converted and the semantic segmentation map of the target perspective image into the trained generation network to obtain the converted target perspective generated image.

[0071] Specifically, the application stage includes the steps of:

[0072] S41, the original view image and the target view segmentation map are processed by the trained generative network model. A first-level generated image (i.e., the final target visual feature map I) with the same resolution as the original view (e.g., 256x 256) is obtained. g `).

[0073] S42, the first-level generated image is fed into the second-level generated network model to obtain the attention weight map of each channel, and the intermediate generated features are multiplied by the attention weight map to obtain the final generated image (i.e., the target perspective generated image I g ``).

[0074] Furthermore, the loss function used in the training is shown below:

[0075] Total loss total :

[0076]

[0077] Among them, λ1, λ2, and λ3 are penalty factors, all of which are set to 1.

[0078] 1) Generate adversarial loss:

[0079] l cGAN =l cGAN (I a ,I' g )+λ`l cGAN (I a ,I" g )

[0080] Among them, λ` takes the value of 1. cGAN (I a ,I' g ), l cGAN (I a ,I" g ) is calculated as follows:

[0081]

[0082]

[0083] Among them, I a is the original view image, I' g is the first-level generated image, I" g is the final generated image.

[0084] 2) Pixel-level constraint loss:

[0085] In the entire generation network, the generated target perspective image has a first-level generation network output I' g , and the output image of the final target perspective after the AMSMCA module I" g , the real image corresponding to the two output images is I g , then we can get the image pair [I g ,I' g ] and [I g,I" g ] calculates the L1 distance in pixel-level Euclidean space as a constraint to guide the learning iteration of the generative network, where the image pair [I g ,I' g ]’s L1 distance l L1 (I g ,I' g ) is calculated as follows:

[0086]

[0087] Similarly, the image pair [I g ,I" g ]’s L1 distance l L1 (I g ,I" g ), image pair [S g ,S' g ]’s L1 distance l L1 (S g ,S' g ), image pair [S g ,S" g ]’s L1 distance l L1 (S g ,S" g ) is shown in the following formula:

[0088]

[0089]

[0090]

[0091] Total pixel-level constraint loss L1 as follows:

[0092] l L1 =λ1`l L1 (I g ,I' g )+λ2`l L1 (I g ,I" g )+λ3`l L1 (S g ,S' g )+λ4`l L1 (S g ,S" g )

[0093] Among them, λ1` is 100, λ2` is 200, λ3` is 1, and λ4` is 2.

[0094] 3) Total variation regularization loss

[0095] In order to make the final generated image I" g It looks smoother and has a better effect on the generated I" g Calculate the total variation loss l tv , its essence is to find the partial derivatives of the generated image in the width and height directions, and then get the sum of the partial derivatives. Let the width direction of the image be x and the height direction be y, then the total variation loss l tv The calculation formula is as follows:

[0096]

[0097] Among them, m is the total number of rows of the generated image, n is the total number of columns of the generated image, i is the current row number, j is the current column number, Gray (i,j) Represents the generated image I" g The grayscale value of row i and column j in .

[0098] The method provided by the present invention is further illustrated below with a specific example.

[0099] 1. Training phase

[0100] 1.1 Sample Generation

[0101] Samples are divided into training samples and test samples. The main process of obtaining training samples is as follows:

[0102] The experimental dataset used is the publicly available visible light dataset (CVUSA). Created by researchers at the University of North Carolina at Chapel Hill, the CVUSA dataset contains a large number of aerial and street-level image pairs, providing a challenging environment for multi-view image generation. The dataset contains 44,416 image pairs, of which 35,532 are used as training data and 8,884 as test data.

[0103] 1.2 Image Preprocessing

[0104] After obtaining the original training data, it needs to be preprocessed so that it can meet the input conditions of the network. In this chapter's experiment, the data is cropped so that its final image resolution meets the input constraints of the network, which is 256x 256. The specific processing steps are as follows:

[0105] (1) Perform center cropping on the rectangular image to make it an image with a length-to-width ratio of 1:1.

[0106] (2) Downsample the image obtained in the first step to a uniform resolution of 256 x 256. After these two steps, a training dataset is obtained that retains most of the original image scene.

[0107] (3) The different angles of each scene in the dataset are combined in pairs to obtain 35,532 sets of training data and 8,884 sets of test data.

[0108] (4) Perform semantic segmentation on the image pair obtained in the third step to obtain its corresponding segmentation map, and combine the four obtained images into an image group of "original perspective image-target perspective real image-original perspective image semantic segmentation map-target perspective image semantic segmentation map".

[0109] 1.3 Network Construction

[0110] This method is a general method. The generation network includes a first-level generation network containing a spatial feature conversion module and a second-level generation network containing a multi-scale channel attention module. The structure of the generation network can use a common encoder-decoder structure. Preferably, this example uses a first-level generator composed of 9 ResSFT blocks. A multi-scale channel attention module with void convolution is used. The overall network structure is as follows Figure 3 shown.

[0111] 1.4 Loss Function Design and Network Training

[0112] The network training process and loss function are described as follows:

[0113] The loss in network training is generated by the adversarial loss l cGAN , pixel-level constraint loss l L1 , total variation regularization loss l tv The total loss is composed of three weighted parts.

[0114] Total loss total :

[0115]

[0116] Among them, λ1, λ2, and λ3 are penalty factors, all of which are set to 1.

[0117] 1) Generate adversarial loss:

[0118] l cGAN =l cGAN (I a ,I' g )+λl cGAN (I a ,I" g )

[0119] Among them, λ is 1. cGAN (I a ,I' g ), l cGAN (I a ,I" g) is calculated as follows:

[0120]

[0121]

[0122] Among them, I a is the original view image, I' g is the first-level generated image, I" g is the final generated image.

[0123] 2) Pixel-level constraint loss:

[0124] In the entire generation network, the generated target perspective image has a first-level generation network output I' g , and the final target perspective output image after AMS and MCA modules I" g , the real image corresponding to the two output images is I g , then we can get the image pair [I g ,I' g ] and [I g ,I" g ] calculates the L1 distance in pixel-level Euclidean space as a constraint to guide the learning iteration of the generative network, where the image pair [I g ,I' g ]’s L1 distance l L1 (I g ,I' g ) is calculated as follows:

[0125]

[0126] Similarly, the image pair [I g ,I" g ]’s L1 distance l L1 (I g ,I" g ), image pair [S g ,S' g ]’s L1 distance l L1 (S g ,S' g ), image pair [S g ,S" g ]’s L1 distance l L1 (S g ,S" g ) is shown in the following formula:

[0127]

[0128]

[0129]

[0130] The total pixel-level constraint loss l L1 as follows:

[0131] l L1 =λ1l L1 (I g ,I' g )+λ2l L1 (I g ,I" g )+λ3l L1 (S g ,S' g )+λ4l L1 (S g ,S" g )

[0132] Among them, λ1 is 100, λ2 is 200, λ3 is 1, and λ4 is 2.

[0133] 3) Total variation regularization loss

[0134] In order to make the final generated image I" g It looks smoother and has a better effect on the generated I" g Calculate the total variation loss l tv , its essence is to find the partial derivatives of the generated image in the width and height directions, and then get the sum of the partial derivatives. Let the width direction of the image be x and the height direction be y, then the total variation loss l tv The calculation formula is as follows:

[0135]

[0136] Among them, m is the total number of rows of the generated image, n is the total number of columns of the generated image, i is the current row number, j is the current column number, Gray (i,j) Represents the generated image I" g The grayscale value of row i and column j in .

[0137] 2. Application stage

[0138] The application phase of the algorithm mainly consists of two steps:

[0139] 1) Figure 6 The original view image and the target view segmentation map shown in (a) are processed by the trained generative network model. A first-level generated image with the same resolution of 256x 256 as the original view is obtained, as shown in Figure 6 As shown in (b) in .

[0140] 2) The first-level generated image is fed into the second-level generated network model to obtain the attention weight map of each channel, and the intermediate generated features are multiplied by the attention weight map to obtain the final generated image. The final generated image is as follows Figure 6 As shown in (c), the semantic segmentation map of the target perspective real image is as follows Figure 6 As shown in (f) in the figure, the real image is as follows Figure 6 The performance of the algorithm is compared with that of Selection GAN in Table 1.

[0141] Table 1 Algorithm performance test results

[0142]

[0143] The following describes the variable-viewing angle image generation device based on a generative adversarial network provided by the present invention. The variable-viewing angle image generation device based on a generative adversarial network described below and the variable-viewing angle image generation method based on a generative adversarial network described above can refer to each other.

[0144] An embodiment of the present invention provides a device for generating a variable-viewing-angle image based on a generative adversarial network, comprising:

[0145] The first processing module is used to process the original view image I a Perform semantic segmentation on A in the multi-view image pair of the target perspective real image A to obtain the corresponding semantic segmentation map S g ;

[0146] The second processing module is used to construct a generation network; the generation network includes a cascade of:

[0147] The first-level generation network includes multiple cascaded spatial feature residual modules RSB; each RSB includes multiple cascaded combinations, each combination includes interconnected spatial feature transformation modules SFT and convolutional layers; the first RSB is used to extract target visual features based on the input splicing features and public semantic features, and the other RSBs are used to use the target visual features output by the previous RSB as input features H, and extract target visual features based on H and the public semantic features; the last RSB outputs the final target visual feature map I g `; The SFT is used to perform convolution processing on the common semantic features to obtain the scaling feature matrix T and the translation feature matrix S, and perform addition operation on the corresponding elements of the intermediate result obtained by performing point multiplication of the splicing feature and T and S to obtain the output target visual feature; the common semantic feature is the sum of the S g Feature extraction is performed to obtain the splicing feature. a With S g Splicing obtained;

[0148] The secondary generation network includes a mutually cascaded hole convolution multi-scale feature extraction module AMS and a multi-channel attention MCA; the AMS includes a combination of N hole convolutions with different expansion rates, which are used to extract the feature according to I g ` and S g Generate N corresponding feature maps with different receptive fields; the MCA is used to process the multi-scale feature map H' obtained by splicing the N feature maps with different receptive fields to obtain a generated image I under the target perspective g `` and enter;

[0149] The third processing module is used to g ``The goal is to train the generative network with the highest similarity to A;

[0150] The fourth processing module is used to input the original perspective image to be converted and the semantic segmentation map of the target perspective image into the trained generation network to obtain the generated image under the converted target perspective.

[0151] An embodiment of the present invention provides a system for generating a variable-viewing-angle image based on a generative adversarial network, comprising: a computer-readable storage medium and a processor;

[0152] The computer-readable storage medium is used to store executable instructions;

[0153] The processor is configured to read the executable instructions stored in the computer-readable storage medium and execute the method described in any one of the above embodiments.

[0154] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to execute the method described in any of the above embodiments.

[0155] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for generating a variable-view image based on a generative adversarial network, characterized in that: include: S1, including the original view image I a Perform semantic segmentation on A in the multi-view image pair of the target perspective real image A to obtain the corresponding semantic segmentation map S g ; S2, constructing a generative network; the generative network includes a cascade of: The first-level generation network includes multiple cascaded spatial feature residual modules (RSBs); each RSB includes multiple cascaded combinations, and each combination includes interconnected spatial feature transformation modules (SFTs) and convolutional layers; The first RSB is used to extract the target visual features based on the input splicing features and public semantic features. The other RSBs are used to take the target visual features output by the previous RSB as the input features H and extract the target visual features based on H and the public semantic features. The last RSB outputs the final target visual feature map I g `; The SFT is used to perform convolution processing on the common semantic features to obtain the scaling feature matrix T and the translation feature matrix S, and the intermediate result obtained by performing dot multiplication of the splicing feature and T is added to the corresponding elements of S to obtain the output target visual features; The common semantic features are g Feature extraction is performed to obtain the splicing feature. a With S g Splicing obtained; The secondary generation network includes a mutually cascaded hole convolution multi-scale feature extraction module AMS and a multi-channel attention MCA; the AMS includes a combination of N hole convolutions with different expansion rates, which are used to extract the feature according to I g ` and S g Generate N corresponding feature maps with different receptive fields; the MCA is used to process the multi-scale feature map H' obtained by splicing the N feature maps with different receptive fields to obtain the target perspective generated image I g ``; S3, with I g ``The goal is to train the generative network with the highest similarity to A; S4, inputting the original perspective image to be converted and the semantic segmentation map of the target perspective image into the trained generation network to obtain the converted target perspective generated image.

2. The method according to claim 1, wherein Before step S1, the method further includes: The multi-view image pairs are pre-processed to adjust their resolution and aspect ratio.

3. The method according to claim 1 or 2, wherein: The MCA is used to perform a two-way convolution process on H' to map the multi-scale features of H' to an intermediate generated image of M dimensions and an attention weight of M dimensions respectively; And perform one-to-one multiplication of the attention weights of the M dimensions with the intermediate generated images of the M dimensions, and then perform addition operation on the corresponding elements to obtain I g ``.

4. The method according to claim 1, wherein The loss function used in the training is: the cGAN =l cGAN (THE a ,THE' g )+λl cGAN (THE a ,THE" g ); l L1 =λ1`l L1 (I g ,I' g )+λ2`l L1 (I g ,I" g )+λ3`l L1 (S g ,S' g )+λ4`l L1 (S g ,S" g ); Among them: λ1, λ2, λ3, λ, λ1`, λ2`, λ3`, λ4` are penalty factors; m is the total number of rows of the generated image, n is the total number of columns of the generated image, i is the current row number, j is the current column number, Gray (i,j) Represents the generated image I" g The grayscale value of row i and column j in S' g and S" g I' g and I" g Semantic segmentation map.

5. A device for generating a variable-viewing angle image based on a generative adversarial network, characterized in that: include: The first processing module is used to process the original view image I a Perform semantic segmentation on A in the multi-view image pair of the target perspective real image A to obtain the corresponding semantic segmentation map S g ; The second processing module is used to construct a generation network; the generation network includes a cascade of: The first-level generation network includes multiple cascaded spatial feature residual modules (RSBs); each RSB includes multiple cascaded combinations, and each combination includes interconnected spatial feature transformation modules (SFTs) and convolutional layers; The first RSB is used to extract the target visual features based on the input splicing features and public semantic features. The other RSBs are used to take the target visual features output by the previous RSB as the input features H and extract the target visual features based on H and the public semantic features. The last RSB outputs the final target visual feature map I g `; The SFT is used to perform convolution processing on the common semantic features to obtain the scaling feature matrix T and the translation feature matrix S, and the intermediate result obtained by performing dot multiplication of the splicing feature and T is added to the corresponding elements of S to obtain the output target visual features; The common semantic features are g Feature extraction is performed to obtain the splicing feature. a With S g Splicing obtained; The secondary generation network includes a mutually cascaded hole convolution multi-scale feature extraction module AMS and a multi-channel attention MCA; the AMS includes a combination of N hole convolutions with different expansion rates, which are used to extract the feature according to I g ` and S g Generate N corresponding feature maps with different receptive fields; the MCA is used to process the multi-scale feature map H' obtained by splicing the N feature maps with different receptive fields to obtain a generated image I under the target perspective g ``; The third processing module is used to g ``The goal is to train the generative network with the highest similarity to A; The fourth processing module is used to input the original perspective image to be converted and the semantic segmentation map of the target perspective image into the trained generation network to obtain the generated image under the converted target perspective.

6. A system for generating variable-view images based on a generative adversarial network, characterized in that: include: Computer-readable storage medium and processor; The computer-readable storage medium is used to store executable instructions; The processor is configured to read the executable instructions stored in the computer-readable storage medium and execute the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to execute the method according to any one of claims 1 to 4.

Images