Image semantic segmentation method and device based on generative adversarial network

Abstract

The invention relates to the technical field of image processing, and discloses an image semantic segmentation method and device based on a generative adversarial network, and the method comprises the steps: obtaining image data; calibrating the image data to obtain a calibrated image; constructing a generative adversarial network model; the generative adversarial network model comprises a generator model and a discriminator model; generating a counterfeit image based on the generator model and the image data; identifying the calibration image and the forged image by using a discriminator model to generate an identification result; and training the generator model and the discriminator model according to a predefined loss function until convergence. According to the method, the generator model and the discriminator model in the generative adversarial network model are trained, so that the generator model and the discriminator model in the generative adversarial network model work together, the generator model generates a false sample to cheat the discriminator, and the discriminator distinguishes a real sample from the false sample, so that the optimal threshold is determined; and the segmentation is more accurate.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image semantic segmentation method and device based on a generative adversarial network. Background Art

[0002] With the development of AI technology, products based on artificial intelligence generated content (AIGC) have made significant progress and attracted widespread attention around the world. More and more users have shown great interest in the development of computer vision in the field of AIGC.

[0003] At present, image segmentation is often performed based on threshold segmentation algorithms. The specific steps are as follows: the grayscale threshold is obtained by calculating the grayscale and characteristics of the image. Then, the grayscale characteristics of various pixels obtained are compared with the given threshold, and finally the comparison results are classified. However, the existing threshold segmentation algorithm ignores the spatial information in the image and only processes the grayscale of the pixels in the image, resulting in more noise, which leads to inaccurate image segmentation. Summary of the invention

[0004] In view of this, the present invention provides an image semantic segmentation method and device based on a generative adversarial network to solve the technical problem that a threshold-based segmentation algorithm ignores spatial information in an image and only performs grayscale processing on pixels in the image, resulting in more noise and thus inaccurate image segmentation.

[0005] In a first aspect, the present invention provides an image semantic segmentation method based on a generative adversarial network, the method comprising: acquiring image data; calibrating the image data to obtain a calibrated image; constructing a generative adversarial network model; wherein the generative adversarial network model comprises: a generator model and a discriminator model; generating a forged image based on the generator model and the image data; identifying the calibrated image and the forged image using the discriminator model to generate a recognition result; and training the generator model and the discriminator model according to a predefined loss function until convergence.

[0006] In an optional embodiment, the predefined loss function includes: a first loss function for training a generator model and a second loss function for training a discriminator model; wherein, according to the predefined loss function, the generator model and the discriminator model are trained until convergence, including: alternately training the generator model and the discriminator model until both the first loss function and the second loss function converge; wherein,

[0007] Loss = E x,y [logD(x,y)]+E x,G(x)[log(1-D(x,G(x)))]; x is the image data, z is the input noise vector of G, E represents the expected size, G is the generator model, y is the forged image, Loss is the first loss function, and D is the discriminator model;

[0008] Loss D =-(log(D true ))+log(1-D fake )); Loss D is the second loss function, D true To calibrate the image, D fake Fake images identified by the discriminator model.

[0009] In an optional embodiment, the generator model includes: a backbone network, a feature pyramid network, a Hypercolumn module and a fusion module, the backbone network includes at least five first convolution modules and a global average pooling layer, the feature pyramid network includes at least four second convolution modules, and the Hypercolumn module includes at least five third convolution modules; wherein, based on the generator model and the image data, generating a forged image includes: using the first convolution module of the i-th layer to extract features from the output of the i-1-th layer, and inputting the extracted features to the i+1-th layer; wherein the input of the first layer is the image data, and the input of the global average pooling layer is the first convolution module of the last layer. The output of the convolution module; input the features of the first convolution module of the g-th layer to the second convolution module of the g-th layer; where g is not equal to 1; fuse the features of two adjacent second convolution modules to obtain a first target feature; input the first target feature of the second convolution module of the g-th layer to the third convolution module of the g-th layer; input the features of the global average pooling layer to the third convolution module of the top layer; determine the loss function of the third convolution module of each layer; use the fusion module to fuse the features of the global average pooling layer, each target feature and the loss function corresponding to the target feature to obtain a second target feature; generate a forged image based on the second target feature and the loss function corresponding to the second target feature.

[0010] In an optional embodiment, the features of two adjacent second convolution modules are fused to obtain the first target feature, including: fusing the feature of the second convolution module of the g-th layer with the feature of the second convolution module of the g-1th layer to obtain the corresponding first target feature.

[0011] In an optional embodiment, a discriminator model is used to identify the calibrated image and the forged image to generate a recognition result, including: fusing the features of the calibrated image and the forged image to obtain a vector matrix; and processing the vector matrix through a matrix convolution module to obtain a recognition result.

[0012] In an optional embodiment, the above method also includes: obtaining a test data set; wherein the test data set includes multiple historical images; calibrating the historical images to obtain calibrated historical images; inputting the test data set into a generative adversarial network model to obtain test results; detecting the similarity between the test results and the calibrated historical images; if the similarity meets a preset similarity, determining that the generative adversarial network model is qualified; if the similarity does not meet the preset similarity, determining that the generative adversarial network model is unqualified.

[0013] In an optional embodiment, the weight of the loss function of the third convolution module of the g-th layer is 0.1, and the weight of the loss function of the third convolution module of the top layer is 0.01.

[0014] In a second aspect, the present invention provides an image semantic segmentation device based on a generative adversarial network, the device comprising: an acquisition module for acquiring image data; a calibration module for calibrating image data to obtain a calibrated image; a construction module for constructing a generative adversarial network model; wherein the generative adversarial network model comprises: a generator model and a discriminator model; a first generation module for generating a forged image based on the generator model and image data; a second generation module for identifying the calibrated image and the forged image using the discriminator model to generate an identification result; a training module for training the generator model and the discriminator model according to a predefined loss function until convergence.

[0015] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the image semantic segmentation method based on a generative adversarial network according to the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0016] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the image semantic segmentation method based on a generative adversarial network according to the first aspect or any corresponding embodiment thereof.

[0017] The present invention provides an image semantic segmentation method and device based on a generative adversarial network, the method comprising: acquiring image data; calibrating the image data to obtain a calibrated image; constructing a generative adversarial network model; wherein the generative adversarial network model comprises: a generator model and a discriminator model; generating a forged image based on the generator model and the image data; identifying the calibrated image and the forged image using the discriminator model to generate a recognition result; and training the generator model and the discriminator model according to a predefined loss function until convergence.

[0018] The embodiment of the present invention trains the generator model and the discriminator model in the generative adversarial network model, so that the image data is input into the generative adversarial network model, and the generator model and the discriminator model in the generative adversarial network model work together. The generator model generates false samples to deceive the discriminator, and the discriminator distinguishes between real samples and false samples, thereby determining the optimal threshold and making the segmentation more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 is a flowchart of an image semantic segmentation method based on a generative adversarial network according to an embodiment of the present invention;

[0021] Figure 2 is a schematic diagram of an image semantic segmentation method based on a generative adversarial network according to an embodiment of the present invention;

[0022] Figure 3 is a flowchart of another image semantic segmentation method based on a generative adversarial network according to an embodiment of the present invention;

[0023] Figure 4 is a schematic diagram of a generator corresponding to the image segmentation method based on a generative adversarial network according to an embodiment of the present invention;

[0024] Figure 5 is a schematic diagram of a discriminator corresponding to an image segmentation method based on a generative adversarial network according to an embodiment of the present invention;

[0025] Figure 6 is a structural block diagram of an image semantic segmentation device based on a generative adversarial network according to an embodiment of the present invention;

[0026] Figure 7 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0028] The image semantic segmentation method based on generative adversarial network provided by the embodiment of the present invention trains the generator model and the discriminator model in the generative adversarial network model, so that the image data is input into the generative adversarial network model, so that the generator model and the discriminator model in the generative adversarial network model work together, the generator model generates false samples to deceive the discriminator, and the discriminator distinguishes between real samples and false samples, thereby determining the optimal threshold value, making the segmentation more accurate.

[0029] According to an embodiment of the present invention, an embodiment of an image semantic segmentation method based on a generative adversarial network is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0030] In this embodiment, a method for image semantic segmentation based on a generative adversarial network is provided, which can be used in computer devices such as computers, servers, etc. Figure 1 is a flow chart of an image semantic segmentation method based on a generative adversarial network according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0031] Step S101, acquiring image data.

[0032] The image data can be used to represent the image data that needs to be segmented. The image data can be a real picture, etc., which is not specifically limited here. Specifically, the image data can be obtained by shooting with a shooting device, or can be obtained by other means, which is not specifically limited here.

[0033] Step S102, calibrating the image data to obtain a calibrated image.

[0034] Select the size of the calibration plate and the distribution of marker points to ensure that it covers the required image area. Place the calibration plate: Place the calibration plate in the scene to be calibrated and ensure that it can be clearly captured by the camera. Collect calibration images: Use the camera to take multiple images of the calibration plate at different angles and positions to ensure that various possible image distortions can be captured. Extract marker point coordinates: Manually or automatically extract the coordinates of the marker points of the calibration plate from the calibration image. Calculate camera parameters: Based on the extracted marker point coordinates and the camera intrinsic parameters, calculate the camera's external parameters (rotation matrix and translation vector) and distortion coefficients. Verify the calibration results: Verify the accuracy of the calibration results by comparing the calibration results with the known calibration plate parameters. Optimize camera parameters: Based on the calibration results, adjust the camera's intrinsic parameters and distortion coefficients to improve the accuracy and reliability of image segmentation.

[0035] Step S103, constructing a generative adversarial network model; wherein the generative adversarial network model includes: a generator model and a discriminator model.

[0036] Generative Adversarial Network (GAN) is a deep learning model that learns by training two neural networks - the generator and the discriminator. The task of the generator is to generate data samples that are as realistic as possible from the latent space, while the task of the discriminator is to distinguish whether the input data is a real sample or a fake sample generated by the generator. During the training process, the generator and the discriminator are trained adversarially and their parameters are continuously optimized to reach a state of equilibrium. Specifically, the generator tries to maximize the probability that the discriminator will judge its generated samples as real samples, while the discriminator tries to maximize the probability of correctly distinguishing between real samples and generated samples.

[0037] Step S104: Generate a forged image based on the generator model and the image data.

[0038] The forged image can be used to represent a fake sample image generated by the generator model based on the image data. Specifically, when the image data is used as the input of the generator model, the forged image can be used as the output of the generator model. The specific generation method of the forged image is described in detail below.

[0039] Step S105: using the discriminator model to identify the calibrated image and the forged image to generate a recognition result.

[0040] The recognition result can be used to characterize the similarity between the calibration image and the forged image. The recognition result can be that the calibration image and the forged image are similar, or that the calibration image and the forged image are not similar. Specifically, the calibration image and the forged image are used as inputs of the discriminator model, and the recognition result is used as the output of the discriminator model. The specific method for generating the recognition result is described in detail below.

[0041] Step S106, training the generator model and the discriminator model according to a predefined loss function until convergence.

[0042] In the training process of Generative Adversarial Network (GAN), it is necessary to continuously optimize the parameters of the generator model and the discriminator model to minimize the predefined loss function until the model converges. The specific training process is described in detail below.

[0043] Combination Figure 2 As shown in the figure, the generator model is a generator model based on the backbone network structure, which can receive image data and generate forged images. The discriminator model can receive forged images and calibration images, and then detect the forged images and calibration images. If the forged images and calibration images are not similar (i.e. Figure 2 Negative in the generator model, the forged image is regenerated, and the regenerated forged image and the calibration image are detected by the discriminator model until the forged image and the calibration image are similar (that is, Figure 2 until Positive in .

[0044] The image semantic segmentation method based on generative adversarial network provided in this embodiment trains the generator model and the discriminator model in the generative adversarial network model, so that the image data is input into the generative adversarial network model, and the generator model and the discriminator model in the generative adversarial network model work together. The generator model generates false samples to deceive the discriminator, and the discriminator distinguishes between real samples and false samples, thereby determining the optimal threshold to make the segmentation more accurate.

[0045] In an optional embodiment, the predefined loss function includes: a first loss function for training the generator model and a second loss function for training the discriminator model; wherein the above step S106 includes:

[0046] The generator model and the discriminator model are trained alternately until the first loss function and the second loss function converge; wherein,

[0047] Loss = E x,y [logD(x,y)]+E x,G(x)[log(1-D(x,G(x)))]; x is the image data, z is the input noise vector of G, E represents the expected size, G is the generator model, y is the forged image, Loss is the first loss function, and D is the discriminator model;

[0048] Loss D =-(log(D true ))+log(1-D fake )); Loss D is the second loss function, D true To calibrate the image, D fake Fake images identified by the discriminator model.

[0049] Specifically, the training process is as follows: Initialize the parameters of the generator model and the discriminator model. Iterate and update the parameters of the generator model and the discriminator model until the convergence conditions are met. In each iteration, we perform the following steps: Generate a batch of fake samples using the generator model under the current parameters. Use the discriminator model to discriminate between real samples and generated fake samples, and calculate the loss function of the discriminator. Use an optimization algorithm (such as gradient descent) to update the parameters of the discriminator model to minimize the loss function. Use the discriminator model under the current parameters to backpropagate the generator model and calculate the loss function of the generator. Use the optimization algorithm to update the parameters of the generator model to minimize the loss function. Check whether the convergence conditions are met (such as the loss function reaches the preset threshold, the number of iterations reaches the preset value, etc.). If the conditions are met, the training ends.

[0050] Specifically, the parameters of the discriminator model are updated, including: θ=θ-η·Δf(θ0); wherein θ0 is the independent variable parameter, η is the learning factor, i.e., the step length, and θ is the updated θ0.

[0051] like Figure 3 As shown, in an optional embodiment, the generator model includes: a backbone network, a feature pyramid network, a Hypercolumn module and a fusion module, the backbone network includes at least five first convolution modules and a global average pooling layer, the feature pyramid network includes at least four second convolution modules, and the Hypercolumn module includes at least five third convolution modules; wherein the above step S104 includes:

[0052] Step S1041, use the first convolution module of the i-th layer to extract features from the output of the i-1-th layer, and input the extracted features to the i+1-th layer; wherein the input of the first layer is image data, and the input of the global average pooling layer is the output of the first convolution module of the last layer.

[0053] Combination Figure 4As shown, the backbone network may be a ResNet network, wherein the ResNet network includes five first convolution modules, and the five first convolution modules are respectively Figure 3 In C1-C5, the global average pooling layer is Figure 4 Fullyconnected+loss in . Among them, C1 is the first layer, whose input is image data, C2 is the second layer, whose input is the output of C1, C3 is the third layer, whose input is the output of C2, C4 is the fourth layer, whose input is the output of C3, C5 is the fifth layer, whose input is the output of C5, and the input of the global average pooling layer is the output of C5.

[0054] Step S1042, inputting the features of the first convolution module of the g-th layer into the second convolution module of the g-th layer; wherein g is not equal to 1.

[0055] Combination Figure 4 As shown in the figure, the Feature Pyramid Networks (FPN) includes at least four second convolution modules, namely L2-L5. Among them, L2 and C2, L3 and C3, L4 and C4, L5 and C5 are horizontally connected through 64 1x1 convolutions (i.e. 8×8 convolution matrices) with a stride of 1. Specifically, the input of L2 is the output of C2, the input of L3 is the output of C3, the input of L4 is the output of C4, and the input of L5 is the output of C5.

[0056] Step S1043, fusing the features of two adjacent second convolution modules to obtain the first target feature.

[0057] Specifically, the above step S1043 includes: fusing the features of the second convolution module of the g-th layer with the features of the second convolution module of the g-1th layer to obtain the corresponding first target features.

[0058] Combination Figure 4 As shown, the features of L5 and L4 are fused, the features of L4 and L3 are fused, and the features of L3 and L2 are fused to obtain multiple first target features. Among them, the feature of S5 can be transmitted by L5 through a 3x3 convolution with a stride of 1 and a ReLU activation function.

[0059] Step S1044, inputting the first target feature of the second convolution module of the g-th layer into the third convolution module of the g-th layer.

[0060] Combination Figure 4 As shown, the Hypercolumn module includes at least five third convolution modules, and the five third convolution modules are S1-S5, wherein the output of L2 is the input of S2, the output of L3 is the input of S3, and the output of L4 is the input of S4.

[0061] It should be noted that, combined with Figure 4 As shown, after obtaining the first target feature, the size of the image can be doubled by using a 2x up operation.

[0062] Step S1045, input the features of the global average pooling layer to the third convolution module of the top layer.

[0063] Combination Figure 4 As shown in Figure 1, the features of the global average pooling layer are input to the third convolutional module at the top layer. That is, the output of the features of the global average pooling layer is used as the input of S1.

[0064] Step S1046, determining the loss function of the third convolution module of each layer.

[0065] The loss function of the third convolution module of each layer can be determined by model structure guidance. Specifically, the loss function is determined according to the position and function of the third convolution module in the model. The loss function of the third convolution module of each layer can be determined by data guidance. Specifically, the loss function is determined according to the characteristics of the data processed by the third convolution module, etc., which is not specifically limited here.

[0066] Step S1047, using a fusion module to fuse the features of the global average pooling layer, each target feature, and the loss function corresponding to the target feature to obtain a second target feature.

[0067] Step S1048: Generate a forged image based on the second target feature and the loss function corresponding to the second target feature.

[0068] In the Hypercolumn module, S1 is the feature information from the encoder end, which has strong semantic information. S2, S3, S4, and S5 are the feature information from the decoder end, which have undergone the first multi-scale feature fusion operation. Finally, the five modules S1, S2, S3, S4, and S5 use bilinear upsampling, normalize to the same size, and perform a concate operation (the concat operation is usually used to connect two or more tensors together in a certain dimension to generate a larger tensor). Among them, the auxiliary loss weights at S2, S3, S4, and S5 are kept at 0.1, and the auxiliary loss weights at the encoder end are kept at 0.01.

[0069] Preferably, combined Figure 4 As shown, the generator model also includes: 4 groups of scSE modules. The scSE modules are introduced in the decoder stage to highlight important feature information before multi-scale fusion, which corresponds to the previous decoder.

[0070] The image semantic segmentation method based on generative adversarial network provided in this embodiment introduces four groups of scSE modules in the Hypercolumn module of the decoder stage of the ResNet-FPN model to fully extract features, and then perform multi-scale feature fusion on the output features, thereby not only improving the accuracy of semantic segmentation, but also improving the speed of semantic segmentation.

[0071] In an optional implementation, the above step S105 includes:

[0072] Step a1: perform feature fusion on the calibration image and the forged image to obtain a vector matrix.

[0073] The calibrated image may be a real image, and the forged image may be an image generated by the above generator according to the image data, wherein the image data may include multiple pixels, and the pixels may be feature fused to obtain a vector matrix.

[0074] Step a2, processing the vector matrix through the matrix convolution module to obtain the recognition result.

[0075] Combination Figure 5 As shown in the figure, after the discriminator model fuses the above-mentioned calibrated image and the forged image, the two high-dimensional inputs (i.e., the calibrated image and the forged image) are stacked separately to form each high-dimensional result feature of one of the six high-dimensional channels, and then convolution is used between the two high-dimensional inputs to continuously stack and extract each high-dimensional result feature. Finally, vvsigmoid can be used to perform high-dimensional binary division on the extracted result features.

[0076] Combination Figure 5 As shown in the figure, the calibration image and the forged image are fused into a 256×256 vector matrix, and then the vector matrix is ​​feature extracted through a 4×4 matrix convolution module, so that the vector matrix changes from 256×256 to 128×128, from 128×128 to 64×64, and finally to a 1×1 vector matrix. Among them, the 1×1 vector matrix is ​​the recognition result.

[0077] In an optional embodiment, the above method further includes:

[0078] Step b1, obtaining a test data set; wherein the test data set includes a plurality of historical images;

[0079] Step b2, calibrating the historical image to obtain a calibrated historical image;

[0080] Step b3, input the test data set into the generative adversarial network model to obtain the test results;

[0081] Step b4, detecting the similarity between the test result and the calibrated historical image;

[0082] Step b5: if the similarity meets the preset similarity, the generative adversarial network model is determined to be qualified;

[0083] Step b6: If the similarity does not meet the preset similarity, the generative adversarial network model is determined to be unqualified.

[0084] After constructing the generative adversarial network model, the generative adversarial network model can be tested through historical images to determine the accuracy of the generative adversarial network model. Specifically, a test data set is obtained. Among them, the test data set can include multiple historical images, and the test data set can be directly obtained from the database or photographed from a real scene, which is not specifically limited here. After obtaining the test data set, the test data set can be input into the generative adversarial network model to obtain the test results, and then the similarity between the test results and the calibrated historical images is detected. Among them, if the similarity meets the preset similarity, the generative adversarial network model is determined to be qualified; if the similarity does not meet the preset similarity, the generative adversarial network model is determined to be unqualified. Among them, the preset similarity can be M or N, which is not specifically limited here.

[0085] The image semantic segmentation method based on the generative adversarial network provided in this embodiment can ensure the accuracy of semantic segmentation performed by the generative adversarial network model by testing the accuracy of the generative adversarial network model through a test data set.

[0086] In this embodiment, a device for image semantic segmentation based on a generative adversarial network is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0087] This embodiment provides an image semantic segmentation device based on a generative adversarial network. Figure 6 As shown, it includes: an acquisition module 601, used to acquire image data; a calibration module 602, used to calibrate the image data to obtain a calibrated image; a construction module 603, used to construct a generative adversarial network model; wherein the generative adversarial network model includes: a generator model and a discriminator model; a first generation module 604, used to generate a forged image based on the generator model and the image data; a second generation module 605, used to use the discriminator model to identify the calibrated image and the forged image to generate a recognition result; a training module 606, used to train the generator model and the discriminator model according to a predefined loss function until convergence.

[0088] In some optional embodiments, the predefined loss function includes: a first loss function for training the generator model and a second loss function for training the discriminator model; wherein the training module 606 includes: a training unit for alternately training the generator model and the discriminator model until the first loss function and the second loss function converge; wherein,

[0089] Loss = E x,y [logD(x,y)]+E x,G(x) [log(1-D(x,G(x)))]; x is the image data, z is the input noise vector of G, E represents the expected size, G is the generator model, y is the forged image, Loss is the first loss function, and D is the discriminator model;

[0090] Loss D =-(log(D true ))+log(1-D fake )); Loss D is the second loss function, D true To calibrate the image, D fake Fake images identified by the discriminator model.

[0091] In some optional embodiments, the generator model includes: a ResNet network, an FPN network, a Hypercolumn module and a fusion module, the ResNet network includes at least five first convolution modules and a global average pooling layer, the FPN network includes at least four second convolution modules, and the Hypercolumn module includes at least five third convolution modules; wherein the first generation module 504 includes: a first feature extraction unit, for extracting features from the output of the i-1th layer using the first convolution module of the i-th layer, and inputting the extracted features to the i+1th layer; wherein the input of the first layer is image data, and the input of the global average pooling layer is the output of the first convolution module of the last layer;

[0092] A first input unit, used to input the features of the first convolution module of the g-th layer into the second convolution module of the g-th layer; wherein g is not equal to 1;

[0093] A first fusion unit, used for fusing the features of two adjacent second convolution modules to obtain a first target feature;

[0094] A second input unit is used to input the first target feature of the second convolution module of the g-th layer into the third convolution module of the g-th layer;

[0095] A third input unit is used to input the features of the global average pooling layer into the third convolution module of the top layer;

[0096] A determination unit, used to determine the loss function of the third convolution module of each layer;

[0097] A second fusion unit is used to fuse the features of the global average pooling layer, each target feature and the loss function corresponding to the target feature using a fusion module to obtain a second target feature;

[0098] The first generating unit is used to generate a forged image based on the second target feature and the loss function corresponding to the second target feature.

[0099] In some optional embodiments, the first fusion unit is used to fuse the features of the second convolution module of the g-th layer with the features of the second convolution module of the g-1th layer to obtain the corresponding first target features.

[0100] In some optional embodiments, the second generation module 505 includes: a third fusion unit, used to fuse the features of the calibrated image and the forged image to obtain a vector matrix; and a processing unit, used to process the vector matrix through a matrix convolution module to obtain a recognition result.

[0101] In some optional embodiments, the above-mentioned device also includes: a test data set acquisition module, used to acquire a test data set; wherein the test data set includes multiple historical images; a historical image calibration module, used to calibrate the historical images to obtain calibrated historical images; a test data set input module, used to input the test data set into the generative adversarial network model to obtain test results; a detection module, used to detect the similarity between the test results and the calibrated historical images; a first judgment module, used to judge that the generative adversarial network model is qualified if the similarity meets the preset similarity; and a second judgment module, used to judge that the generative adversarial network model is unqualified if the similarity does not meet the preset similarity.

[0102] In some optional embodiments, the weight of the loss function of the third convolution module of the g-th layer is 0.1, and the weight of the loss function of the third convolution module of the top layer is 0.01.

[0103] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0104] The image semantic segmentation device based on the generative adversarial network in this embodiment is presented in the form of a functional unit, where the functional unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0105] The embodiment of the present invention also provides a computer device having the above Figure 6 The image semantic segmentation device based on generative adversarial network is shown.

[0106] See also Figure 7 , Figure 7 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 7 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 10 is taken as an example.

[0107] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0108] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0109] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0110] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0111] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0112] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0113] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for image semantic segmentation based on generative adversarial network, characterized in that: include: Get image data; Calibrate the image data to obtain a calibrated image; Constructing a generative adversarial network model; wherein the generative adversarial network model includes: a generator model and a discriminator model; generating a forged image based on the generator model and the image data; Using the discriminator model to identify the calibration image and the forged image, and generate a recognition result; The generator model and the discriminator model are trained according to a predefined loss function until convergence.

2. The image semantic segmentation method based on generative adversarial network according to claim 1 is characterized in that: The predefined loss function includes: a first loss function for training the generator model and a second loss function for training the discriminator model; wherein, according to the predefined loss function, the generator model and the discriminator model are trained until convergence, including: The generator model and the discriminator model are alternately trained until both the first loss function and the second loss function converge; wherein, Loss = E x,y [logD(x,y)]+E x,G(x) [log(1-D(x,G(x)))]; x is the image data, z is the input noise vector of G, E represents the expected size, G is the generator model, y is the forged image, Loss is the first loss function, and D is the discriminator model; Loss D =-(log(D true ))+log(1-D fake )); Loss D is the second loss function, D true To calibrate the image, D fake Fake images identified by the discriminator model.

3. The image semantic segmentation method based on generative adversarial network according to claim 1, characterized in that: The generator model includes: a backbone network, a feature pyramid network, a Hypercolumn module and a fusion module, the backbone network includes at least five first convolution modules and a global average pooling layer, the feature pyramid network includes at least four second convolution modules, and the Hypercolumn module includes at least five third convolution modules.

4. The image semantic segmentation method based on generative adversarial network according to claim 3 is characterized in that: Based on the generator model and the image data, generating a forged image includes: Using the first convolution module of the i-th layer to extract features from the output of the i-1-th layer, and inputting the extracted features to the i+1-th layer; wherein the input of the first layer is the image data, and the input of the global average pooling layer is the output of the first convolution module of the last layer; Input the features of the first convolution module of the g-th layer to the second convolution module of the g-th layer; wherein g is not equal to 1; The features of two adjacent second convolution modules are merged to obtain the first target feature; Input the first target feature of the second convolution module of the g-th layer to the third convolution module of the g-th layer; Input the features of the global average pooling layer to the third convolutional module of the top layer; Determine the loss function of the third convolution module of each layer; Using the fusion module to fuse the features of the global average pooling layer, each of the target features, and the loss function corresponding to the target features, to obtain a second target feature; A forged image is generated based on the second target feature and a loss function corresponding to the second target feature.

5. The image semantic segmentation method based on generative adversarial network according to claim 4 is characterized in that: The features of two adjacent second convolution modules are fused to obtain the first target features, including: The features of the second convolution module of the g-th layer are fused with the features of the second convolution module of the g-1th layer to obtain the corresponding first target features.

6. The image semantic segmentation method based on generative adversarial network according to claim 1, characterized in that: Using the discriminator model to identify the calibration image and the forged image to generate a recognition result includes: Performing feature fusion on the calibration image and the forged image to obtain a vector matrix; The vector matrix is ​​processed by a matrix convolution module to obtain the recognition result.

7. The image semantic segmentation method based on generative adversarial network according to claim 1, characterized in that: Also includes: Acquire a test data set; wherein the test data set includes a plurality of historical images; Calibrate the historical image to obtain a calibrated historical image; Inputting the test data set into a generative adversarial network model to obtain a test result; Detecting the similarity between the test result and the calibrated historical image; If the similarity meets the preset similarity, the generative adversarial network model is determined to be qualified; If the similarity does not meet the preset similarity, the generative adversarial network model is determined to be unqualified.

8. The image semantic segmentation method based on generative adversarial network according to claim 4, characterized in that: The weight of the loss function of the third convolution module of the g-th layer is 0.1, and the weight of the loss function of the third convolution module of the top layer is 0.

01.

9. An image semantic segmentation device based on a generative adversarial network, characterized in that: The device comprises: An acquisition module, used for acquiring image data; A calibration module, used for calibrating the image data to obtain a calibrated image; A construction module is used to construct a generative adversarial network model; wherein the generative adversarial network model includes: a generator model and a discriminator model; A first generating module, configured to generate a forged image based on the generator model and the image data; A second generation module is used to identify the calibration image and the forged image using the discriminator model to generate a recognition result; The training module is used to train the generator model and the discriminator model according to a predefined loss function until convergence.

10. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the image semantic segmentation method based on a generative adversarial network according to any one of claims 1 to 8 by executing the computer instructions.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the image semantic segmentation method based on a generative adversarial network according to any one of claims 1 to 8.

Images