Training method, image generation method, device, electronic equipment and storage medium

Abstract

The present disclosure relates to a method for training an image generation model, comprising: generating a first to-be-identified image according to a sample image by using a first generation network; identifying the first to-be-identified image to output a first feature by using a first discrimination network, and identifying a reference image to output a second feature; determining a first similarity between the first to-be-identified image and the reference image according to the first feature, the second feature and a similarity function; and updating network parameters of the first generation network and the first discrimination network according to the first similarity. By comparing the first feature and the second feature, the first similarity between the first to-be-identified image and the reference image can be determined more accurately, and the accuracy of the first similarity is improved.

Description

Technical Field

[0001] This disclosure relates to the field of image processing, and more particularly to a training method for an image generation model, an image generation method, an apparatus, an electronic device, and a storage medium. Background Technology

[0002] With the development of image processing technology, it has been applied in many fields, enabling the production of higher-quality images. Many electronic devices have imaging capabilities and image acquisition modules. After acquiring images, these modules can be processed, such as optimized, to generate higher-quality images.

[0003] Machine vision technology, such as network models, can be used to process images acquired by image acquisition modules, thereby generating higher-quality images that meet the requirements. Before using the network model to process the images acquired by the image acquisition module, training samples need to be obtained first. Then, the network model is trained using the training samples to obtain a trained network model. Summary of the Invention

[0004] This disclosure provides a training method for an image generation model, an image generation method, an apparatus, an electronic device, and a storage medium.

[0005] A first aspect of this disclosure provides a training method for an image generation model, comprising: generating a first image to be identified using a first generation network based on a sample image; identifying the first image to be identified using a first discriminator network and outputting a first feature, and identifying a reference image and outputting a second feature; determining a first similarity between the first image to be identified and the reference image based on the first feature, the second feature, and a similarity function; and updating the network parameters of the first generation network and the first discriminator network based on the first similarity.

[0006] In one embodiment, determining the first similarity between the image to be identified and the reference image based on the first feature, the second feature, and the similarity function includes: performing spectral normalization on the first feature to obtain a first result, performing spectral normalization on the second feature to obtain a second result, and determining the first similarity based on the first result, the second result, and the similarity function.

[0007] In one embodiment, the method includes: determining a first reference factor of the first image to be identified based on the pixel values ​​of pixels in the first image to be identified; determining a second reference factor of the reference image based on the pixel values ​​of pixels in the reference image; the first reference factor and the second reference factor are used to represent at least one type of image information in the same dimension; determining a first similarity between the first image to be identified and the reference image based on the first feature, the second feature and the similarity function further includes: determining the first similarity based on the first reference factor, the second reference factor, the first feature, the second feature and the similarity function.

[0008] In one embodiment, the first reference factor and the second reference factor include at least one of the following: brightness, contrast, image structure, and / or signal-to-noise ratio.

[0009] In one embodiment, generating a first image to be identified using a first generative network based on a sample image includes: generating the first image to be identified through a spatial attention mechanism.

[0010] In one embodiment, the method includes: generating a second image to be identified based on a first image to be identified using a second generative network; identifying the second image to be identified and outputting a third feature using a second discriminative network, and identifying a sample image and outputting a fourth feature; determining a second similarity between the second image to be identified and the sample image based on the third feature, the fourth feature, and the similarity function; and updating the network parameters of the first generative network, the second generative network, the first discriminative network, and the second discriminative network based on the first similarity and the second similarity.

[0011] In one embodiment, determining the second similarity between the second image to be identified and the sample image based on the third feature, the fourth feature, and the similarity function includes: performing spectral normalization on the third feature to obtain a third result, performing spectral normalization on the fourth feature to obtain a fourth result, and determining the second similarity based on the third result, the fourth result, and the similarity function.

[0012] In one embodiment, the method includes: determining a third reference factor of the second image to be identified based on the pixel values ​​of pixels in the second image to be identified, and determining a fourth reference factor of the sample image based on the pixel values ​​of pixels in the sample image; the third reference factor and the fourth reference factor are used to represent at least one type of image information in the same dimension; determining a second similarity between the second image to be identified and the sample image based on the third feature, the fourth feature and the similarity function includes: determining the second similarity based on the third reference factor, the fourth reference factor, the third feature, the fourth feature and the similarity function.

[0013] In one embodiment, the third reference factor and the fourth reference factor include at least one of the following: brightness, contrast, image structure, and / or signal-to-noise ratio.

[0014] In one embodiment, generating a second image to be identified based on the first image to be identified using a second generative network includes generating the second image to be identified through a spatial attention mechanism.

[0015] A second aspect of this disclosure provides an image generation method, comprising: configuring acquisition parameters; the acquisition parameters including at least: image resolution; acquiring a sample image in a target application through multiple acquisition threads according to the acquisition parameters; the sample image having depth information; optimizing the sample image, the optimization including at least: distortion correction; and generating a target image based on the optimized sample image using an image generation model trained according to any one of claims 1 to 10.

[0016] A third aspect of this disclosure provides a training apparatus for an image generation model, comprising: a generation module for generating a first image to be recognized using a first generation network based on a sample image; a recognition module for recognizing the first image to be recognized using a first discriminant network and outputting a first feature, and recognizing a reference image and outputting a second feature; a similarity determination module for determining a first similarity between the first image to be recognized and the reference image based on the first feature, the second feature, and a similarity function; and an update module for updating the network parameters of the first generation network and the first discriminant network based on the first similarity.

[0017] A fourth aspect of this disclosure provides an image generation apparatus, comprising: a configuration module for configuring acquisition parameters; the acquisition parameters including at least: image resolution; an acquisition module for acquiring a sample image in a target application through multiple acquisition threads according to the acquisition parameters; the sample image having depth information; an optimization module for optimizing the sample image, the optimization including at least: distortion correction; and a generation module for generating a target image based on the optimized sample image using an image generation model trained by any of the above embodiments.

[0018] A fifth aspect of this disclosure provides an electronic device, comprising:

[0019] A processor and a memory for storing executable instructions that can run on the processor, wherein: when the processor runs the executable instructions, the executable instructions perform the method described in any of the above embodiments.

[0020] A sixth aspect of this disclosure provides a non-transitory computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the methods described in any of the above embodiments.

[0021] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0022] In this embodiment, a first generator network generates a first image to be identified based on a sample image. Then, a first discriminator network identifies the first image to be identified and outputs a first feature, and identifies a reference image and outputs a second feature. The first similarity between the first image to be identified and the reference image is determined based on the first feature, the second feature, and a similarity function. The network parameters of the first generator network and the first discriminator network are then updated based on the first similarity.

[0023] The first discriminant network identifies the first image to be identified and outputs a first feature, and the reference image outputs a second feature. The first feature can represent the features of the first image to be identified, and the second feature can represent the features of the reference image. By comparing the first feature and the second feature, the first similarity between the first image to be identified and the reference image can be determined more accurately, thus improving the accuracy of the first similarity. Based on the first similarity, the network parameters of the first generator network and the first discriminant network can be updated more effectively, thereby improving the update efficiency of the first generator network and the first discriminant network.

[0024] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0026] Figure 1 This is a flowchart illustrating a training method for an image generation model according to an exemplary embodiment;

[0027] Figure 2 This is a schematic diagram illustrating the determination of a first similarity according to an exemplary embodiment;

[0028] Figure 3 This is a flowchart illustrating another method for training an image generation model according to an exemplary embodiment;

[0029] Figure 4 This is a schematic diagram illustrating the determination of a second similarity according to an exemplary embodiment;

[0030] Figure 5 This is an image generation method illustrated according to an exemplary embodiment;

[0031] Figure 6 This is a schematic diagram of the structure of a training device for an image generation model according to an exemplary embodiment;

[0032] Figure 7 This is a schematic diagram of the structure of an image generation apparatus according to an exemplary embodiment;

[0033] Figure 8 This is a schematic diagram illustrating another image generation method according to an exemplary embodiment;

[0034] Figure 9 This is a schematic diagram of the structure of a network model according to an exemplary embodiment;

[0035] Figure 10 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0036] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses consistent with some aspects of this disclosure as detailed in the appended claims.

[0037] refer to Figure 1 This is a flowchart illustrating a training method for an image generation model provided by this technical solution.

[0038] The method includes the following steps:

[0039] Step S100: Generate a first image to be identified using a first generator network based on the sample image.

[0040] In step S200, the first discriminant network is used to identify the first image to be identified and output the first feature, and the reference image is identified and output the second feature.

[0041] Step S300: Determine the first similarity between the first image to be identified and the reference image based on the first feature, the second feature, and the similarity function.

[0042] Step S400: Update the network parameters of the first generator network and the first discriminator network based on the first similarity.

[0043] In this embodiment, the network model includes at least a first generator network and a first discriminator network. The first generator network can be a generator, and the first discriminator network can be a discriminator. The specific structures of the first generator network and the first discriminator network are not limited, and can be networks with generation and discrimination functions, including structures such as convolutional networks and residual networks. For example, the network model can be a recurrent generative adversarial network (Cycle GAN), where the first generator network can be a generator within the recurrent generator network, and the first discriminator network can be a discriminator that judges the results generated by the first generator network.

[0044] This embodiment can update the network model, at least the first generator network and the first discriminator network in the network model. By updating the network parameters of the first generator network and the first discriminator network, the network model can be trained and optimized.

[0045] In step S100, the first generator network can generate a first image to be recognized based on the sample image. The sample image is input into the first generator network, which then generates the first image to be recognized. For example, the sample image x1 in the X domain is input into the first generator network, and the first generator network generates the first image to be recognized y' based on the sample image x1.

[0046] In one embodiment, the sample image has depth information, and the first network to be identified generated by the first generator network also has depth information, thereby obtaining a first network to be identified with depth information.

[0047] In step S200, after generating the first image to be recognized through the first generator network, the first discriminator network is used to recognize the first image to be recognized and output a first feature, and a reference image is recognized and output a second feature. The reference image is used to compare with the first image to be recognized, and may be an image y1 in the Y domain.

[0048] The first discriminant network can identify a first image to be identified and output the characteristics of the first image to be identified. Here, the output characteristics of the first image to be identified are used as the first feature. The first discriminant network can also identify a reference image and output the characteristics of the reference image. Here, the output characteristics of the reference image are used as the second feature. The first feature may include the characteristics of different regions in the first image to be identified, and the second feature may represent the characteristics of different regions in the reference image.

[0049] For example, the first feature can be a feature map of size N*N, and the second feature can also be a feature map of size N*N. For the first feature, each element in the N*N feature map represents a feature of a receptive field of a certain size in the first image to be recognized. For the second feature, each element in the N*N feature map represents a feature of a receptive field of a certain size in the reference image. The size of N can be determined according to actual needs, such as the size of the first image to be recognized and the reference image, the size of the convolution kernel, and / or the stride of the convolution.

[0050] Since the first discriminant network outputs the first feature and the second feature, it does not directly determine the similarity between the first image to be identified and the reference image, i.e., it does not directly determine whether the first image to be identified is real or fake. Using the first and second features allows for a more accurate determination of the first similarity between the first image to be identified and the reference image, thereby improving the accuracy of the first similarity score. Simultaneously, based on the first similarity score, it facilitates better updating of the network parameters of the first generator network and the first discriminant network, thus improving the update efficiency of the first generator network and the first discriminant network.

[0051] For step S300, after obtaining the first feature and the second feature, a first similarity between the first image to be identified and the reference image can be determined based on the first feature, the second feature, and the similarity function. The similarity function can be any function capable of determining the similarity between the first feature and the second feature. This step can be performed by an optimizer or by other structures.

[0052] For example, the similarity function can calculate the arithmetic mean of each element in the N*N feature map of the first feature to obtain the first arithmetic mean, and calculate the arithmetic mean of each element in the N*N feature map of the second feature to obtain the second arithmetic mean. Then, the difference between the first and second arithmetic means is determined, and a first similarity is determined based on this difference. If the difference is greater than a first threshold, the first similarity is determined to be 1, indicating that the first image to be identified is the same as the reference image. If the difference is less than the first threshold, the first similarity is determined to be 0, indicating that the first image to be identified is different from the reference image.

[0053] In another embodiment, the difference can also have a mapping relationship with the first similarity. The first similarity can be determined based on the difference. When the first similarity is greater than the first similarity threshold, it means that the first image to be identified is the same as the reference image. When the first similarity is less than the first similarity threshold, it means that the first image to be identified is different from the reference image.

[0054] For step S400, after determining the first similarity, the network parameters of the first generator network and the first discriminator network are updated based on the first similarity. The specific update process is not limited, as long as it updates the network parameters of the first generator network and the first discriminator network based on the first similarity. Updates can be performed using an update function, a loss function, an optimizer, etc.

[0055] When the first image to be identified is determined to be different from the reference image based on the first similarity, the network parameters of the first generator network are updated to improve the similarity between the first image to be identified and the reference image generated by the first generator network based on the image samples in the future. This makes the first image to be identified and the reference image more similar, and reduces the probability that the first discriminator will identify the first image to be identified and the reference image as different images.

[0056] When the first image to be identified is determined to be the same as the reference image based on the first similarity, the network parameters of the first discriminator network are updated to improve the ability of the first discriminator network to distinguish between the first image to be identified and the reference image, so that the similarity between the first image to be identified and the reference image is lower, thereby increasing the probability of distinguishing the first image to be identified and the reference image as different images, that is, increasing the probability of the first discriminator distinguishing the first image to be identified as fake.

[0057] By iteratively updating the network parameters of the first generator network and the first discriminator network, the similarity between the first image to be identified generated by the first generator network and the reference image can be increased, making the first image to be identified more like a "real image" and reducing the probability that the first discriminator will identify the first image to be identified as different from the reference image. Simultaneously, it can also increase the probability that the first discriminator network will identify the first image to be identified as different from the reference image, i.e., the probability of identifying the first image to be identified as a "fake image." This process is repeated until the probability that the first discriminator network identifies the first image to be identified as the same image as the reference image approaches 50%, at which point the first generator network can generate the required image based on the sample image.

[0058] The first and second features determined by this embodiment can more accurately determine the first similarity between the first image to be identified and the reference image, improving the accuracy of the first similarity. Therefore, the network parameters of the first generator network and the first discriminator network can be updated better based on the first similarity, thereby improving the update efficiency of the first generator network and the first discriminator network.

[0059] In another embodiment, reference Figure 2 This is a schematic diagram for determining a first similarity. Step S300, determining the first similarity between the image to be identified and the reference image based on the first feature and the second feature, includes:

[0060] Step S301: Perform spectral normalization on the first feature to obtain the first result, and perform spectral normalization on the second feature to obtain the second result.

[0061] Step S302: Determine the first similarity based on the first result, the second result, and the similarity function.

[0062] After obtaining the first and second features, the first feature is spectrally normalized to obtain the first result, and the second feature is spectrally normalized to obtain the second result. Then, the first similarity is determined based on the first result, the second result, and the similarity function. Spectral normalization of the first and second features can reduce problems such as hindered network model training or overfitting when there is a large difference between the sample image and the reference image.

[0063] The specific process of spectral normalization will not be described in detail. For example, spectral normalization can be performed on the first and second features based on the image gradient.

[0064] In one embodiment, the first feature includes N*N first elements, and the second feature includes N*N second elements.

[0065] The first result is obtained by performing spectral normalization on the first feature, and the second result is obtained by performing spectral normalization on the second feature, including:

[0066] Perform spectral normalization on the N*N first elements to obtain the first result;

[0067] Perform spectral normalization on the N*N second elements to obtain the second result.

[0068] In another embodiment, the method further includes:

[0069] A first reference factor for the first image to be identified is determined based on the pixel values ​​of the pixels in the first image to be identified, and a second reference factor for the reference image is determined based on the pixel values ​​of the pixels in the first reference image; the first reference factor and the second reference factor are used to represent at least one type of image information of the same dimension.

[0070] The first reference factor is image information of at least one dimension of the first image to be identified, and the second reference factor is image information of at least one dimension of the reference image. The first reference factor and the second reference factor are image information of the same dimension.

[0071] For example, if the first reference factor is image information of dimension A, and the second reference factor is also image information of dimension A, then the second reference factor also includes image information of dimensions A and B. The first and second reference factors identify image information of at least the same dimension, which facilitates a comparison of the similarity between the first image to be identified and the reference image based on image information of the same dimension, thereby more accurately determining the first similarity and improving the accuracy of the first similarity.

[0072] Step S300, determining the first similarity between the first image to be identified and the reference image based on the first feature and the second feature, further includes:

[0073] The first similarity is determined based on the first reference factor, the second reference factor, the first feature, and the second feature.

[0074] After determining the first reference factor and the second reference factor, the first similarity can be determined based on the first reference factor, the second reference factor, the first feature, and the second feature. By adding the first reference factor and the second reference factor on the basis of the first feature and the second feature, more image information is added to determine the first similarity, thereby determining a more accurate first similarity. After the accuracy of the first similarity is improved, the first generation network and the first discriminator network can be updated more accurately, which makes it easier to reduce the noise of the first image to be identified and generate a first image to be identified with a higher similarity to the reference image, thus improving the image quality of the generated first image to be identified.

[0075] In one embodiment, the first reference factor and the second reference factor include at least one of the following: brightness, contrast, image structure, and / or signal-to-noise ratio.

[0076] The first similarity can be determined based on the brightness similarity between the first image to be identified and the reference image, as well as the similarity between the first feature and the second feature. The greater the sum of the similarity between the first feature and the second feature and the brightness similarity, the greater the first similarity, indicating that the first image to be identified and the reference image are more similar, and the less likely the first discriminator is to distinguish the first image to be identified and the reference image as different images, that is, the less likely the first discriminator is to distinguish the first image to be identified as a "fake image".

[0077] The first similarity can also be determined based on the contrast similarity between the first image to be identified and the reference image, as well as the similarity between the first feature and the second feature. The greater the sum of the similarity between the first feature and the second feature and the contrast similarity, the greater the first similarity, indicating that the first image to be identified and the reference image are more similar, and the less likely the first discriminator is to distinguish the first image to be identified and the reference image as different images, that is, the less likely the first discriminator is to distinguish the first image to be identified as a "fake image".

[0078] The first similarity can also be determined based on the image structure similarity between the first image to be identified and the reference image, as well as the similarity between the first feature and the second feature. The greater the sum of the similarity between the first feature and the second feature and the image structure similarity, the greater the first similarity, indicating that the first image to be identified and the reference image are more similar, and the less likely the first discriminator is to distinguish the first image to be identified and the reference image as different images, that is, the less likely the first discriminator is to distinguish the first image to be identified as a "fake image".

[0079] The first similarity can also be determined based on the contrast similarity, brightness similarity, image structure similarity, and the similarity of the first feature and the second feature between the first image to be identified and the reference image. A reference factor similarity can be determined based on the contrast similarity, brightness similarity, and image structure similarity between the first image to be identified and the reference image. Then, the first similarity is determined based on the similarity of the first feature and the second feature combined with the reference factor similarity. A higher reference factor similarity indicates a greater similarity between the first image to be identified and the reference image.

[0080] The reference factor similarity between the first image to be identified and the reference image can be determined by the pixel values ​​of the first image to be identified and the reference image. Specifically, it can be determined by the Structural Similarity (SSIM) formula, which calculates the brightness similarity, contrast similarity, image structure similarity, and reference factor similarity between the first image to be identified and the reference image. The reference factor similarity is the calculated value of SSIM. The value of SSIM can be in the range of [0, 1]. The larger the value of SSIM, the greater the similarity between the first image to be identified and the reference image.

[0081] The first similarity can also be determined based on the contrast similarity, brightness similarity, image structure similarity, signal-to-noise ratio (SNR) of the first image to be identified, the SNR of the reference image, and the similarity between the first feature and the second feature. The SNR can be determined by the number of pixels and the grayscale value of the pixels in the image. The specific determination process can be based on the SNR formula for an image.

[0082] The smaller the difference between the signal-to-noise ratio (SNR) of the first image to be identified and the SNR of the reference image, the greater the similarity between the first image to be identified and the reference image.

[0083] In one embodiment, step S100, generating a first image to be identified using a first generative network based on a sample image, includes: generating the first image to be identified through a spatial attention mechanism.

[0084] When generating the first image to be identified, the spatial attention mechanism can extract the spatial correlation information of the sample image. Based on the extracted spatial correlation information of the sample image, the first image to be identified with a higher similarity to the reference image is generated. The first image to be identified has a higher first similarity value with the reference image, and the generated first image to be identified is more like a "real image". This improves the first similarity and reduces the probability that the first discriminator will identify the first image to be identified as a different image from the reference image.

[0085] In another embodiment, reference Figure 3 The diagram below illustrates a training method for another image generation model, the method comprising:

[0086] Step S10: Using the second generation network, generate a second image to be recognized based on the first image to be recognized.

[0087] Step S20: Use the second discriminant network to identify the second image to be identified and output the third feature, and identify the sample image and output the fourth feature.

[0088] Step S30: Determine the second similarity between the second image to be identified and the sample image based on the third feature, the fourth feature, and the similarity function.

[0089] Step S40: Update the network parameters of the first generator network, the second generator network, the first discriminator network, and the second discriminator network based on the first similarity and the second similarity.

[0090] In this embodiment, the network model may further include a second generator network and a second discriminator network. The second generator network may be a generator, and the second discriminator network may be a discriminator. The specific structures of the second generator network and the second discriminator network are not limited, and may be networks with generation and discrimination functions, including structures such as convolutional networks and residual networks. For example, the network model may be a recurrent generative adversarial network (Cycle GAN), the second generator network may be a generator in the recurrent generator network, and the second discriminator network may be a discriminator that judges the results generated by the second generator network.

[0091] The second generator network and the first generator network can be two different generator networks, and the second discriminator network and the first discriminator network can be two different discriminator networks.

[0092] This embodiment can update the network model, at least the second generator network and the second discriminator network in the network model. By updating the network parameters of the generator network and the second discriminator network, the network model can be trained and optimized.

[0093] For step S10, a second generating network is used to generate a second image to be recognized based on the first image to be recognized. The first image to be recognized generated by the first generating network is used as input to the second generating network, which then generates the second image to be recognized. For example, the first image to be recognized y' generated by the first generating network based on the sample image x1 in the X domain is input to the second generating network, and the second generating network outputs the second image to be recognized x'.

[0094] In step S20, the second image to be identified, x', and the sample image are used as inputs to the second discriminant network. The second discriminant network can identify the second image to be identified and output a third feature, and it can also identify the sample image and output a fourth feature. The features of the output second image to be identified are used as the third feature, and the features of the output sample image are used as the fourth feature. The third feature may include features of different regions in the second image to be identified, and the fourth feature may represent features of different regions in the sample image.

[0095] The third feature can be an N*N feature map, and the fourth feature can also be an N*N feature map. For the third feature, each element in the N*N feature map represents a feature of a receptive field of a certain size in the second image to be recognized. For the fourth feature, each element in the N*N feature map represents a feature of a receptive field of a certain size in the sample image. The size of N can be determined according to actual needs, such as the size of the second image to be recognized and the sample image, the size of the convolution kernel, and / or the stride of the convolution.

[0096] Because the second discriminator network outputs the third and fourth features, it does not directly determine the similarity between the second image to be identified and the sample image; that is, it does not directly determine whether the second image to be identified is real or fake. Using the third and fourth features allows for a more accurate determination of the second similarity between the second image to be identified and the sample image, thereby improving the accuracy of the second similarity score. Simultaneously, based on the second similarity score, it is easier to update the network parameters of the second generator network and the second discriminator network, thus improving the update efficiency of the second generator network and the second discriminator network.

[0097] Step S30: Determine the second similarity between the second image to be identified and the sample image based on the third feature, the fourth feature, and the similarity function.

[0098] After obtaining the third and fourth features, a second similarity between the second image to be identified and the sample image can be determined based on the third and fourth features and the similarity function. The similarity function can be any function that can determine the similarity between the third and fourth features. This step can be performed by an optimizer or by other structures.

[0099] For example, the similarity function can calculate the arithmetic mean of each element in the N*N feature map of the third feature to obtain the third arithmetic mean, and calculate the arithmetic mean of each element in the N*N feature map of the fourth feature to obtain the fourth arithmetic mean. Then, the difference between the third and fourth arithmetic means is determined, and the second similarity is determined based on this difference. If the difference is greater than a second threshold, the second similarity is determined to be 1, indicating that the second image to be identified is the same as the sample image. If the difference is less than the second threshold, the second similarity is determined to be 0, indicating that the second image to be identified is different from the sample image.

[0100] The similarity function here can be the same as the form similarity function in step S300, or it can be a different similarity function.

[0101] In another embodiment, the difference can also have a mapping relationship with the second similarity. The second similarity can be determined based on the difference. When the second similarity is greater than the second similarity threshold, it means that the second image to be identified is the same as the sample image. When the second similarity is less than the second similarity threshold, it means that the second image to be identified is different from the sample image.

[0102] Step S40: Update the network parameters of the first generator network, the second generator network, the first discriminator network, and the second discriminator network based on the first similarity and the second similarity.

[0103] After determining the second similarity, the network parameters of the second generator network and the second discriminator network can be updated based on the second similarity. The specific update process is not limited; it only needs to update the network parameters of the second generator network and the second discriminator network according to the second similarity. Updates can be performed using an update function, a loss function, or an optimizer, etc.

[0104] When the second image to be identified is determined to be different from the sample image based on the second similarity, the network parameters of the second generation network are updated to improve the similarity between the second image to be identified generated by the second generation network based on the first image to be identified and the sample image in the future. This makes the second image to be identified more similar to the sample image and reduces the probability that the second discriminator will identify the second image to be identified as different from the sample image.

[0105] When the second image to be identified is determined to be the same as the sample image based on the second similarity, the network parameters of the second discriminator network are updated to improve the ability of the second discriminator network to distinguish between the second image to be identified and the sample image, so that the similarity between the second image to be identified and the sample image is lower, thereby increasing the probability of distinguishing the second image to be identified from the sample image as different images, that is, increasing the probability of the second discriminator distinguishing the second image to be identified as fake.

[0106] By iteratively updating the network parameters of the second generator network and the second discriminator network, the similarity between the second image to be identified generated by the second generator network and the sample image can be increased, making the second image to be identified more like a "real image" and reducing the probability that the second discriminator will identify the second image to be identified as different from the sample image. Simultaneously, it can also increase the probability that the second discriminator network will identify the second image to be identified as different from the sample image, i.e., the probability of identifying the second image to be identified as a "fake image." This process is repeated until the probability that the second discriminator network identifies the second image to be identified as the same image as the sample image approaches 50%.

[0107] The third and fourth features determined by this embodiment can more accurately determine the second similarity between the second image to be identified and the sample image, improving the accuracy of the second similarity. Therefore, the network parameters of the second generator network and the second discriminator network can be updated better based on the second similarity, thereby improving the update efficiency of the second generator network and the second discriminator network.

[0108] After training, the second image to be identified, x', generated by the second generator network is the same as the sample image x, thus maintaining the cyclic consistency of the recurrent generative adversarial network and reducing the probability that the first generator network generates the same first image to be identified based on different sample images. This enables different input images to generate different output images, meaning that the first generator network can generate different first images to be identified based on different sample images.

[0109] In another embodiment, the network parameters of the first generator network, the second generator network, the first discriminator network, and the second discriminator network can also be updated based on the first similarity and the second similarity.

[0110] In another embodiment, reference Figure 4 This is a schematic diagram for determining the second similarity. Step S30 involves determining the second similarity between the second image to be identified and the sample image based on the third and fourth features, including:

[0111] Step S31: Perform spectral normalization on the third feature to obtain the third result, and perform spectral normalization on the fourth feature to obtain the fourth result.

[0112] Step S32: Determine the second similarity based on the third result, the fourth result, and the similarity function.

[0113] After obtaining the third and fourth features, the third feature is spectrally normalized to obtain the third result, and the fourth feature is spectrally normalized to obtain the fourth result. Then, the second similarity is determined based on the third and fourth results and the similarity function. By performing spectral normalization on the third and fourth features, problems such as hindered network model training or overfitting can be reduced when the difference between the first image to be identified and the sample image is large.

[0114] The specific process of spectral normalization will not be explained in detail. For example, spectral normalization can be performed on the third and fourth features based on the image gradient.

[0115] In another embodiment, the third feature includes N*N third elements, and the fourth feature includes N*N fourth elements;

[0116] The third result is obtained by performing spectral normalization on the third feature, and the fourth result is obtained by performing spectral normalization on the fourth feature, including:

[0117] Perform spectral normalization on the N*N third elements to obtain the third result;

[0118] Perform spectral normalization on the N*N fourth elements to obtain the fourth result.

[0119] In another embodiment, the method further includes:

[0120] A third reference factor for the second image to be identified is determined based on the pixel values ​​of the pixels in the second image to be identified, and a fourth reference factor for the sample image is determined based on the pixel values ​​of the pixels in the sample image; the third and fourth reference factors are used to represent at least one type of image information in the same dimension.

[0121] The third reference factor is image information of at least one dimension of the second image to be identified, and the fourth reference factor is image information of at least one dimension of the sample image. The third and fourth reference factors are image information of the same dimension.

[0122] For example, if the third reference factor is image information of dimension A, and the fourth reference factor is also image information of dimension A, then the fourth reference factor also includes image information of dimensions A and B. The third and fourth reference factors identify image information of at least the same dimension, which facilitates comparison of the similarity between the second image to be identified and the sample image based on image information of the same dimension, thereby more accurately determining the second similarity and improving the accuracy of the second similarity.

[0123] Step S30, determining the second similarity between the second image to be identified and the sample image based on the third and fourth features, further includes:

[0124] The second similarity is determined based on the third reference factor, the fourth reference factor, the third feature, and the fourth feature.

[0125] After determining the third and fourth reference factors, the second similarity can be determined based on the third and fourth reference factors, the third feature, and the fourth feature. The addition of the third and fourth reference factors, based on the third and fourth features, provides more image information for determining the second similarity, resulting in a more accurate second similarity. This improved accuracy allows for more accurate updates to the second generator network and the second discriminator network, facilitating the reduction of noise in the second image to be identified and generating a second image with higher similarity to the reference image, thus improving the image quality of the generated second image to be identified.

[0126] In one embodiment, the third and fourth reference factors include at least one of the following: brightness, contrast, image structure, and / or signal-to-noise ratio.

[0127] The second similarity can be determined based on the brightness similarity between the second image to be identified and the sample image, as well as the similarity between the third and fourth features. The greater the sum of the similarity between the third and fourth features and the brightness similarity, the greater the second similarity, indicating that the second image to be identified and the sample image are more similar, and the less likely the second discriminator is to distinguish the second image to be identified from the sample image as different images, that is, the less likely the discriminator is to distinguish the second image to be identified as a "fake image".

[0128] The second similarity can also be determined based on the contrast similarity between the second image to be identified and the sample image, as well as the similarity between the third and fourth features. The greater the sum of the similarity between the third and fourth features and the contrast similarity, the greater the second similarity, indicating that the second image to be identified and the sample image are more similar, and the less likely the second discriminator is to distinguish the second image to be identified from the sample image as different images, that is, the less likely the second discriminator is to distinguish the second image to be identified as a "fake image".

[0129] The second similarity can also be determined based on the image structure similarity between the second image to be identified and the sample image, as well as the similarity between the third and fourth features. The greater the sum of the similarity between the third and fourth features and the image structure similarity, the greater the second similarity, indicating that the second image to be identified and the sample image are more similar, and the less likely the second discriminator is to distinguish the second image to be identified from the sample image as different images, that is, the less likely the second discriminator is to distinguish the second image to be identified as a "fake image".

[0130] The second similarity can also be determined based on the contrast similarity, brightness similarity, image structure similarity, and the similarity of the third and fourth features between the second image to be identified and the sample image. A reference factor similarity can be determined based on the contrast similarity, brightness similarity, and image structure similarity between the second image to be identified and the sample image. Then, the second similarity is determined based on the similarity of the third and fourth features and the reference factor similarity. A higher reference factor similarity indicates a greater similarity between the second image to be identified and the sample image.

[0131] The reference factor similarity between the second image to be identified and the sample image can be determined by the pixel values ​​of the second image to be identified and the sample image. Specifically, it can be determined by the Structural Similarity (SSIM) formula, which calculates the brightness similarity, contrast similarity, image structure similarity, and reference factor similarity between the second image to be identified and the sample image. The reference factor similarity is the calculated value of SSIM. The value of SSIM can be in the range of [0, 1]. The larger the value of SSIM, the greater the similarity between the second image to be identified and the sample image.

[0132] The similarity score can also be determined based on the contrast similarity, brightness similarity, image structure similarity, signal-to-noise ratio (SNR) of the second image to be identified, the SNR of the sample image, and the similarity between the third and fourth features. The SNR can be determined by the number of pixels and the grayscale value of the pixels in the image. The specific determination process can be based on the SNR formula for images.

[0133] The smaller the difference between the signal-to-noise ratio (SNR) of the second image to be identified and the SNR of the sample image, the greater the similarity between the second image to be identified and the sample image.

[0134] In one embodiment, step S10, generating a second image to be identified based on a first image to be identified using a second generative network, includes: generating the second image to be identified through a spatial attention mechanism.

[0135] When generating the second image to be identified, the spatial attention mechanism can be used to obtain the spatial correlation information of the image to be identified. Based on the extracted spatial correlation information of the image to be identified, a second image to be identified with a higher similarity to the sample image is generated. The second image to be identified has a higher second similarity value with the sample image, and the generated second image to be identified is more like a "real image". This improves the second similarity and reduces the probability that the second discriminator will identify the second image to be identified as a different image from the sample image.

[0136] In another embodiment, reference Figure 5 This is an image generation method, which includes:

[0137] Step a, configure the acquisition parameters; the acquisition parameters include at least the image resolution.

[0138] Step b: Based on the acquisition parameters, sample images are acquired in the target application through multiple acquisition threads; the sample images contain depth information.

[0139] Step c: Optimize the sample image, including at least distortion correction.

[0140] Step d: Using the image generation model trained in any of the above-described image generation model training method embodiments, generate the target image based on the optimized sample image.

[0141] After training the image generation model using the above-mentioned training method, the target image can be generated using the trained image generation model.

[0142] refer to Figure 6 This is a schematic diagram of a training device for an image generation model, which includes:

[0143] First image generation module 1 is used to generate a first image to be recognized based on a sample image using a first generation network.

[0144] The first recognition module 2 is used to recognize the first image to be recognized using a first discriminant network and output a first feature, and recognize a reference image and output a second feature;

[0145] The first similarity determination module 3 is used to determine the first similarity between the first image to be identified and the reference image based on the first feature, the second feature and the similarity function;

[0146] The first update module 4 is used to update the network parameters of the first generation network and the first discrimination network based on the first similarity.

[0147] In another embodiment, the first similarity determination module 3 includes:

[0148] The first spectral normalization unit is used to perform spectral normalization processing on the first feature to obtain a first result, and to perform spectral normalization processing on the second feature to obtain a second result;

[0149] The first similarity determination unit is used to determine the first similarity based on the first result, the second result, and the similarity function.

[0150] In another embodiment, the training device further includes:

[0151] The first determining module is configured to determine a first reference factor of the first image to be identified and a second reference factor of the reference image based on pixel values; the first reference factor and the second reference factor are used to represent at least one type of image information in the same dimension;

[0152] The first similarity determination module 3 is also used for:

[0153] The first similarity is determined based on the first reference factor, the second reference factor, the first feature, the second feature, and the similarity function.

[0154] In another embodiment, the first reference factor and the second reference factor include at least one of the following: brightness, contrast, image structure, and / or signal-to-noise ratio.

[0155] In another embodiment, the first image generation module 1 is specifically used for:

[0156] The first image to be identified is generated using a spatial attention mechanism.

[0157] In another embodiment, the device further includes:

[0158] The second image to be identified module is used to generate a second image to be identified based on the first image to be identified using a second generation network.

[0159] The second recognition module is used to recognize the second image to be recognized using the second discriminant network and output the third feature, and recognize the sample image and output the fourth feature;

[0160] The second similarity determination module is used to determine the second similarity between the second image to be identified and the sample image based on the third feature, the fourth feature and the similarity function;

[0161] The second update module is used to update the network parameters of the first generator network, the second generator network, the first discriminator network, and the second discriminator network based on the first similarity and the second similarity.

[0162] In another embodiment, the second similarity determination module includes:

[0163] The second spectral normalization unit is used to perform spectral normalization processing on the third feature to obtain the third result, and to perform spectral normalization processing on the fourth feature to obtain the fourth result.

[0164] The second similarity determination unit is used to determine the second similarity based on the third result, the fourth result, and the similarity function.

[0165] In another embodiment, the device further includes:

[0166] The second determining module determines a third reference factor of the second image to be identified and a fourth reference factor of the sample image based on pixel values; the third reference factor and the fourth reference factor are used to represent at least one type of image information in the same dimension.

[0167] The second similarity determination module is also used for:

[0168] The second similarity is determined based on the third reference factor, the fourth reference factor, the third feature, the fourth feature, and the similarity function.

[0169] In another embodiment, the third reference factor and the fourth reference factor include at least one of the following: brightness, contrast, image structure, and / or signal-to-noise ratio.

[0170] In another embodiment, the second image-to-be-recognized generation module is specifically used for:

[0171] The second image to be identified is generated using a spatial attention mechanism.

[0172] refer to Figure 7 This is a schematic diagram of an image generation apparatus, which includes:

[0173] Configuration module 10 is used to configure acquisition parameters; the acquisition parameters include at least: image resolution;

[0174] Acquisition module 20 is used to acquire the sample image in the target application through multiple acquisition threads according to the acquisition parameters; the sample image has depth information;

[0175] Optimization module 30 is used to optimize the sample image, the optimization including at least: distortion correction;

[0176] The generation module 40 is used to generate a target image based on the optimized sample image using an image generation model trained by any of the above training methods. The generated target image contains depth information and can be used as input to a depth estimation model in an electronic device for training the depth estimation model.

[0177] In another embodiment, a specific application scenario example is also provided.

[0178] With the development of machine vision, the superiority of deep learning methods in various vision tasks is becoming increasingly prominent. In order to train a depth estimation model that can be deployed on electronic devices such as mobile phones and tablets to generate images, it is necessary to obtain training samples that meet business requirements, such as image training samples that include RGB and Depth information, with each image training sample including RGB and Depth datasets.

[0179] Typically, training samples are collected using Kinect depth cameras or stereo cameras based on specific needs. This process requires significant manpower and resources, and acquiring data from scarce scenarios is even more costly. Even with substantial investment, the quality of the obtained depth information is often poor. Furthermore, for images involving human figures, numerous data privacy concerns further complicate the acquisition of large-scale data. Moreover, many current mobile phone cameras require night mode, for which training samples including RGB and depth data are virtually nonexistent, necessitating simulation. To address the challenge of obtaining high-quality RGB and depth training samples, this embodiment proposes a training sample generation method based on the GTA V game and the CycleGAN network.

[0180] The reason why RGB and depth datasets can be constructed based on simulation methods is that the real-time graphics rendering field has achieved increasingly higher levels of realism in recent years, and the available resources are also relatively abundant, most notably in large-scale, highly realistic games such as GTA V and Watch Dogs. In the field of image data simulation, one approach involves building scenes using Unreal Engine or Unity, constructing acquisition logic, and developing acquisition scripts to complete the acquisition. Acquisition is often based on relatively realistic game scenes, with GTA V being a particularly well-researched example. This is mainly because the game boasts high realism, rich scenes, and freely changeable weather. Developing acquisition scripts specifically for the GTA V game interface significantly improves efficiency compared to commonly used methods.

[0181] Typically, you can build the required game scene based on the Unreal or Unity game graphics engine and with the help of open source resources. Based on the engine's RGB and depth camera interfaces, you can develop scripts to collect RGB & depth data and compile them together with the built scene code. In this way, during the game scene operation, you can collect data frame by frame through shortcut keys or automation.

[0182] This method of building scenes based on Unreal or Unity game graphics engines and leveraging open-source resources offers high scene flexibility, but simulating high-quality, large-scale data requires a huge investment of time and effort in scene construction, and the realism is still insufficient, leading to high data collection costs.

[0183] Another approach is based on the existing GTA V game. This approach only requires configuring a script to collect data, compiling the script into a mod file and a motion trajectory file, and placing them in the root directory of the GTA V game installation. During the game's startup process, the mod file and motion trajectory file can be loaded, and the data collection can be completed automatically when the game is running.

[0184] This acquisition scheme is mainly designed for autonomous driving scenarios. The perspective and speed of movement are too limited, and the degree of control freedom is poor. It is not suitable for mobile phone camera scenarios. Moreover, in road driving scenarios, the vehicle moves quickly, resulting in poor local alignment of RGB & depth, distortion at the edges of RGB images, and a lack of image realization post-processing.

[0185] This embodiment can configure a acquisition reference and then acquire sample images according to the acquisition parameters.

[0186] refer to Figure 8 This is a schematic diagram of another image generation method.

[0187] The acquisition parameters include information such as image resolution, acquisition scene, and acquisition angle. Based on these parameters, corresponding sample images are acquired. For example, in free mode, RGB and depth data from various angles and scenes can be acquired based on the GTA V game. Distortion correction can be performed using a lens calibration plugin, image alignment issues can be addressed with additional acquisition strategies, and finally, an optimized cycleGAN network is used to achieve image realism, making the acquired RGB & depth simulation data more realistic and flexible.

[0188] Step a may specifically include: Based on the open scripts of the GTA V game, using the open-source plugin project GTAVisionExport, the advantages of this plugin are its ease of modification and compilation, and it does not interfere with GTA V's own parameter setting shortcuts, while also being well-compatible with character and vehicle density setting scripts. For ease of subsequent data parsing, we modified the depth data type in the acquired image samples to DXGI_FORMAT_R32G8X24_TYPELESS.

[0189] In one embodiment, the image resolution can be determined based on the task density and / or processing density in the image by configuring acquisition parameters for the character density and / or image density included in the sample images, and then acquiring sample images of the corresponding density according to these acquisition parameters. For example, character and / or vehicle density control scripts and configuration files can be imported into the GTA V game using OpenVI software.

[0190] Step b specifically includes:

[0191] Acquiring RGB data, depth data, and stencil information via multiple acquisition lines, such as using `export_get_color_buffer`, `export_get_depth_buffer`, and `export_get_stencil_buffer`, can make the acquisition process smoother. This plugin is highly compatible with GTA V's built-in acquisition scripts, which offer high freedom of movement, controllable weather and lighting conditions, time settings, and controlled density of characters and vehicles. Furthermore, with optimized acquisition logic, it can maintain smooth gameplay even at high resolutions and continuous frame rates. Compile the modified project into a .mod file and place it in the GTA V game's installation directory. Step b will then allow you to acquire sample images with depth information.

[0192] Step c specifically includes: Since we discovered significant edge distortion in the sample images during the trial data collection process, calibration of the RGB and depth data is particularly important. Without calibration, the RGB and depth data would be unusable. This is achieved by optimizing the sample images through distortion correction, etc. For example, in GTA V, we used a pre-compiled No Chromatic aberration & Lens distortion 1.57 plugin, and imported the camera distortion correction script into the GTA V game via OpenVI software to optimize the sample images.

[0193] For example, when launching the GTA V game, the loading of acquisition parameters, including the acquisition script and distortion correction script, is completed. To prevent crashes during loading, the game mode can be adjusted to two slow motions 10 seconds after the game screen enters, and then acquisition can be performed by pressing the R key. The two slow motions are mainly to better align high-speed moving objects in local areas of the image.

[0194] If crashes still occur during the data acquisition process, the image resolution can be appropriately reduced, such as by adjusting the density of people and / or vehicles, thereby reducing the resolution of the acquired sample images.

[0195] Step d specifically includes:

[0196] After data acquisition, since the brightness and contrast distribution in the GTA V game was still not rich enough, we first performed random brightness adjustments on the dataset before obtaining the target image. Next, to obtain more realistic RGB images, we used the acquired sample images and RGB images taken with actual mobile phones as the training set. Based on actual realism requirements, our goal was to achieve realistic style transfer. To achieve this, we trained a recurrent generative adversarial network (CycGAN) model. A schematic diagram of the CycGAN structure is shown below. Figure 9 The discriminators Dx and Dy, and the generators G and F, can be basic convolutional networks with some residual structures. The first generator network can be generator G, the first discriminator network can be discriminator Dy, the second generator network can be generator F, and the second discriminator network can be discriminator Dx.

[0197] X and Y come from different image domains. G:X->Y, the generator G realizes the transfer from X to Y; F:Y->X, the generator F realizes the transfer from Y to X; the discriminator Dx is used to judge the similarity between X and F(Y); the discriminator Dy is used to judge the similarity between Y and G(X).

[0198] First, a spatial attention mechanism is added to the recurrent generative adversarial network structure to extract spatial correlation information, which is beneficial for the network to generate structure-preserving images.

[0199] Typically, the discriminator in a Recurrent Generative Adversarial Network (RGAN) outputs a single value, indicating the degree of truth or falsehood. However, the discriminator in this embodiment outputs N*N features, where each value represents a receptive field of a certain size on the input image fed into the generative network. Intuitively, this means judging the truth or falsehood of a repeatable region under a crop in the original image, providing a more refined local judgment. This RGAN discriminator borrows the patching concept from the PatchGAN discriminator structure; specifically, the output of the last convolutional layer of the discriminator is modified to an N*N feature map.

[0200] To reduce noise in the output image, we use SSIM loss as a constraint. SSIM loss is often used in image quality, depth estimation, image segmentation and other fields. Since SSIM loss can constrain brightness, contrast and structure at the same time, it can improve the image generation quality in CycleGAN image generation.

[0201] When the image spans a large distance between the source and target domains, mode collapse or overfitting can easily occur during actual training due to training instability. To address this issue, we found through repeated experiments that it is necessary to constrain the discriminator. Specifically, a spectral normalization strategy can be added to the discriminator in each iteration, which can largely avoid the problems of mode collapse or overfitting.

[0202] The final trained network model can achieve style transfer to the maximum extent while also producing low noise in the output image.

[0203] refer to Figure 9The training objective of the CycleGAN model is for the generator to strive to generate near-realistic fake target domain images from the source domain, while the discriminator strives to distinguish between real and fake images. Therefore, our training strategy is to train the generator and discriminator alternately based on the target loss. The specific training process is as follows:

[0204] (a) Feed a real image into the decision maker, label the sample as real, and train the decision maker.

[0205] (b) The generator generates a fake image and sends it to the decision-maker, marking the sample as fake, and trains the decision-maker.

[0206] (c) The generator generates a fake image and sends it to the judge. The generator is then trained based on the judgment result.

[0207] (d) Repeat the above steps to complete the training of a cycleGAN.

[0208] After training the modified CycleGAN network, it learns style transfer from simulation to realism. Finally, the trained CycleGAN model is applied to all acquired simulation images to complete the realism conversion. For the acquired depth data, it needs to be parsed into 16-bit images using analytical formulas. For high-resolution images, it should be saved in .png format whenever possible. The final output is RGB & depth data that meets the project requirements.

[0209] This embodiment enables efficient acquisition of large-scale RGB & depth datasets with a high degree of freedom. It allows for optimization of the acquired sample images, reducing distortion and misalignment, such as misalignment of RGB data and / or depth data. Distortion correction can be performed after data acquisition using an alignment algorithm.

[0210] By using the cycleGAN network, simulation data can be made more realistic and richer, meeting project requirements.

[0211] In another embodiment, it can also be implemented using a larger GAN model such as forkGAN or styleGAN.

[0212] When acquiring sample images, you can also configure the scene, viewpoint, etc., and efficiently and flexibly adjust the time, weather, scene, viewpoint, etc.

[0213] Distortion correction was performed during the acquisition process, and a two-stage slow-motion acquisition strategy was adopted to ensure that the acquired images were accurately aligned.

[0214] The collected data was subjected to random parameter brightness adjustment, and the image was made realistic through a modified cycleGAN network. The resulting image better met the project requirements in terms of brightness, contrast, and realism.

[0215] For the original CycleGAN network, we incorporate a joyful attention mechanism and the patch concept, apply SSIM constraints to the loss function, and address overfitting or pattern collapse issues caused by training instability using spectral normalization. This results in images with significant style transfer and low noise. Furthermore, our network achieves high-quality generated images with relatively low training costs.

[0216] In another embodiment, an electronic device is also provided, comprising:

[0217] A processor and a memory for storing executable instructions capable of running on the processor, wherein:

[0218] When the processor runs the executable instructions, the executable instructions perform the method described in any of the above embodiments.

[0219] In another embodiment, a non-transitory computer-readable storage medium is also provided, wherein computer-executable instructions are stored therein, which, when executed by a processor, implement the method described in any of the above embodiments.

[0220] It should be noted that the terms "first" and "second" in the embodiments of this disclosure are for ease of description and distinction only, and have no other specific meaning.

[0221] Figure 10 This is a block diagram illustrating an electronic device according to an exemplary embodiment. For example, the electronic device may be a mobile phone, computer, digital broadcasting electronic device, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0222] Reference Figure 10 The electronic device may include one or more of the following components: processing component 802, memory 804, power component 806, multimedia component 808, audio component 810, input / output (I / O) interface 812, sensor component 814, and communication component 816.

[0223] Processing component 802 typically controls the overall operation of an electronic device, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 802 may include one or more processors 820 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0224] Memory 804 is configured to store various types of data to support the operation of an electronic device. Examples of this data include instructions for any application or method operating on the electronic device, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0225] Power component 806 provides power to various components of the electronic device. Power component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device.

[0226] Multimedia component 808 includes a screen that provides an output interface between the electronic device and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the electronic device is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0227] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when the electronic device is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0228] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0229] Sensor assembly 814 includes one or more sensors for providing state assessments of various aspects of the electronic device. For example, sensor assembly 814 can detect the on / off state of the electronic device, the relative positioning of components such as the display and keypad of the electronic device, changes in the position of the electronic device or a component of the electronic device, the presence or absence of user contact with the electronic device, orientation or acceleration / deceleration of the electronic device, and temperature changes of the electronic device. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0230] Communication component 816 is configured to facilitate wired or wireless communication between electronic devices and other devices. The electronic devices can access wireless networks based on communication standards, such as WiFi, 4G, or 5G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra-Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0231] In an exemplary embodiment, the electronic device may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0232] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0233] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A training method for an image generation model, characterized in that, include: A first image to be identified is generated based on the sample image using a first generative network; The first discriminant network is used to identify the first image to be identified and output a first feature, and the reference image is identified and output a second feature; Based on the first feature, the second feature, and the similarity function, a first similarity between the first image to be identified and the reference image is determined. A second generative network is used to generate a second image to be identified based on the first image to be identified. The second discriminant network is used to identify the second image to be identified and output a third feature, and the sample image is identified and output a fourth feature; Based on the third feature, the fourth feature, and the similarity function, a second similarity between the second image to be identified and the sample image is determined. Based on the first similarity and the second similarity, update the network parameters of the first generator network, the second generator network, the first discriminator network, and the second discriminator network.

2. The method according to claim 1, characterized in that, Determining the first similarity between the image to be identified and the reference image based on the first feature, the second feature, and the similarity function includes: The first feature is subjected to spectral normalization to obtain a first result, and the second feature is subjected to spectral normalization to obtain a second result; The first similarity is determined based on the first result, the second result, and the similarity function.

3. The method according to claim 1, characterized in that, The method includes: A first reference factor for the first image to be identified is determined based on the pixel values ​​of the pixels in the first image to be identified, and a second reference factor for the reference image is determined based on the pixel values ​​of the pixels in the reference image; the first reference factor and the second reference factor are used to represent at least one type of image information of the same dimension. The step of determining the first similarity between the first image to be identified and the reference image based on the first feature, the second feature, and the similarity function further includes: The first similarity is determined based on the first reference factor, the second reference factor, the first feature, the second feature, and the similarity function.

4. The method according to claim 3, characterized in that, The first reference factor and the second reference factor include at least one of the following: brightness, contrast, image structure and / or signal-to-noise ratio.

5. The method according to claim 1, characterized in that, The step of generating a first image to be identified based on a sample image using a first generative network includes: The first image to be identified is generated using a spatial attention mechanism.

6. The method according to claim 1, characterized in that, Determining the second similarity between the second image to be identified and the sample image based on the third feature, the fourth feature, and the similarity function includes: The third feature is subjected to spectral normalization to obtain the third result, and the fourth feature is subjected to spectral normalization to obtain the fourth result; The second similarity is determined based on the third result, the fourth result, and the similarity function.

7. The method according to claim 1, characterized in that, The method includes: A third reference factor for the second image to be identified is determined based on the pixel values ​​of the pixels in the second image to be identified, and a fourth reference factor for the sample image is determined based on the pixel values ​​of the pixels in the sample image; the third reference factor and the fourth reference factor are used to represent at least one type of image information of the same dimension. Determining the second similarity between the second image to be identified and the sample image based on the third feature, the fourth feature, and the similarity function includes: The second similarity is determined based on the third reference factor, the fourth reference factor, the third feature, the fourth feature, and the similarity function.

8. The method according to claim 7, characterized in that, The third and fourth reference factors include at least one of the following: brightness, contrast, image structure, and / or signal-to-noise ratio.

9. The method according to claim 1, characterized in that, The step of generating a second image to be identified based on the first image to be identified using a second generative network includes: The second image to be identified is generated using a spatial attention mechanism.

10. A method for generating an image, characterized in that, include: Configure the data acquisition parameters; The acquisition parameters include at least: image resolution; Based on the acquisition parameters, the sample image is acquired in the target application through multiple acquisition threads; the sample image has depth information. The sample image is optimized, and the optimization includes at least: distortion correction; Using the image generation model trained according to any one of claims 1 to 9, a target image is generated based on the optimized sample image.

11. A training device for an image generation model, characterized in that, include: The first image to be identified module is used to generate a first image to be identified based on a sample image using a first generation network. The first recognition module is used to recognize the first image to be recognized using a first discriminant network and output a first feature, and recognize a reference image and output a second feature; The first similarity determination module is used to determine a first similarity between the first image to be identified and the reference image based on the first feature, the second feature and the similarity function; The first update module is configured to use a second generator network to generate a second image to be identified based on the first image to be identified; use a second discriminator network to identify the second image to be identified and output a third feature, and identify a sample image and output a fourth feature; determine a second similarity between the second image to be identified and the sample image based on the third feature, the fourth feature, and the similarity function; and update the network parameters of the first generator network, the second generator network, the first discriminator network, and the second discriminator network based on the first similarity and the second similarity.

12. An image generation apparatus, characterized in that, include: The configuration module is used to configure the acquisition parameters; The acquisition parameters include at least: image resolution; The acquisition module is used to acquire the sample image in the target application through multiple acquisition threads according to the acquisition parameters; the sample image has depth information; An optimization module is used to optimize the sample image, wherein the optimization includes at least: distortion correction; The generation module is used to generate a target image based on the optimized sample image using the image generation model trained according to claim 11.

13. An electronic device, characterized in that, include: A processor and a memory for storing executable instructions capable of running on the processor, wherein: When the processor is used to run the executable instructions, the executable instructions perform the method described in any one of claims 1 to 9 or claim 10.

14. A non-transitory computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when executed by a processor, implement the method described in any one of claims 1 to 9 or claim 10.

Images