Image generation method and device, electronic device and storage medium

Abstract

The disclosed embodiments disclose an image generation method and device, an electronic device and a storage medium, the method comprising: extracting content features of a first image; respectively extracting full-image style features of a second image and object style features of a local image block including an object in the second image; determining a target style feature based on at least the full-image style feature and the object style feature; and generating a third image based on the content features and the target style feature.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on the Chinese patent application with application number 201910352792.1 and application date April 29, 2019, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby incorporated into this application by introduction. Technical Field

[0003] The present disclosure relates to the field of computer technology, and in particular to an image generating method and device, an electronic device and a storage medium. Background Art

[0004] In related technologies, the overall conversion of image style can be achieved through deep learning and other methods. For example, the brightness, contrast, lighting, color, artistic features or artwork in an image can be converted to obtain images with different styles. However, the conversion of style can only work on the overall image, and the style of the target object in the image can only be converted along with the overall style of the image, which cannot reflect the style of the target object, resulting in monotonous style of the converted image, and the overall image and the area where the target object is located are not coordinated. Summary of the invention

[0005] The embodiments of the present disclosure provide an image generating method and device, an electronic device, and a storage medium.

[0006] In a first aspect, an embodiment of the present disclosure provides an image generation method, comprising:

[0007] extracting content features of the first image;

[0008] Respectively extracting a full-image style feature of a second image and an object style feature of a local image block including an object in the second image, wherein the second image and the first image have different styles;

[0009] Determining a target style feature at least according to the overall image style feature and the object style feature;

[0010] A third image is generated according to the content feature and the target style feature, so that the third image has content corresponding to the content feature and a style corresponding to the target style feature.

[0011] According to the image generation method of the embodiment of the present disclosure, the target style feature determined by the full-image style feature and the object style feature of the second image and the content feature of the first image can be used to obtain a third image having content corresponding to the content feature and a style corresponding to the target style feature. When performing style conversion on the image, not only the full-image style of the image can be converted, but also the style of the local image block where the object is located in the image can be appropriately converted, so that the full image of the image and the area where the object is located are coordinated, the realism of the image of the style transfer is improved, and the detection accuracy of the object in the image can be improved. Furthermore, the generated third image has content corresponding to the content feature of the first image and a style corresponding to the target style feature, which expands the richness of image samples having content corresponding to the content feature of the first image, especially for difficult image samples or a small number of image samples, greatly reducing the cost of manual collection and annotation.

[0012] In a second aspect, an embodiment of the present disclosure provides an image generating device, including:

[0013] A first extraction module, used to extract content features of the first image;

[0014] A second extraction module is used to extract the overall style features of a second image and the object style features of a local image block including an object in the second image, wherein the second image and the first image have different styles;

[0015] A determination module, configured to determine a target style feature based at least on the overall image style feature and the object style feature;

[0016] The first generating module is used to generate a third image according to the content feature and the target style feature, so that the third image has content corresponding to the content feature and a style corresponding to the target style feature.

[0017] In a third aspect, an embodiment of the present disclosure provides an electronic device, including:

[0018] processor;

[0019] a memory for storing processor-executable instructions;

[0020] Wherein, the processor is configured to: execute the above-mentioned image generation method of the embodiment of the present disclosure.

[0021] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having computer program instructions stored thereon, and when the computer program instructions are executed by a processor, the above-mentioned image generation method of the embodiment of the present disclosure is implemented.

[0022] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

[0023] Other features and aspects of embodiments of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to illustrate the technical solutions of the embodiments of the present disclosure.

[0025] Figure 1 The process of the image generation method according to the embodiment of the present disclosure is shown Figure 1 ;

[0026] Figure 2 The process of the image generation method according to the embodiment of the present disclosure is shown Figure 2 ;

[0027] Figure 3 A schematic diagram of style feature fusion according to an embodiment of the present disclosure is shown;

[0028] Figure 4 The process of the image generation method according to the embodiment of the present disclosure is shown Figure 3 ;

[0029] Figure 5 The process of the image generation method according to the embodiment of the present disclosure is shown Figure 4 ;

[0030] Figures 6A-6C A schematic diagram showing an application of the image generation method according to an embodiment of the present disclosure;

[0031] Figure 7 A block diagram showing an image generating device according to an embodiment of the present disclosure Figure 1 ;

[0032] Figure 8 A block diagram showing an image generating device according to an embodiment of the present disclosure Figure 2 ;

[0033] Fig. 9 A schematic diagram showing the structure of an electronic device according to an embodiment of the present disclosure Figure 1 ;

[0034] Fig.10 A schematic diagram showing the structure of an electronic device according to an embodiment of the present disclosure Figure 2 . DETAILED DESCRIPTION

[0035] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0036] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0037] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0038] In addition, in order to better illustrate the embodiments of the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the embodiments of the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the purpose of the embodiments of the present disclosure.

[0039] Figure 1 The process of the image generation method according to the embodiment of the present disclosure is shown Figure 1 ,like Figure 1 As shown, the method includes:

[0040] In step S11, extracting content features of the first image;

[0041] In step S12, the full-image style features of the second image and the object style features of the local image block including the object in the second image are respectively extracted, wherein the second image and the first image have different styles;

[0042] In step S13, a target style feature is determined based on at least the full image style feature and the object style feature;

[0043] In step S14, a third image is generated according to the content feature and the target style feature, so that the third image has content corresponding to the content feature and style corresponding to the target style feature.

[0044] According to the image generation method of the embodiment of the present disclosure, the target style feature determined by the full-image style feature and the object style feature of the second image and the content feature of the first image can be used to obtain a third image having content corresponding to the content feature and a style corresponding to the target style feature. When performing style conversion on the image, not only the full-image style of the image can be converted, but also the style of the local image block where the object is located in the image can be appropriately converted, so that the full image of the image and the area where the object is located are coordinated, the realism of the image of the style transfer is improved, and the detection accuracy of the object in the image can be improved. Furthermore, the generated third image has content corresponding to the content feature of the first image and a style corresponding to the target style feature, which expands the richness of image samples having content corresponding to the content feature of the first image, especially for difficult image samples or a small number of image samples, greatly reducing the cost of manual collection and annotation.

[0045] In a possible implementation, the method may be executed by a terminal device, which may be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The method may be implemented by a processor calling a computer-readable instruction stored in a memory. Alternatively, the method may be executed by a server.

[0046] In a possible implementation, both the first image and the second image may be images including one or more objects, and the objects may be objects of different categories. The first image and / or the second image may include at least one of the following objects: motor vehicles, non-motor vehicles, people, traffic signs, traffic lights, trees, animals, buildings, and obstacles.

[0047] In a possible implementation, the first image and the second image have different image styles, and the image styles may include brightness, contrast, lighting, color, artistic features or artwork in the image. In the example, the first image and the second image may be taken in different environments, for example, images taken in daytime, nighttime, rain, fog, etc. For example, the first image is taken at a certain location during the daytime, and the second image is taken at another location at night.

[0048] In a possible implementation, in step S11, content features of the first image may be extracted. The content features may be used to represent the content included in the first image. As an example, the content features may be at least one of the information such as the category, shape, and position of the object in the first image, or the content features may also include the background of the first image. As another example, the content features may include the content features of the entire first image. In the example, the content features of the first image may be extracted by a feature extractor. In practical applications, the feature extractor may be implemented by a convolutional neural network, etc., and the embodiments of the present disclosure do not limit the implementation of the feature extractor. The content features may include a feature map or feature vector of the first image obtained by the feature extractor.

[0049] In a possible implementation, the content feature of the first image may include an object content feature of a local image block including an object in the first image, and multiple local image blocks may be intercepted in the first image, each of which may include an object, and the object content feature of each local image block may be extracted respectively. For example, the object content feature of each local image block of the first image may be extracted by a feature extractor.

[0050] In a possible implementation, in step S12, a feature extractor may be used to extract the overall style features of the second image and the object style features of the local image blocks including the objects in the second image. Exemplarily, the feature extractor may be implemented by a neural network such as a convolutional neural network, and the embodiment of the present disclosure does not limit the implementation of the feature extractor. For example, the feature extractor may perform extraction processing on the second image and the local image blocks of the second image respectively, and obtain overall style features and object style features of each object respectively. The overall style features and object style features may be feature vectors with a length of 8 bits. The embodiment of the present disclosure does not limit the representation of overall style features and object style features.

[0051] In a possible implementation, when the second image includes multiple objects, extracting object style features of local image blocks including objects in the second image includes: intercepting multiple local image blocks in the second image, each local image block including an object; and extracting object style features of each of the local image blocks respectively.

[0052] In the example, local image blocks each including an object in the second image may be intercepted in the second image, and the object style features of each local image block may be extracted using a feature extractor. Furthermore, in the case where the multiple objects belong to multiple categories, extracting the object style features of the local image blocks including the objects in the second image also includes: fusing the object style features extracted from the multiple local image blocks of objects of the same category. For example, the object style features of multiple objects of the same category may be subjected to residual connection and other processing to fuse the multiple object style features of objects of the same category. In the example, the full-image style features and object style features of the first image may be extracted in the same manner.

[0053] In the example, if the first image is an image taken at a certain location during the day, and the second image is an image taken at another location at night, the full-image style features and object style features of the first image can express the style of the image taken during the day. For example, the brightness, contrast, illumination, and color of the image are all the styles of the image taken during the day. The full-image style features of the first image represent the style information of the first image, and the object style features of the first image represent the style information of the area where the object in the first image is located (i.e., the local image block in the first image). The full-image style features and object style features of the second image can express the style of the image taken at night. The full-image style features of the second image represent the style information of the second image, and the object style features of the second image represent the style information of the area where the object in the second image is located (i.e., the local image block in the second image).

[0054] In a possible implementation, in step S13, a target style feature is determined based on at least the overall style feature and the object style feature of the second image. The second image and the first image have different styles, and the target style feature can be determined based on the overall style feature and the object style feature of the second image, and the style of the first image is changed to an image with a style corresponding to the target style feature and having the content of the first image, that is, a third image, and the third image may include the overall first image after style transformation (that is, an image with content corresponding to the content feature of the overall first image and a style corresponding to the target style feature), and a local image block after style transformation (that is, an image with content corresponding to the object content feature of the local image block of the first image and a style corresponding to the target style feature).

[0055] In a possible implementation, step S13 may include: fusing the full-image style feature into the object style feature to obtain the target style feature. In the example, the full-image style feature and the object style feature of the second image are both feature vectors, and the full-image style feature and the object style feature may be subjected to residual connection or other processing to fuse the full-image style feature into the object style feature to obtain the target style feature.

[0056] In this way, the target style feature can be obtained by fusing the full image style feature into the object style feature. In the process of generating the third image through the target style feature, the third image has a style corresponding to the full image style feature, and the object in the third image has a style corresponding to the object style feature and coordinated with the style corresponding to the full image style feature, thereby improving the realism of the style-transferred image. Compared with fusing the object style feature into the full image style feature, the style of the object in the third image obtained by fusing the full image style feature into the object style feature is more coordinated with the style of the full image.

[0057] Based on the above embodiments, Figure 2 The process of the image generation method according to the embodiment of the present disclosure is shown Figure 2 ,like Figure 2 As shown, the method also includes:

[0058] In step S15, background style features of background image blocks in the second image are extracted, wherein the background image blocks are other image blocks in the second image except the local image blocks.

[0059] In an example, the background style features of the background image block in the second image may be extracted by a feature extractor.

[0060] Then, step S13 in the aforementioned embodiment may include: determining the target style feature according to the overall image style feature, the object style feature and the background style feature.

[0061] In this way, the target style features can be determined based on the overall image style features, the object style features and the background style features, so that the third image has a style corresponding to the overall image style features, the background of the third image has a style corresponding to the background style features, and the objects in the third image have a style corresponding to the object style features.

[0062] In a possible implementation, determining the target style feature based on the overall image style feature, the object style feature and the background style feature includes: fusing the overall image style feature into the background style feature; fusing the background style feature that has been fused with the overall image style feature into the object style feature to obtain the target style feature.

[0063] In this embodiment, the target style feature may be obtained by fusing the overall style feature of the second image to the background style feature of the second image, and then fusing the background style feature that has been fused with the overall style feature of the second image to the object style feature of the second image. In other embodiments, the target style feature may also be obtained by fusing the overall style feature of the second image to the object style feature of the second image.

[0064] For example, Figure 3 FIG. 2 shows a schematic diagram of style feature fusion according to an embodiment of the present disclosure. Figure 3 As shown, the full-image style features, background style features and object style features of the second image are all feature vectors. The full-image style features and background style features can be processed by residual connection, etc. to fuse the full-image style features into the background style features, and the background style features and object style features that have been fused with the full-image style features can be processed by residual connection, etc. to fuse the background style features that have been fused with the full-image style features into the object style features to obtain the target style features.

[0065] In this way, the target style feature can be obtained by fusing the full image style feature into the background style feature and then into the object style feature. In the process of generating the third image through the target style feature, the third image has a style corresponding to the full image style feature, and the background in the third image has a style corresponding to the background style feature and coordinated with the style corresponding to the full image style feature, and the object in the third image has a style corresponding to the object style feature and coordinated with the style corresponding to the full image style feature and the style corresponding to the background style feature, thereby improving the realism of the style-transferred image. Compared with fusing the object style feature into the background style feature and then into the full image style feature, the coordination between the style of the object, the style of the background and the style of the full image in the third image obtained by fusing the full image style feature into the background style feature and then into the object style feature is better.

[0066] In a possible implementation, in step S14, a third image having content corresponding to the content feature of the first image and a style corresponding to the target style feature may be generated according to the content feature of the first image and the target style feature. For example, the third image may be generated according to the content feature of the first image and the target style feature by an image generator.

[0067] In this embodiment, the content features of the first image may include the content features of the entire image of the first image and the object content features of each local image block of the first image. In a possible implementation, the content features of the entire image of the first image may be residually connected with the target style features, and / or the object content features of each local image block of the first image may be residually connected with the target style features. For example, the residual connection may be performed by methods such as Adaptive Instance Normalization (AdaIN) to obtain feature information having the content features of the entire image of the first image and the target style features, and / or to obtain feature information having the object content features of each local image block of the first image and the target style features.

[0068] In a possible implementation, the feature information (the feature information may include: the content features of the entire first image and the target style features, and / or, the object content features of each local image block of the first image and the target style features) may be decoded and processed by an image generator to generate an image having content corresponding to the content features of the entire first image and a style corresponding to the target style features, and / or to generate a local image block having content corresponding to the object content features of each local image block of the first image and a style corresponding to the target style features.

[0069] In a possible implementation, an image having content corresponding to the content features of the entire image of the first image and a style corresponding to the target style features, and / or content corresponding to the object content features of each local image block of the first image and a local image block of a style corresponding to the target style features may be input into an image generator to obtain an image having content corresponding to the content features of the entire image of the first image and a style corresponding to the target style features, and / or local image blocks having content corresponding to the object content features of each local image block of the first image and a style corresponding to the target style features. The above-mentioned images and local image blocks may both be referred to as third images.

[0070] In one possible implementation, Figure 4 As shown, step S14 may include:

[0071] Step S141, an image generator generates an image according to the content features and the target style features, and an image discriminator discriminates the authenticity of the generated image;

[0072] Step S142: training the image generator based on the confrontation between the discrimination result of the image discriminator and the image generated by the image generator;

[0073] Step S143: the image generator after training generates the third image.

[0074] In this embodiment, the content features of the generated image should be consistent with the content features of the first image, the content features of the local image blocks in the generated image should be consistent with the content features of the local image blocks of the first image, and the style of the generated image should be consistent with the target style features. However, the generated image is an image generated by the image generator, and there may be deviations between the content of the generated image and the first image or the local image blocks of the first image, and there may also be deviations between the style of the generated image and the target style features; the network loss can be determined based on the deviation, and the image generator and feature extractor can be trained based on the network loss.

[0075] In a possible implementation, the generated image may be input into an image discriminator, and the authenticity of the generated image may be discriminated by the image discriminator; and based on the confrontation between the discrimination result of the image discriminator and the image generated by the image generator, the image generator may be trained.

[0076] In one possible implementation, the first comprehensive network loss may be determined based on the difference between the generated image and the first image. For example, a cross entropy loss may be determined based on the difference between corresponding pixels between the generated image and the first image, and the cross entropy loss may be used as the first comprehensive network loss.

[0077] In another possible implementation, if the generated image is a local image block, the second comprehensive network loss may be determined based on the difference between the generated local image block and the local image block of the first image. For example, the cross entropy loss may be determined based on the difference between the corresponding pixels between the generated local image block and the local image block of the first image, and the cross entropy loss may be used as the second comprehensive network loss.

[0078] In a possible implementation, the first comprehensive network loss and the second comprehensive network loss can both be expressed by the following formula (1):

[0079]

[0080] Among them, L r represents the first comprehensive network loss or the second comprehensive network loss, represents a pixel of the generated image or a pixel of the generated local image block, k represents a pixel of the first image or a pixel of the local image block of the first image, express The 1-norm of the difference between the corresponding pixel of and k. In addition, It can also represent content information, style information, etc. of a generated image or a generated partial image block, and k can also represent content information, style information, etc. of a first image or a partial image block of the first image.

[0081] In a possible implementation, the generated image or the generated local image block may be input into an image discriminator for discriminant processing to obtain a discriminant result; wherein the discriminant result may include a first discriminant result corresponding to the generated image, and / or a second discriminant result corresponding to the generated local image block, and the first discriminant result and the second discriminant result may be used to respectively indicate the authenticity of the generated image and the generated local image block. In the example, the first discriminant result and the second discriminant result may be in the form of probabilities, for example, the first discriminant result indicates that the probability that the generated image is a real image is 80%.

[0082] In a possible implementation, the feature extractor, the image generator and the image discriminator can be adversarially trained according to at least one of the first comprehensive network loss and the second comprehensive network loss, and at least one of the first discrimination result and the second discrimination result, that is, the network parameters of the feature extractor, the image generator and the image discriminator are adjusted until the first training condition and the second training condition reach a balance; wherein the first training condition, for example, the first comprehensive network loss and / or the second comprehensive network loss of the feature extractor and the image generator are minimized or less than a set first threshold; the second training condition, for example, the probability that the first discrimination result and / or the second discrimination result output by the image discriminator is a real image is maximized or greater than a set second threshold.

[0083] In a possible implementation, since the first image and the second image are two images of different styles, during the training process, the positions of the two images can be exchanged to train the feature extractor, image generator, and image discriminator. For example, in one training, the first image is image A and the second image is image B. In another training, the first image is image B and the second image is image A. The two trainings can be used as a training group, and the network loss in the two trainings can be determined as the comprehensive network loss of the feature extractor, image generator, and image discriminator, or the average of the network losses in the two trainings can be determined as the comprehensive network loss of the feature extractor, image generator, and image discriminator.

[0084] Based on the above example, in a possible implementation, the comprehensive network loss can be expressed by the following formula (2):

[0085]

[0086] in, An adversarial loss for determining a first discrimination result in the one training; Determine the adversarial loss of the first discrimination result in the another training; Determine the adversarial loss of the second discrimination result in the one training; Determine an adversarial loss for a second discrimination result in the another training; In the training, k represents the pixel of the first image and In the case of pixels representing the generated image, a first integrated network loss is determined; In the another training, k represents the pixel of the first image and In the case of pixels representing the generated image, a first integrated network loss is determined; In the training, k represents the content information of the first image and Determining a first integrated network loss in the presence of content information representing the generated image; In the another training, k represents the content information of the first image and Determining a first integrated network loss in the presence of content information representing the generated image; In the training, k represents the style information of the first image and Determining a first integrated network loss in the case of representing style information of a generated image; In the another training, k represents the style information of the first image and Determining a first integrated network loss in the case of representing style information of a generated image; In the training, k represents the pixel points of the local image block of the first image and A second integrated network loss is determined in the case of pixels representing the generated local image block; In the another training, k represents the pixel point of the local image block of the first image and A second integrated network loss is determined in the case of pixels representing the generated local image block; In the one training, k represents the content information of the local image block of the first image and A second integrated network loss is determined in the case of representing content information of the generated local image block; In the another training, k represents the content information of the local image block of the first image and A second integrated network loss is determined in the case of representing content information of the generated local image block; In the training, k represents the style information of the local image block of the first image and A second integrated network loss determined in the case of style information representing a generated local image patch; In the another training, k represents the style information of the local image block of the first image and Represents the second comprehensive network loss determined when generating style information of a local image block.

[0087] In a possible implementation, the feature extractor, image generator, and image discriminator can be trained adversarially according to the comprehensive network loss determined by the above formula (2). The training can be terminated until the first training condition and the second training condition reach a balance state, and the trained image generator and feature extractor are obtained.

[0088] In a possible implementation, a third image may be generated by a trained image generator. For example, the image generator may decode the content features of the first image (e.g., the content features of the entire first image or the object content features of each local image block of the first image) and the target style features through decoding processing such as upsampling to generate a third image, and the generated third image may have content corresponding to the content features of the first image and a style corresponding to the target style features. In the example, if the first image includes annotations for each object in the image (e.g., category annotations, etc.), since the generated third image has content corresponding to the content features of the first image, the third image may have annotations for each object in the image.

[0089] In a possible implementation, the feature extractor and the image generator may be trained by two consistent images, which may improve the training efficiency and the training effect. The two consistent images indicate that the first image and the second image are images with consistent content.

[0090] Wherein, in the case where the first image and the second image are images with the same content, the second image may be an image obtained by scaling the first image, that is, the second image has the same content and style as the first image, but the resolution is different from the first image. The feature extractor and the image generator can be trained by the first image and the second image, which can improve the universality of the feature extractor and the image generator for resolution, and improve the robustness of the feature extractor and the image generator.

[0091] According to the image generation method of the embodiment of the present disclosure, on the one hand, the feature extractor and the image generator can be used to generate an image consistent with the first image or a local image block of the first image to train the feature extractor and the image generator, which can improve the training efficiency and enhance the training effect. On the other hand, the target style features determined by the overall style features and the object style features of the second image and the content features of the first image can be used to obtain a third image having content corresponding to the content features and a style corresponding to the target style features. When performing style conversion on the image, not only the overall style of the image can be converted, but also the style of the local image block where the object is located in the image can be appropriately converted, so that the overall image and the area where the object is located are coordinated, the realism of the style-transferred image can be improved, and the detection accuracy of the object in the image can be improved. Furthermore, the generated third image has content corresponding to the content features of the first image and a style corresponding to the target style features, and may also have the same annotations as the first image, thereby expanding the richness of image samples having content corresponding to the content features of the first image, especially for difficult image samples (such as images collected in a weather environment that is difficult to encounter, such as extreme weather conditions) or a small number of image samples (such as images collected in an environment with less collection, such as images collected at night), greatly reducing the cost of manual collection and annotation.

[0092] Based on the above embodiments, in a possible implementation, Figure 4 As shown, the method also includes:

[0093] In step S16, the content features of the source image are extracted; the content features of the source image and the target style features are input into the trained image generator to obtain a first target image, wherein the first target image has content corresponding to the content features of the source image and a style corresponding to the target style features.

[0094] In this embodiment, after the feature extractor and the image generator are trained, the content features of the source image can be extracted by the feature extractor, and the style of the source image of any style can be converted into a specific style by the image generator, and a first target image having content corresponding to the content features of the first image and having the target style features can be output.

[0095] Based on the above embodiments, Figure 5 The process of the image generation method according to the embodiment of the present disclosure is shown Figure 4 ,like Figure 5 As shown, the method also includes:

[0096] In step S17, the content features of the source image are extracted; the content features of the source image, the target style features and random noise are input into the trained image generator to obtain a second target image, wherein: the second target image has content corresponding to the content features of the source image, and the second target image has a style corresponding to the target style feature, or the second target image has content corresponding to the content features of the source image and the random noise, and the second target image has a style corresponding to the target style feature, or the second target image has content corresponding to the content features of the source image, and the second target image has a style corresponding to the target style feature and the random noise, or the second target image has content corresponding to the content features of the source image and the random noise, and the second target image has a style corresponding to the target style feature and the random noise, or the second target image has content corresponding to the content features of the source image and the random noise, and the second target image has a style corresponding to the target style feature and the random noise, or the second target image has content corresponding to the content features of the source image, and the second target image has a style corresponding to the random noise, or the second target image has content corresponding to the content features of the source image and the random noise, and the second target image has a style corresponding to the random noise.

[0097] In the example, the random noise may include random content noise and / or random style features extracted from an image with random content or style, or random content noise and / or random style features of an image whose RGB values ​​of each pixel are randomly generated as random values. The random content noise and / or random style features of the above image may be extracted by a feature extractor, and the image style corresponding to the random style features is random. For example, the source image is an image of a certain place taken during the day, and the random noise may be a random style feature extracted from an artificially randomly generated image. For example, one style of random noise may be randomly selected from styles such as night, dusk, cloudy, dawn, rainy, snowy, etc., and at least one of motor vehicles, non-motor vehicles, people, traffic signs, traffic lights, trees, animals, buildings, and obstacles may be randomly selected as content to generate an image for obtaining random noise.

[0098] Exemplarily, the image generator may generate a second target image based on at least one of the content features of the source image, random noise (the random noise may include random style features and / or random content features), and target style features. In the example, the content features of the source image and the random content features of the random noise may be fused to obtain content features corresponding to the source image and the random noise. The target style features and the random style features of the random noise may also be fused to obtain style features corresponding to the target style features and the random noise. The image generator may generate a second target image with content corresponding to the source image and a style corresponding to the target style features based on the above features, or generate a second target image with content corresponding to the source image and random noise and a style corresponding to the target style features, or generate a second target image with content corresponding to the source image and random noise and a style corresponding to the target style features, or generate a second target image with content corresponding to the source image and a style corresponding to the target style features and random noise, or generate a second target image with content corresponding to the source image and random noise and a style corresponding to the target style features and random noise, or generate a second target image with content corresponding to the source image and random noise and a style corresponding to the target style features and random noise, or generate a second target image with content corresponding to the source image and random noise and a style corresponding to the random noise, or generate a second target image with content corresponding to the source image and random noise and a style corresponding to the random noise. This enables more than one style of image to be realized based on a neural network, thereby improving the diversity of content and / or style of the generated images.

[0099] By adopting the technical solution of the embodiment of the present application, style conversion can be performed on images of any style through a feature extractor and an image generator, so that the overall converted image and the area where the target object is located can be coordinated.

[0100] Figures 6A-6C A schematic diagram showing an application of the image generation method according to an embodiment of the present disclosure is shown. Fig. 6A As shown, the styles of the first image and the second image are different. For example, the first image is an image taken at a certain location during the day, and the second image is an image taken at another location at night. The style transformation can be performed on the entire image of the first image and the local image block including the object in the first image to obtain a third image. The third image has the same content as the first image and corresponds to the target style feature determined according to the style feature of the entire image and the style feature of the object. For example, Figure 6B As shown, the first image is an image of a street taken during the day (eg Figure 6B The image shown in the upper left figure in the figure) can determine the local image block (eg Figure 6B The local image block shown in the lower left figure in the figure) is used, and the style of the first image is transformed by the target style features determined by the full-image style features and the object style features to obtain a third image of the night style of the street.

[0101] Specifically, Figure 6CAs shown, the feature extractor can extract the full-image style features of the first image, the object style features of the local image blocks of the first image, the full-image style features of the second image, and the object style features of the style features of the local image blocks of the second image; further, the background style features of the background image blocks in the first image and the background style features of the background image blocks in the second image can also be extracted.

[0102] Specifically, refer to Figure 6C As shown, on the one hand, the content features of the first image can be obtained by a feature extractor. In the example, the first image can be downsampled to obtain a feature map of the first image, and the content features of the feature map can be extracted. The extracted content features may include content features of the entire first image and object content features of local image blocks including objects in the first image.

[0103] On the other hand, the full-image style feature of the second image, the background style feature of the background image block in the second image, and the object style feature of the local image block including the object in the second image can be respectively extracted by the feature extractor. In the example, the second image can be downsampled, and the full-image style feature, background style feature and object style feature can be extracted from the second image after downsampling. Then, the full-image style feature, the object style feature and the background style feature are fused to determine the target style feature.

[0104] Exemplarily, the overall image style feature is fused to the background style feature; the background style feature that has been fused with the overall image style feature is fused to the object style feature to obtain the target style feature.

[0105] Furthermore, the third image may be generated based on the content features of the first image as a whole and / or the object content features of the local image block of the first image including the object, and the above-mentioned target style features. In a possible implementation, the third image may be generated by a trained image generator.

[0106] Exemplarily, the content features of the entire first image may be subjected to residual connection processing with the target style features to obtain feature information having the content features of the entire first image and the target style features, and / or, the object content features of local image blocks including objects in the first image may be subjected to residual connection processing with the target style features to obtain feature information having the object content features of each local image block of the first image and the target style features; the obtained feature information may further be upsampled to obtain an image having content corresponding to the content features of the entire first image and a style corresponding to the target style features, and / or, to obtain local image blocks having content corresponding to the object content features of each local image block of the first image and a style corresponding to the target style features.

[0107] In some possible implementations, the overall style feature of the first image may be fused to the object style feature of the first image to obtain the fused style feature of the first image; or the overall style feature of the first image may be fused to the background style feature of the first image, and then the background style feature that has been fused with the overall style feature may be fused to the object style feature of the first image to obtain the fused style feature of the first image. The implementation method of the fused style feature of the first image may specifically refer to the implementation method of the target style feature in the aforementioned embodiment, and will not be described in detail here.

[0108] Further, the content features of the entire first image and the fusion style features of the first image may be subjected to residual connection processing to obtain feature information having the content features of the entire first image and the fusion style features of the first image, and / or, the object content features of the local image blocks including the object in the first image and the fusion style features of the first image may be subjected to residual connection processing to obtain feature information having the object content features of each local image block of the first image and the fusion style features of the first image; the obtained feature information may be further subjected to upsampling processing to obtain an image having content corresponding to the content features of the entire first image and a style corresponding to the fusion style features of the first image, and / or, to obtain local image blocks having content corresponding to the object content features of each local image block of the first image and a style corresponding to the fusion style features of the first image. The generated image should be completely consistent with the first image, and the generated local image blocks should be completely consistent with the local image blocks of the first image.

[0109] Figure 7 A block diagram showing an image generating device according to an embodiment of the present disclosure Figure 1 ,like Figure 7 As shown, the device comprises:

[0110] A first extraction module 11, used to extract content features of the first image;

[0111] A second extraction module 12 is used to extract the overall style features of a second image and the object style features of a local image block including an object in the second image, respectively, wherein the second image and the first image have different styles;

[0112] A determination module 13, configured to determine a target style feature based at least on the overall image style feature and the object style feature;

[0113] The first generating module 14 is configured to generate a third image according to the content feature and the target style feature, so that the third image has content corresponding to the content feature and a style corresponding to the target style feature.

[0114] In a possible implementation, the determination module 13 is configured to merge the overall image style feature with the object style feature to obtain the target style feature.

[0115] Figure 8 A block diagram showing an image generating device according to an embodiment of the present disclosure Figure 2 ,like Figure 8 As shown, the device also includes:

[0116] A third extraction module 15, configured to extract background style features of a background image block in the second image, wherein the background image block is other image blocks in the second image except the local image block;

[0117] The determination module 13 is used to determine the target style feature according to the overall image style feature, the object style feature and the background style feature.

[0118] In a possible implementation, the determination module 13 is used to merge the overall image style feature into the background style feature; merge the background style feature merged with the overall image style feature into the object style feature to obtain the target style feature.

[0119] In a possible implementation, the first generation module 14 is used to generate an image according to the content features and the target style features through an image generator, and to judge the authenticity of the generated image through an image discriminator; based on the confrontation between the judgment result of the image discriminator and the image generated by the image generator, train the image generator; and after the training, the image generator generates the third image.

[0120] In a possible implementation, the device further includes: a second generating module 16;

[0121] The first extraction module 11 is used to extract content features of the source image;

[0122] The second generating module 16 is used to input the content features of the source image and the target style features into the trained image generator to obtain a first target image, wherein the first target image has content corresponding to the content features of the source image and a style corresponding to the target style features.

[0123] In a possible implementation manner, the first image and / or the second image includes at least one of the following types of objects: motor vehicles, non-motor vehicles, people, traffic signs, traffic lights, trees, animals, buildings, and obstacles.

[0124] In a possible implementation, the second extraction module 12 is used to capture multiple local image blocks in the second image when the second image includes multiple objects, each local image block includes an object; and extract object style features of each local image block respectively.

[0125] In a possible implementation, the second extraction module 12 is configured to fuse object style features extracted from multiple local image blocks of objects of the same category when the multiple objects belong to multiple categories.

[0126] In a possible implementation, the device further includes: a third generating module 17;

[0127] The first extraction module 11 is used to extract content features of the source image;

[0128] The third generating module 17 is used to input the content features of the source image, the target style features and the random noise into the image generator after training to obtain a second target image, wherein: the second target image has content corresponding to the content features of the source image, and the second target image has a style corresponding to the target style feature, or the second target image has content corresponding to the content features of the source image and the random noise, and the second target image has a style corresponding to the target style feature, or the second target image has content corresponding to the content features of the source image, and the second target image has a style corresponding to the target style feature and the random noise, or the second target image has content corresponding to the content features of the source image and the random noise, and the second target image has a style corresponding to the target style feature and the random noise, or the second target image has content corresponding to the content features of the source image, and the second target image has a style corresponding to the target style feature and the random noise, or the second target image has content corresponding to the content features of the source image, and the second target image has a style corresponding to the random noise, or the second target image has content corresponding to the content features of the source image and the random noise, and the second target image has a style corresponding to the random noise.

[0129] It can be understood that the above-mentioned method embodiments and device embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the embodiments of the present disclosure will not be described in detail.

[0130] In addition, the embodiments of the present disclosure also provide an image generating device, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding records in the method part and will not be repeated here.

[0131] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0132] In some embodiments, the functions or modules included in the apparatus provided in the embodiments of the present disclosure may be used to execute the method described in the above method embodiment. The specific implementation thereof may refer to the description of the above method embodiment. For the sake of brevity, it will not be repeated here.

[0133] The embodiment of the present disclosure also provides a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above method of the embodiment of the present disclosure is implemented. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

[0134] The embodiment of the present disclosure also provides an electronic device, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to perform the above method of the embodiment of the present disclosure. The electronic device can be provided as a terminal, a server or other forms of equipment.

[0135] Fig. 9 8 is a schematic diagram of the structure of an electronic device according to an exemplary embodiment. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, or a personal digital assistant.

[0136] Reference Fig. 9 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0137] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0138] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0139] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 800.

[0140] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0141] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0142] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0143] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the electronic device 800, the relative positioning of the components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include a light sensor, such as a metal oxide semiconductor element (Complementary Metal-Oxide Semiconductor, CMOS) or an image sensor or a charge coupled device (Charge Coupled Device, CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0144] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology and other technologies.

[0145] In an exemplary embodiment, the electronic device 800 can be implemented by one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), controllers, microcontrollers, microprocessors or other electronic components to execute the above method.

[0146] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions, which can be executed by a processor 820 of an electronic device 800 to perform the above method.

[0147] Fig.10 1 is a schematic diagram of an electronic device structure according to an exemplary embodiment. For example, the electronic device 1900 can be provided as a server. Fig.10 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0148] The electronic device 1900 may further include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or the like.

[0149] The embodiments of the present disclosure also provide a non-volatile computer-readable storage medium, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to complete the above method.

[0150] The embodiments of the present disclosure may be systems, methods and / or computer program products. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0151] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a protruding structure in a groove on which instructions are stored, and any suitable combination thereof. The computer-readable storage medium used herein is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., light pulses through a fiber optic cable), or an electrical signal transmitted through wires.

[0152] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0153] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0154] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0155] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0156] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0157] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0158] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for generating an image, comprising: extracting content features of the first image; Respectively extracting a full-image style feature of a second image and an object style feature of a local image block including an object in the second image, wherein the second image and the first image have different styles; Determining a target style feature at least according to the overall image style feature and the object style feature; generating a third image according to the content feature and the target style feature, so that the third image has content corresponding to the content feature and a style corresponding to the target style feature; The method further comprises: extracting background style features of a background image block in the second image, wherein the background image block is other image blocks in the second image except the local image block; The step of determining a target style feature at least based on the overall image style feature and the object style feature comprises: Performing a residual connection on the full-image style feature and the background style feature to fuse the full-image style feature into the background style feature; The target style feature is obtained by performing a residual connection on the background style feature that has been fused with the full image style feature and the object style feature.

2. The method according to claim 1, wherein: The determining the target style feature according to the full image style feature and the object style feature further includes: The full-image style feature is fused with the object style feature to obtain the target style feature.

3. The method according to claim 1 or 2, wherein: The step of generating a third image according to the content feature and the target style feature so that the third image has content corresponding to the content feature and a style corresponding to the target style feature includes: An image generator generates an image according to the content features and the target style features, and an image discriminator discriminates the authenticity of the generated image; Training the image generator based on the confrontation between the discrimination result of the image discriminator and the image generated by the image generator; The image generator after training generates the third image.

4. The method according to claim 3, wherein: The method further comprises: Extracting content features of source images; The content features of the source image and the target style features are input into the trained image generator to obtain a first target image, wherein the first target image has content corresponding to the content features of the source image and a style corresponding to the target style features.

5. The method according to claim 1, wherein: The first image and / or the second image includes at least one of the following types of objects: motor vehicles, non-motor vehicles, people, traffic signs, traffic lights, trees, animals, buildings, and obstacles.

6. The method according to claim 5, wherein: When the second image includes a plurality of objects, extracting object style features of a local image block including the object in the second image includes: intercepting a plurality of local image blocks in the second image, each local image block including an object; The object style features of each local image block are extracted respectively.

7. The method according to claim 6, wherein: In a case where the multiple objects belong to multiple categories, extracting object style features of local image blocks including the objects in the second image further includes: The object style features extracted from the multiple local image blocks of the same category of objects are fused.

8. The method according to claim 4, wherein: The method further comprises: Extracting content features of source images; The content features of the source image, the target style features and random noise are input into the trained image generator to obtain a second target image, wherein: the second target image has content corresponding to the content features of the source image, and the second target image has a style corresponding to the target style feature, or the second target image has content corresponding to the content features of the source image and the random noise, and the second target image has a style corresponding to the target style feature, or the second target image has content corresponding to the content features of the source image, and the second target image has a style corresponding to the target style feature and the random noise, or the second target image has content corresponding to the content features of the source image and the random noise, and the second target image has a style corresponding to the target style feature and the random noise, or the second target image has content corresponding to the content features of the source image, and the second target image has a style corresponding to the random noise, or the second target image has content corresponding to the content features of the source image, and the second target image has a style corresponding to the random noise, or the second target image has content corresponding to the content features of the source image and the random noise, and the second target image has a style corresponding to the random noise.

9. An image generating device, comprising: A first extraction module, used to extract content features of the first image; A second extraction module is used to extract the overall style features of a second image and the object style features of a local image block including an object in the second image, wherein the second image and the first image have different styles; A determination module, configured to determine a target style feature based at least on the overall image style feature and the object style feature; A first generating module, configured to generate a third image according to the content feature and the target style feature, so that the third image has content corresponding to the content feature and a style corresponding to the target style feature; The third extraction module is used to extract the background style features of the background image blocks in the second image, wherein the background image blocks are other image blocks in the second image except the local image blocks; the determination module is used to perform residual connection on the full-image style features and the background style features to fuse the full-image style features into the background style features; and perform residual connection on the background style features that have been fused with the full-image style features and the object style features to obtain the target style features.

10. The device according to claim 9, wherein: The determination module is used to merge the full-image style feature with the object style feature to obtain the target style feature.

11. The device according to claim 9 or 10, wherein: The first generation module is used to generate an image according to the content feature and the target style feature through an image generator, and to judge the authenticity of the generated image through an image discriminator; based on the confrontation between the discrimination result of the image discriminator and the image generated by the image generator, train the image generator; The image generator after training generates the third image.

12. The device according to claim 11, wherein The device further comprises: a second generating module; The first extraction module is used to extract content features of the source image; The second generation module is used to input the content features of the source image and the target style features into the trained image generator to obtain a first target image, wherein the first target image has content corresponding to the content features of the source image and a style corresponding to the target style features.

13. The device according to claim 9, wherein: The first image and / or the second image includes at least one of the following types of objects: motor vehicles, non-motor vehicles, people, traffic signs, traffic lights, trees, animals, buildings, and obstacles.

14. The device according to claim 13, wherein: The second extraction module is used to intercept multiple local image blocks in the second image when the second image includes multiple objects, each local image block includes an object; and extract the object style features of each local image block respectively.

15. The device according to claim 14, wherein: The second extraction module is used to fuse the object style features extracted from the multiple local image blocks of objects of the same category when the multiple objects belong to multiple categories.

16. The device according to claim 12, wherein: The device further comprises: a third generating module; The first extraction module is used to extract content features of the source image; The third generating module is used to input the content features of the source image, the target style features and the random noise into the image generator after training to obtain a second target image, wherein: the second target image has content corresponding to the content features of the source image, and the second target image has a style corresponding to the target style feature, or the second target image has content corresponding to the content features of the source image and the random noise, and the second target image has a style corresponding to the target style feature, or the second target image has content corresponding to the content features of the source image, and the second target image has a style corresponding to the target style feature and the random noise, or the second target image has content corresponding to the content features of the source image and the random noise, and the second target image has a style corresponding to the target style feature and the random noise, or the second target image has content corresponding to the content features of the source image, and the second target image has a style corresponding to the target style feature and the random noise, or the second target image has content corresponding to the content features of the source image, and the second target image has a style corresponding to the random noise, or the second target image has content corresponding to the content features of the source image and the random noise, and the second target image has a style corresponding to the random noise.

17. An electronic device comprising: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to: execute the method according to any one of claims 1 to 8.

18. A computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions implement the method according to any one of claims 1 to 8 when executed by a processor.

Images