An image generation method, device, computer equipment and storage medium

Abstract

Embodiments of the present application disclose an image generation method and device, computer equipment and a storage medium; an image generation instruction can be received, the image generation instruction including image attribute information of a target image to be generated, environment information is obtained, the environment information is collected through a display device displaying the target image, data mapping is performed on the environment information through a preset image generation model, a content image corresponding to the environment information is generated, style features are extracted from at least one preset style reference image, content features are extracted from the content image, and the target image is obtained based on the fusion of the content features and the style features; in the embodiments of the present application, the image generation model can map the environment information to obtain the content image, the content features in the content image can be fused with the style features in the style reference image, and finally the target image is generated, so that a wallpaper image with rich content and various styles can be generated, and the visual experience of a user is improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to an image generation method, apparatus, computer device, and storage medium. Background Technology

[0002] With the rapid development of the current economy, various display devices are being used more and more widely in people's lives. Generally, in order to enrich the user's visual experience, display devices can default to displaying wallpaper images with different content and styles.

[0003] Currently, the main method for displaying wallpaper images is for technicians to pre-set a set of wallpaper images, and then randomly select one from the set when needed. However, this approach cannot guarantee that the content of the displayed wallpaper image will fit the current environment, and the number of wallpaper images is also limited, potentially leading to inconsistent image quality and affecting the user's visual experience. Summary of the Invention

[0004] This invention provides an image generation method, apparatus, computer device, and storage medium that can generate wallpaper images with rich content and diverse styles, making the content of the wallpaper images fit the real environment and enhancing the user's visual experience.

[0005] This invention provides an image generation method, comprising:

[0006] Receive an image generation instruction, the image generation instruction including image attribute information of the target image to be generated;

[0007] Environmental information is acquired, which is collected by a display device that displays the target image;

[0008] The environmental information is mapped using a preset image generation model to generate a content image corresponding to the environmental information.

[0009] Style features are extracted from at least one preset style reference image, and content features are extracted from the content image. Based on the fusion of the content features and the style features, a target image is obtained.

[0010] Accordingly, embodiments of the present invention also provide an image generation apparatus, comprising:

[0011] The instruction receiving unit is used to receive an image generation instruction, which includes image attribute information of the target image to be generated.

[0012] An environmental information acquisition unit is used to acquire environmental information, which is collected by a display device that displays the target image;

[0013] The content image generation unit is used to perform data mapping on the environmental information through a preset image generation model to generate a content image corresponding to the environmental information.

[0014] The target image generation unit is used to extract style features from at least one preset style reference image and extract content features from the content image, and obtain a target image based on the fusion of the content features and the style features.

[0015] Optionally, the image generation apparatus provided in this embodiment of the invention further includes an image generation model training unit, used to obtain a generative adversarial model to be trained, wherein the generative adversarial model includes an image generation model to be trained and a discriminative model to be trained, and the generative adversarial model is provided with at least one real content sample image.

[0016] The training input parameters are mapped using the image generation model to be trained, and the training content image corresponding to the training input parameters is generated.

[0017] The discriminant model to be trained is used to calculate the realism parameter of the training content image relative to the real content sample image;

[0018] The image generation model and the discrimination model to be trained are adjusted based on the realism parameters.

[0019] Return to the step of mapping the training input parameters to the image generation model to be trained, until the preset training termination condition is met, and obtain the trained image generation model.

[0020] Optionally, the image generation apparatus provided in this embodiment of the invention further includes an image generation model setting unit, used to obtain the display parameters of the display device;

[0021] Based on the display parameters, the model parameters of the image generation model to be trained are set to obtain a new image generation model to be trained.

[0022] Optionally, the target image generation unit is used to extract style features from at least one preset style reference image according to the style feature mapping parameters of the style extraction layer in the style transfer model, so as to obtain the style features corresponding to the style reference image.

[0023] Based on the content feature mapping parameters of the content extraction layer in the style transfer model, content features are extracted from the content image to obtain the content features corresponding to the content image.

[0024] Optionally, the image generation apparatus provided in this embodiment of the invention further includes a style transfer model training unit, used to extract style features from a sample style reference image through a style transfer model to be trained, so as to obtain sample style features corresponding to the sample style reference image.

[0025] The style transfer model to be trained is used to extract content features from the sample content image to obtain the sample content features corresponding to the sample content image.

[0026] The sample style features and sample content features are fused to obtain a sample style transfer image;

[0027] Based on the sample style transfer image, the sample style reference image, and the sample content image, calculate the model loss of the style transfer model to be trained;

[0028] Based on the model loss, the model parameters of the style transfer model to be trained are adjusted to obtain the trained style transfer model.

[0029] Optionally, the environmental information acquisition unit is used to collect ambient sound audio through a display device;

[0030] The ambient sound audio is converted into the model input format of the image generation model to obtain environmental information.

[0031] Optionally, the target image generation unit is used to select at least one target style reference image corresponding to the image style type from at least one preset style reference image;

[0032] Style features are extracted from each of the target style reference images, and content features are extracted from the content images;

[0033] The content features and style features are weighted and fused to obtain the target image.

[0034] Accordingly, embodiments of the present invention also provide a computer device, including a memory and a processor; the memory stores an application program, and the processor is used to run the application program in the memory to perform the steps in any of the image generation methods provided in the embodiments of the present invention.

[0035] Accordingly, embodiments of the present invention also provide a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the image generation methods provided in the embodiments of the present invention.

[0036] Furthermore, embodiments of the present invention also provide a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps in any of the image generation methods provided in embodiments of the present invention.

[0037] The solution of this invention can receive an image generation instruction, which includes image attribute information of the target image to be generated, obtain environmental information collected by a display device displaying the target image, perform data mapping on the environmental information through a preset image generation model to generate a content image corresponding to the environmental information, extract style features from at least one preset style reference image, and extract content features from the content image. Based on the fusion of the content features and the style features, a target image is obtained. Since the image generation model can map the content image according to the environmental information in this embodiment, it can ensure that the content of the image fits the environment. The content features in the content image can be fused with the style features in the style reference image to finally generate the target image. Therefore, it can generate wallpaper images with rich content and diverse styles, improving the user's visual experience. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a scene diagram illustrating the image generation method provided in an embodiment of the present invention;

[0040] Figure 2 This is a flowchart of the image generation method provided in the embodiments of the present invention;

[0041] Figure 3 This is a schematic diagram illustrating the technical implementation of generating target images provided in an embodiment of the present invention;

[0042] Figure 4 This is a schematic diagram illustrating the application of the present invention on a smart TV according to an embodiment of the invention;

[0043] Figure 5 This is a schematic diagram of the structure of the image generation device provided in an embodiment of the present invention;

[0044] Figure 6 This is another structural schematic diagram of the image generation device provided in an embodiment of the present invention;

[0045] Figure 7This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] This invention provides an image generation method, apparatus, computer device, and computer-readable storage medium. Specifically, this invention provides an image generation method suitable for an image generation apparatus that can be integrated into a computer device.

[0048] The computer device can be a terminal or other device, including but not limited to mobile terminals and fixed terminals. For example, mobile terminals include but are not limited to smartphones, smartwatches, tablets, laptops, smart vehicles, etc., while fixed terminals include but are not limited to desktop computers, smart TVs, etc.

[0049] The computer device can also be a server or other similar device. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, but it is not limited to these.

[0050] The image generation method of this invention can be implemented by a server, or by a terminal and a server together.

[0051] The following example illustrates this image generation method using both a terminal and a server.

[0052] like Figure 1 As shown, the image generation system provided in this embodiment of the invention includes a terminal 10 and a server 20, etc. The terminal 10 and the server 20 are connected via a network, such as a wired or wireless network. The terminal 10 can function as a terminal that displays the target image and sends image generation instructions to the server 20.

[0053] Terminal 10 can be a terminal for users to view the target image, and is used to send image generation instructions for the target image to server 20.

[0054] Server 20 can be used to receive image generation instructions, which include image attribute information of the target image to be generated, obtain environmental information, which is collected by a display device displaying the target image, perform data mapping on the environmental information through a preset image generation model, generate a content image corresponding to the environmental information, extract style features from at least one preset style reference image, and extract content features from the content image, and obtain the target image based on the fusion of the content features and the style features.

[0055] Server 20 can send the target image to terminal 10 for display.

[0056] In some optional embodiments, the target image generation step performed by server 20 may also be performed by terminal 10, and this embodiment of the present invention does not limit this.

[0057] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the preferred order of the embodiments.

[0058] The embodiments of the present invention will be described from the perspective of an image generation device, which can be integrated into a server and / or a terminal.

[0059] like Figure 2 As shown, the specific process of the image generation method in this embodiment can be as follows:

[0060] 201. Receive an image generation instruction, wherein the image generation instruction includes image attribute information of the target image to be generated.

[0061] Specifically, an image generation instruction is an instruction that directs the generation of a target image. Image generation instructions can be generated based on user actions or actively generated by the display device.

[0062] For example, image generation instructions can be generated based on user actions such as powering on or returning to the desktop; or, image generation instructions can be generated periodically or intermittently during the operation of the display device, and so on.

[0063] Image attribute information refers to information related to the target image itself. For example, image attribute information may include, but is not limited to, the style type, resolution, and category of image content of the target image.

[0064] 202. Obtain environmental information, which is collected by a display device that displays the target image.

[0065] Specifically, environmental information can be information collected by the display device about its surrounding environment. For example, environmental information may include, but is not limited to, information related to ambient brightness, ambient sound, ambient temperature, and ambient weather.

[0066] Taking environmental information, including environmental sound-related information, as an example, the step "Obtain Environmental Information" can specifically include:

[0067] Collect ambient sound audio through a display device;

[0068] The ambient sound audio is converted into the model input format of the image generation model to obtain environmental information.

[0069] For example, ambient sound audio can be obtained by receiving audio signals through a microphone device on a smart TV; the audio information (ambient sound audio) can be preprocessed to obtain sound waves from the audio, and then numerical values ​​such as amplitude and pulse can be extracted from the sound waves and converted into the model input format of the image generation model, such as the format of a one-dimensional array, as environmental information to change the image content in the target image.

[0070] The audio information is converted into a one-dimensional numerical value as input. The generator network generates a unique real image that is indistinguishable from the real image. The style transfer network stylizes it into an artistic abstract image and places the stylized 16:9 artistic abstract image on the interface for full-screen display. The system receives sound data in multiple threads, continuously generates wallpapers, and then continuously refreshes the display.

[0071] It is understandable that environmental information may also include other information besides sound. For example, the step "obtaining environmental information" may specifically include:

[0072] Ambient sound audio is collected through a display device to obtain ambient sound information;

[0073] Obtain the environmental weather information corresponding to the display device;

[0074] The environmental sound information and the environmental weather information are fused to generate fused environmental information;

[0075] The fused environmental information is converted into the model input format of the image generation model to obtain the environmental information.

[0076] When integrating ambient sound and weather information, a weighted calculation method can be used. The weights for ambient sound and weather information can be the same or different, and technicians and users can set them according to actual display needs.

[0077] 203. The environmental information is mapped using a preset image generation model to generate a content image corresponding to the environmental information.

[0078] In this embodiment of the invention, the image generation model is a generative model within GAN (Generative Adversarial Network). As a deep learning technique, GAN utilizes the game-like learning between the generative and discriminative models to produce images that are virtually indistinguishable from real photographs.

[0079] However, GANs have problems such as high requirements for the quality and quantity of training data, difficulty in training and instability, and easy corruption of details. Artistic images often have abstract image content, a wide variety of image content, and a unique style for each image. The amount of data is limited and the quality varies. Directly training artistic images results in poor quality.

[0080] Therefore, in this embodiment of the invention, using images of real content rather than artistic images to train the GAN ensures the authenticity of the image content in the target image and avoids problems such as abstract image content. Furthermore, the quality of real images is easier to control and obtain than artistic images, which can improve the image quality of the target image.

[0081] To improve the image generation performance of the image generation model, its parameters can be adjusted through a pre-training process. That is, before the step "data mapping of the environmental information using a preset image generation model to generate the content image corresponding to the environmental information," the image generation method provided in this embodiment of the invention may further include:

[0082] Obtain a generative adversarial model to be trained, the generative adversarial model including an image generation model to be trained and a discriminative model to be trained, the generative adversarial model being set with at least one real content sample image;

[0083] The training input parameters are mapped using the image generation model to be trained, and the training content image corresponding to the training input parameters is generated.

[0084] The discriminant model to be trained is used to calculate the realism parameter of the training content image relative to the real content sample image;

[0085] The image generation model and the discrimination model to be trained are adjusted based on the realism parameters.

[0086] Return to the step of mapping the training input parameters to the image generation model to be trained, until the preset training termination condition is met, and obtain the trained image generation model.

[0087] The image content in the real-world content sample images, such as flowers, plants, and people, all exist in the real environment. This embodiment of the invention does not limit the type of content in the real-world content sample images.

[0088] Specifically, the realism parameter describes the degree of realism of the image content in the training content image. The realism can be determined by the similarity between the training content image and the real content sample image. For example, a discriminative model can score the training content image and output a confidence score between 0 and 1 for the corresponding training content image.

[0089] In actual training, the optimization of the image generation model and the discriminator model does not have to be alternated one after another. Instead, the image generation model is trained once after every k training iterations of the discriminator model. This ensures that the changes in the image generation model are slow enough so that the discriminator model always stays near its optimal solution.

[0090] The training termination condition could be that the number of times the image generation model is adjusted reaches a preset adjustment threshold, or that the realism parameter output by the model is greater than a preset threshold for N consecutive times, etc.

[0091] During the pre-training of GAN networks, such as Figure 3 As shown, a large number of high-quality resolution (720p, 1080p, 2k) real images can be collected as training sets. The image content is grand and beautiful, such as landscape pictures. Select a GAN network (such as the representative StyleGAN[1], BigGAN[2]). The GAN network defaults to random one-dimensional input. Train this GAN network, update the parameters of the model, and the network continuously extracts the detailed features of the image content until it generates an image that is almost indistinguishable from the actual image during the test.

[0092] It is understood that the display parameters of different display devices may differ, and the model parameters of the image generation model can be adjusted in advance according to the display parameters. That is, before the step "data mapping of the training input parameters through the image generation model to be trained to generate the training content image corresponding to the training input parameters", the image generation method provided in this embodiment of the invention may further include:

[0093] Obtain the display parameters of the display device;

[0094] Based on the display parameters, the model parameters of the image generation model to be trained are set to obtain a new image generation model to be trained.

[0095] For example, the convolution kernel size in the model parameters of the image generation model to be trained can be 4*4, generating images with a 1:1 aspect ratio. However, the display parameters of the display device may require a 16:9 wallpaper image, in which case the convolution kernel size could be changed to 3:5, and so on.

[0096] like Figure 4 As shown, the image generation model can be set on terminals such as smart TVs. The smart TV can generate target images in memory based on environmental perturbations (i.e., environmental information) and image generation models (GANs) to serve as the smart TV's art wallpaper.

[0097] 204. Extract style features from at least one preset style reference image and extract content features from the content image, and obtain a target image based on the fusion of the content features and the style features.

[0098] Style can be understood as the subjective feeling that an image evokes in the viewer. Style can be reflected in the color, texture, and other elements of an image. For example, images created by different people generally have different personal characteristics, and these personal characteristics, as expressed in the image, constitute style.

[0099] Optionally, style transfer from the style reference image can be achieved using deep learning techniques. For example, a style transfer model can be built using deep learning to extract style features and content features, and then feature fusion can be performed to achieve style transfer. The step "extracting style features from at least one preset style reference image and extracting content features from the content image" can specifically include:

[0100] Based on the style feature mapping parameters of the style extraction layer in the style transfer model, style features are extracted from at least one preset style reference image to obtain the style features corresponding to the style reference image.

[0101] Based on the content feature mapping parameters of the content extraction layer in the style transfer model, content features are extracted from the content image to obtain the content features corresponding to the content image.

[0102] For example, the style extraction layer and content extraction layer in a style transfer model can be models based on one of the machine learning networks such as CNN, DNN, and GAN.

[0103] Style transfer models are network structures that can extract features from images. For example, style transfer models can include convolutional layers, which can extract image features through convolution operations.

[0104] In some optional embodiments, the style transfer model is obtained through pre-training. The pre-training process allows for adjustments to the parameters of the style transfer model, enabling it to achieve better style transfer performance. Before the step "based on the style feature mapping parameters of the style extraction layer in the style transfer model, extracting style features from at least one preset style reference image to obtain the style features corresponding to the style reference image," the image generation method provided in this embodiment may further include:

[0105] The style transfer model to be trained is used to extract style features from the sample style reference image to obtain the sample style features corresponding to the sample style reference image.

[0106] The style transfer model to be trained is used to extract content features from the sample content image to obtain the sample content features corresponding to the sample content image.

[0107] The sample style features and sample content features are fused to obtain a sample style transfer image;

[0108] Based on the sample style transfer image, the sample style reference image, and the sample content image, calculate the model loss of the style transfer model to be trained;

[0109] Based on the model loss, the model parameters of the style transfer model to be trained are adjusted to obtain the trained style transfer model.

[0110] The sample style reference image can be an image with any style, and the sample content image can be an image with any content.

[0111] Specifically, the model parameters may include the number of feature extraction layers used to extract feature information in the style transfer model, the number of input channels of the feature extraction layers, and so on.

[0112] For example, when pre-training a style transfer network, any image in the style of Van Gogh or Monet can be selected as a style image for training. The training set can be a large real image set of ImageNet. A style transfer network (such as Fast styletransfer[3]) can be selected for training, and the model parameters can be updated. The trained model integrates the feature weights extracted from the image content extraction and the feature weights extracted from the style image. During testing, the stylized image retains both the content details of the real image and the style details of the style image.

[0113] In some optional examples, the model loss can be calculated based on the style loss between the sample style transfer image and the sample style reference image, and the content loss between the sample content image and the sample style transfer image. That is, the step "calculating the model loss of the style transfer model to be trained based on the sample style transfer image, the sample style reference image, and the sample content image" can specifically include:

[0114] Calculate the style similarity between the sample style transfer image and the sample style reference image, and use it as the style loss of the style transfer model to be trained;

[0115] Calculate the content similarity between the sample style transfer image and the sample content image, and calculate the content loss of the style transfer model to be trained.

[0116] The model loss of the style transfer model to be trained is calculated based on style loss and content loss.

[0117] Specifically, the model loss can be calculated by weighting style loss and content loss, and the weights of style loss and content loss can be set by the technical staff.

[0118] In practical applications, style transfer models of various style types can be trained for later use. Optionally, image attribute information includes image style type. The step "extracting style features from at least one preset style reference image and extracting content features from the content image, and obtaining the target image based on the fusion of the content features and the style features" can specifically include:

[0119] From a preset set of at least one style reference image, select at least one target style reference image corresponding to the style type of the image;

[0120] Style features are extracted from each of the target style reference images, and content features are extracted from the content images;

[0121] The content features and style features are weighted and fused to obtain the target image.

[0122] The weights corresponding to content features and style features can be the same or different, and this embodiment of the invention does not limit this.

[0123] As can be seen from the above, the embodiments of the present invention can receive image generation instructions, which include image attribute information of the target image to be generated, obtain environmental information, the environmental information being collected by a display device displaying the target image, perform data mapping on the environmental information through a preset image generation model to generate a content image corresponding to the environmental information, extract style features from at least one preset style reference image, and extract content features from the content image, and obtain the target image based on the fusion of the content features and the style features. Since in the embodiments of the present invention, the image generation model can map the content image according to the environmental information to obtain the content image, it can ensure that the content of the image fits the environment, and the content features in the content image can be fused with the style features in the style reference image to finally generate the target image. Therefore, it can generate wallpaper images with rich content and diverse styles, improving the user's visual experience.

[0124] To better implement the above methods, the present invention also provides an image generation apparatus.

[0125] refer to Figure 5 The device may include:

[0126] The instruction receiving unit 501 can be used to receive an image generation instruction, which may include image attribute information of the target image to be generated.

[0127] The environmental information acquisition unit 502 can be used to acquire environmental information, which is collected by a display device that displays the target image;

[0128] The content image generation unit 503 can be used to perform data mapping on the environmental information through a preset image generation model to generate a content image corresponding to the environmental information.

[0129] The target image generation unit 504 can be used to extract style features from at least one preset style reference image and extract content features from the content image, and obtain a target image based on the fusion of the content features and the style features.

[0130] In some alternative embodiments, such as Figure 6 As shown, the image generation apparatus provided in this embodiment of the invention may further include an image generation model training unit 505, which can be used to acquire a generative adversarial model to be trained. The generative adversarial model may include an image generation model to be trained and a discriminative model to be trained. The generative adversarial model is provided with at least one real content sample image.

[0131] The training input parameters are mapped using the image generation model to be trained, and the training content image corresponding to the training input parameters is generated.

[0132] The discriminant model to be trained is used to calculate the realism parameter of the training content image relative to the real content sample image;

[0133] The image generation model and the discrimination model to be trained are adjusted based on the realism parameters.

[0134] Return to the step of mapping the training input parameters to the image generation model to be trained, until the preset training termination condition is met, and obtain the trained image generation model.

[0135] In some alternative embodiments, such as Figure 6 As shown, the image generation apparatus provided in this embodiment of the invention may further include an image generation model setting unit 506, which can be used to obtain the display parameters of the display device;

[0136] Based on the display parameters, the model parameters of the image generation model to be trained are set to obtain a new image generation model to be trained.

[0137] In some optional embodiments, the target image generation unit 504 can be used to extract style features from at least one preset style reference image according to the style feature mapping parameters of the style extraction layer in the style transfer model, so as to obtain the style features corresponding to the style reference image.

[0138] Based on the content feature mapping parameters of the content extraction layer in the style transfer model, content features are extracted from the content image to obtain the content features corresponding to the content image.

[0139] In some optional embodiments, the image generation apparatus provided in this embodiment of the invention may further include a style transfer model training unit 507, which can be used to extract style features from a sample style reference image through a style transfer model to be trained, so as to obtain the sample style features corresponding to the sample style reference image.

[0140] The style transfer model to be trained is used to extract content features from the sample content image to obtain the sample content features corresponding to the sample content image.

[0141] The sample style features and sample content features are fused to obtain a sample style transfer image;

[0142] Based on the sample style transfer image, the sample style reference image, and the sample content image, calculate the model loss of the style transfer model to be trained;

[0143] Based on the model loss, the model parameters of the style transfer model to be trained are adjusted to obtain the trained style transfer model.

[0144] In some optional embodiments, the environmental information acquisition unit can be used to collect ambient sound audio through a display device;

[0145] The ambient sound audio is converted into the model input format of the image generation model to obtain environmental information.

[0146] In some optional embodiments, the target image generation unit can be used to select at least one target style reference image corresponding to the image style type from at least one preset style reference image;

[0147] Style features are extracted from each of the target style reference images, and content features are extracted from the content images;

[0148] The content features and style features are weighted and fused to obtain the target image.

[0149] As can be seen from the above, the image generation device can receive image generation instructions, which include image attribute information of the target image to be generated, obtain environmental information (collected by a display device displaying the target image), perform data mapping on the environmental information using a preset image generation model, generate a content image corresponding to the environmental information, extract style features from at least one preset style reference image, and extract content features from the content image. Based on the fusion of the content features and the style features, the target image is obtained. Since in this embodiment of the invention, the image generation model can map the content image according to the environmental information, it can ensure that the content of the image fits the environment. The content features in the content image can be fused with the style features in the style reference image to finally generate the target image. Therefore, wallpaper images with rich content and diverse styles can be generated, enhancing the user's visual experience.

[0150] Furthermore, embodiments of the present invention also provide a computer device, which may be a terminal or a server, etc. Figure 7 As shown, it illustrates a structural schematic diagram of a computer device involved in an embodiment of the present invention, specifically:

[0151] The computer device may include a radio frequency (RF) circuit 701, a memory 702 including one or more computer-readable storage media, an input unit 703, a display unit 704, a sensor 705, an audio circuit 706, a wireless Fidelity (WiFi) module 707, a processor 708 including one or more processing cores, and a power supply 709, etc. Those skilled in the art will understand that... Figure 7 The computer device structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0152] RF circuit 701 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and hands it over to one or more processors 708 for processing; additionally, it transmits uplink data to the base station. Typically, RF circuit 701 includes, but is not limited to, an antenna, at least one amplifier, a tuner, one or more oscillators, a Subscriber Identity Module (SIM) card, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc. Furthermore, RF circuit 701 can also communicate wirelessly with networks and other devices. Wireless communication can use any communication standard or protocol, including but not limited to GSM, GPRS, CDMA, WCDMA, LTE, email, and SMS.

[0153] The memory 702 can be used to store software programs and modules. The processor 708 executes various functional applications and data processing by running the software programs and modules stored in the memory 702. The memory 702 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device (such as audio data, telephone directory, etc.). In addition, the memory 702 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 702 may also include a memory controller to provide access to the memory 702 for the processor 708 and the input unit 703.

[0154] The input unit 703 can be used to receive input digital or character information, and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control. Specifically, in one embodiment, the input unit 703 may include a touch-sensitive surface and other input devices. The touch-sensitive surface, also known as a touch display or touchpad, can collect user touch operations on or near it (e.g., user operations using fingers, styluses, or any suitable object or accessory on or near the touch-sensitive surface), and drive corresponding connection devices according to a pre-set program. Optionally, the touch-sensitive surface may include a touch detection device and a touch controller. The touch detection device detects the user's touch orientation and the signal generated by the touch operation, transmitting the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, sends it to the processor 708, and can receive and execute commands from the processor 708. Furthermore, various types of touch-sensitive surfaces, such as resistive, capacitive, infrared, and surface acoustic wave, can be used. In addition to the touch-sensitive surface, the input unit 703 may also include other input devices. Specifically, other input devices may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0155] Display unit 704 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of computer devices. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. Display unit 704 may include a display panel, optionally configured as a liquid crystal display (LCD), organic light-emitting diode (OLED), or similar device. Furthermore, a touch-sensitive surface may cover the display panel. When the touch-sensitive surface detects a touch operation on or near it, it transmits the information to processor 708 to determine the type of touch event. Subsequently, processor 708 provides corresponding visual output on the display panel based on the type of touch event. Although in Figure 7 In this context, the touch-sensitive surface and the display panel are two separate components for implementing input and output functions. However, in some embodiments, the touch-sensitive surface and the display panel can be integrated to achieve both input and output functions.

[0156] The computer device may also include at least one sensor 705, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel according to the ambient light level, and the proximity sensor can turn off the display panel and / or backlight when the computer device is moved to the ear. As a type of motion sensor, a gravity acceleration sensor can detect the magnitude of acceleration in various directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that recognize the phone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometers, taps), etc. Other sensors that may be configured in the computer device, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0157] Audio circuitry 706, a speaker, and a microphone provide an audio interface between the user and the computer device. Audio circuitry 706 converts received audio data into electrical signals, transmits them to the speaker, and the speaker converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by audio circuitry 706, converted back into audio data, and output to processor 708 for processing. The audio data is then transmitted via RF circuitry 701 to, for example, another computer device, or output to memory 702 for further processing. Audio circuitry 706 may also include an earphone jack to facilitate communication between peripheral headphones and the computer device.

[0158] WiFi is a short-range wireless transmission technology. Computer devices using a WiFi module 707 can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access. Although Figure 7 WiFi module 707 is shown, but it is understood that it is not an essential component of computer equipment and can be omitted as needed without changing the nature of the invention.

[0159] The processor 708 is the control center of the computer device, connecting various parts of the mobile phone through various interfaces and lines. It executes various functions of the computer device and processes data by running or executing software programs and / or modules stored in the memory 702, and by calling data stored in the memory 702. Optionally, the processor 708 may include one or more processing cores; preferably, the processor 708 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the aforementioned modem processor may also not be integrated into the processor 708.

[0160] The computer device also includes a power supply 709 (such as a battery) that supplies power to various components. Preferably, the power supply can be logically connected to the processor 708 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 709 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0161] Although not shown, the computer device may also include a camera, Bluetooth module, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 708 in the computer device loads the executable files corresponding to the processes of one or more applications into the memory 702 according to the following instructions, and the processor 708 runs the applications stored in the memory 702 to realize various functions, as follows:

[0162] Receive an image generation instruction, the image generation instruction including image attribute information of the target image to be generated;

[0163] Environmental information is acquired, which is collected by a display device that displays the target image;

[0164] The environmental information is mapped using a preset image generation model to generate a content image corresponding to the environmental information.

[0165] Style features are extracted from at least one preset style reference image, and content features are extracted from the content image. Based on the fusion of the content features and the style features, a target image is obtained.

[0166] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0167] To this end, embodiments of the present invention provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the image generation methods provided in the embodiments of the present invention. For example, the instructions can execute the following steps:

[0168] Receive an image generation instruction, the image generation instruction including image attribute information of the target image to be generated;

[0169] Environmental information is acquired, which is collected by a display device that displays the target image;

[0170] The environmental information is mapped using a preset image generation model to generate a content image corresponding to the environmental information.

[0171] Style features are extracted from at least one preset style reference image, and content features are extracted from the content image. Based on the fusion of the content features and the style features, a target image is obtained.

[0172] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0173] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk, etc.

[0174] Since the instructions stored in the computer-readable storage medium can execute the steps of any of the image generation methods provided in the embodiments of the present invention, the beneficial effects that any of the image generation methods provided in the embodiments of the present invention can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0175] According to one aspect of this application, a computer program product or computer program is also provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations of the above embodiments.

[0176] The foregoing has provided a detailed description of an image generation method, apparatus, computer device, and storage medium provided by embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. An image generation method, characterized in that, include: Receive an image generation instruction, the image generation instruction including image attribute information of the target image to be generated; Environmental information is acquired, which is collected by a display device that displays the target image; The environmental information is mapped using a preset image generation model to generate a content image corresponding to the environmental information. Style features are extracted from at least one preset style reference image, and content features are extracted from the content image. Based on the fusion of the content features and the style features, a target image is obtained. Before generating the content image corresponding to the environmental information by mapping the environmental information using a preset image generation model, the method further includes: Obtain a generative adversarial model to be trained, the generative adversarial model including an image generation model to be trained and a discriminative model to be trained, the generative adversarial model being set with at least one real content sample image; The training input parameters are mapped using the image generation model to be trained, and the training content image corresponding to the training input parameters is generated. The discriminant model to be trained calculates the realism parameter of the training content image relative to the real content sample image; the realism parameter is used to describe the realism of the image content in the training content image, and the realism is determined by the similarity between the training content image and the real content sample image. The image generation model and the discrimination model to be trained are adjusted based on the realism parameters. Return to the step of mapping the training input parameters to the image generation model to be trained, until the preset training termination condition is met, and obtain the trained image generation model.

2. The image generation method according to claim 1, characterized in that, Before generating the training content image corresponding to the training input parameters by mapping the training input parameters through the image generation model to be trained, the method further includes: Obtain the display parameters of the display device; Based on the display parameters, the model parameters of the image generation model to be trained are set to obtain a new image generation model to be trained.

3. The image generation method according to claim 1, characterized in that, The extraction of style features from at least one preset style reference image and the extraction of content features from the content image include: Based on the style feature mapping parameters of the style extraction layer in the style transfer model, style features are extracted from at least one preset style reference image to obtain the style features corresponding to the style reference image. Based on the content feature mapping parameters of the content extraction layer in the style transfer model, content features are extracted from the content image to obtain the content features corresponding to the content image.

4. The image generation method according to claim 1, characterized in that, Before extracting style features from at least one preset style reference image based on the style feature mapping parameters of the style extraction layer in the style transfer model to obtain the style features corresponding to the style reference image, the method further includes: The style transfer model to be trained is used to extract style features from the sample style reference image to obtain the sample style features corresponding to the sample style reference image. The style transfer model to be trained is used to extract content features from the sample content image to obtain the sample content features corresponding to the sample content image. The sample style features and sample content features are fused to obtain a sample style transfer image; Based on the sample style transfer image, the sample style reference image, and the sample content image, calculate the model loss of the style transfer model to be trained; Based on the model loss, the model parameters of the style transfer model to be trained are adjusted to obtain the trained style transfer model.

5. The image generation method according to claim 1, characterized in that, The acquisition of environmental information includes: Collect ambient sound audio through a display device; The ambient sound audio is converted into the model input format of the image generation model to obtain environmental information.

6. The image generation method according to claim 1, characterized in that, The image attribute information includes image style type. The steps of extracting style features from at least one preset style reference image and extracting content features from the content image, and obtaining the target image based on the fusion of the content features and the style features, include: From a preset set of at least one style reference image, select at least one target style reference image corresponding to the style type of the image; Style features are extracted from each of the target style reference images, and content features are extracted from the content images; The content features and style features are weighted and fused to obtain the target image.

7. An image generation apparatus, characterized in that, include: The instruction receiving unit is used to receive an image generation instruction, which includes image attribute information of the target image to be generated. An environmental information acquisition unit is used to acquire environmental information, which is collected by a display device that displays the target image; The content image generation unit is used to perform data mapping on the environmental information through a preset image generation model to generate a content image corresponding to the environmental information. The target image generation unit is used to extract style features from at least one preset style reference image and extract content features from the content image, and obtain a target image based on the fusion of the content features and the style features; The image generation device further includes an image generation model training unit, which is used for: Obtain a generative adversarial model to be trained, the generative adversarial model including an image generation model to be trained and a discriminative model to be trained, the generative adversarial model being set with at least one real content sample image; The training input parameters are mapped using the image generation model to be trained, and the training content image corresponding to the training input parameters is generated. The discriminant model to be trained is used to calculate the realism parameter of the training content image relative to the real content sample image; The realism parameter is used to describe the realism of the image content in the training content image, and the realism is determined by the similarity between the training content image and the real content sample image; The image generation model and the discrimination model to be trained are adjusted based on the realism parameters. Return to the step of mapping the training input parameters to the image generation model to be trained, until the preset training termination condition is met, and obtain the trained image generation model.

8. A computer device, characterized in that, It includes a memory and a processor; the memory stores an application program, and the processor is used to run the application program within the memory to perform the steps of the image generation method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps of the image generation method according to any one of claims 1 to 6.

Images