Style image generation method and device, computer equipment and readable storage medium

Abstract

The invention relates to a style image generation method and device, computer equipment and a readable storage medium. The method comprises the steps of obtaining a to-be-processed image and style conversion information for the to-be-processed image, inputting the style conversion information for the to-be-processed image into a style recognition network in a style model, and obtaining position coding information and style coding information of the position coding information, and inputting the to-be-processed image, the position coding information and the style coding information of the position coding information into a style generation network in a style model to obtain a target style image corresponding to the to-be-processed image. By adopting the method, the accuracy of style image generation can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a style image generation method, apparatus, computer equipment and readable storage medium. Background Art

[0002] With the continuous development of artificial intelligence technology, style image generation methods have emerged to improve the flexibility of image style transfer. Existing style image generation methods usually perform style transfer on the image to be processed based on the weight value corresponding to the template image.

[0003] However, the existing style image generation method directly performs a simple weighted calculation on the entire image to be processed, which easily ignores the local image information in the image to be processed, thereby reducing the accuracy of style image generation. Summary of the invention

[0004] Based on this, it is necessary to provide a style image generation method, apparatus, computer device and readable storage medium that can improve the accuracy of style image generation in response to the above technical problems.

[0005] In a first aspect, the present application provides a style image generation method, comprising:

[0006] Obtaining an image to be processed and style conversion information for the image to be processed;

[0007] Inputting style conversion information of the image to be processed into a style recognition network in the style model to obtain position encoding information and style encoding information of the position encoding information;

[0008] The image to be processed, the position encoding information and the style encoding information of the position encoding information are input into the style generation network in the style model to obtain a target style image corresponding to the image to be processed.

[0009] In the embodiment of the present application, compared with the related art in which a simple weighted calculation is directly performed on the entire image to be processed to perform image style conversion, the above method is adopted to effectively ensure the accuracy of image style conversion by extracting the local position coding information of the image to be processed and the style coding information corresponding to the position coding information from the style conversion information, thereby ensuring the accuracy of style image generation.

[0010] In one embodiment, the style generation network includes an encoder and a decoder, and the image to be processed, the position encoding information, and the style encoding information of the position encoding information are input into the style generation network in the style model to obtain a target style image corresponding to the image to be processed, including:

[0011] The image to be processed is input into the encoder to obtain image coding information of the image to be processed; the image coding information, the position coding information and the style coding information of the position coding information are fused to obtain fused coding information; the fused coding information is input into the decoder to obtain a target style image corresponding to the image to be processed.

[0012] In an embodiment of the present application, by using an encoder to encode the image to be processed, and using a decoder to decode the fused encoding information after fusion processing to obtain a target style image, the image information in the image to be processed can be analyzed more accurately, thereby ensuring the accuracy of the target style image.

[0013] In one embodiment, the image coding information, the position coding information and the style coding information of the position coding information are fused to obtain fused coding information, including:

[0014] According to the style coding information, a style correction parameter is determined; according to the style correction parameter and the position coding information, the image coding information is processed to obtain fused coding information.

[0015] In an embodiment of the present application, a style correction parameter is introduced. By processing the image coding information according to the style correction parameter and the position coding information, fused coding information is obtained. The style features corresponding to the style coding information can be effectively added to the fused coding information, thereby ensuring the accuracy of the fused coding information.

[0016] In one embodiment, the image coding information includes pixel coding information of each pixel in the image to be processed, and the fused coding information includes conversion coding information and smoothing coding information; the image coding information is processed according to the style correction parameter and the position coding information to obtain the fused coding information, including:

[0017] The pixel point corresponding to the position coding information is taken as the target pixel point, and the pixel points other than the target pixel point in the image to be processed are taken as other pixel points; the pixel coding information corresponding to the target pixel point is processed using the style correction parameter to obtain the converted coding information of the target pixel point; the pixel coding information corresponding to the other pixel points is smoothed according to the pixel distance between the other pixel points and the target pixel point to obtain the smoothed coding information of the other pixel points.

[0018] In the embodiment of the present application, by smoothing other pixels according to the pixel distance between other pixels and the target pixel, the problem of image tearing can be effectively avoided, thereby ensuring the reliability of the fused coding information.

[0019] In one embodiment, if the number of target pixel points is at least two, smoothing the pixel coding information corresponding to other pixel points is performed according to the pixel distance between the other pixel points and the target pixel point to obtain the smoothed coding information of the other pixel points, including:

[0020] For each other pixel point, the pixel distance between the other pixel point and each target pixel point is determined; the target pixel point corresponding to the minimum pixel distance among the pixel distances is used as the associated pixel point of the other pixel points; according to the pixel distance between the other pixel point and the associated pixel point, the pixel coding information corresponding to the other pixel point is smoothed to obtain the smoothed coding information of the other pixel point.

[0021] In an embodiment of the present application, when there are multiple target pixel points, the reliability of the smoothing process can be ensured by using the target pixel point corresponding to the minimum pixel distance among the pixel distances as the associated pixel point of other pixel points and then smoothing the other pixel points.

[0022] In one embodiment, the style model is trained in the following manner:

[0023] The sample image and the style conversion information for the sample image are input into the style model to obtain a predicted style image; the style recognition network in the style model is trained using the predicted style image, the sample image and the style conversion information for the sample image; the style generation network in the style model is trained using a set of standard style images corresponding to the predicted style image and the style conversion information for the sample image.

[0024] In the embodiment of the present application, by adopting different methods to train the style recognition network and the style generation network in the style model, the rationality of the model training can be ensured, thereby ensuring the accuracy of the style model.

[0025] In one embodiment, the predicted style image, the sample image, and the style conversion information for the sample image are used to train the style recognition network in the style model, including:

[0026] According to the predicted style image and the sample image, actual conversion information is determined; according to the error value between the actual conversion information and the style conversion information for the sample image, the style recognition network in the style model is trained.

[0027] In the embodiment of the present application, by training the style recognition network in the style model according to the error value between the actual conversion information and the style conversion information, the accuracy of the style recognition network training can be ensured.

[0028] In a second aspect, the present application further provides a style image generation device, comprising:

[0029] An information acquisition module, used to acquire an image to be processed and style conversion information for the image to be processed;

[0030] A style recognition module, used for inputting style conversion information of the image to be processed into a style recognition network in the style model to obtain position encoding information and style encoding information of the position encoding information;

[0031] The style conversion module is used to input the image to be processed, the position encoding information and the style encoding information of the position encoding information into the style generation network in the style model to obtain the target style image corresponding to the image to be processed.

[0032] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0033] Obtaining an image to be processed and style conversion information for the image to be processed;

[0034] Inputting style conversion information of the image to be processed into a style recognition network in the style model to obtain position encoding information and style encoding information of the position encoding information;

[0035] The image to be processed, the position encoding information and the style encoding information of the position encoding information are input into the style generation network in the style model to obtain a target style image corresponding to the image to be processed.

[0036] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0037] Obtaining an image to be processed and style conversion information for the image to be processed;

[0038] Inputting style conversion information of the image to be processed into a style recognition network in the style model to obtain position encoding information and style encoding information of the position encoding information;

[0039] The image to be processed, the position encoding information and the style encoding information of the position encoding information are input into the style generation network in the style model to obtain a target style image corresponding to the image to be processed.

[0040] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0041] Obtaining an image to be processed and style conversion information for the image to be processed;

[0042] Inputting style conversion information of the image to be processed into a style recognition network in the style model to obtain position encoding information and style encoding information of the position encoding information;

[0043] The image to be processed, the position encoding information and the style encoding information of the position encoding information are input into the style generation network in the style model to obtain a target style image corresponding to the image to be processed.

[0044] The above-mentioned style image generation method, apparatus, computer equipment and readable storage medium obtain the image to be processed and the style conversion information for the image to be processed, and input the style conversion information for the image to be processed into the style recognition network in the style model to obtain the position coding information and the style coding information of the position coding information; and then input the image to be processed, the position coding information and the style coding information of the position coding information into the style generation network in the style model to obtain the target style image corresponding to the image to be processed. Compared with the related art, in which the image style conversion is directly performed by performing a simple weighted calculation on the entire image to be processed, the above-mentioned method can effectively ensure the accuracy of the image style conversion by extracting the position coding information of the local part of the image to be processed and the style coding information corresponding to the position coding information from the style conversion information, thereby ensuring the accuracy of the style image generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 is a schematic diagram of a flow chart of a method for generating a style image in one embodiment;

[0047] Figure 2 A schematic diagram of a process for determining a target style image in one embodiment;

[0048] Figure 3 A schematic diagram of a process for determining fusion coding information in an embodiment;

[0049] Figure 4 A schematic diagram of a process for determining fusion coding information in another embodiment;

[0050] Figure 5A is a schematic diagram of a process of training a style model in one embodiment;

[0051] Figure 5B A schematic diagram of style model training in one embodiment;

[0052] Figure 6 is a schematic diagram of a flow chart of a style image generation method in another embodiment;

[0053] Figure 7 is a structural block diagram of a style image generating device in one embodiment;

[0054] Figure 8 is a structural block diagram of a style image generating device in another embodiment;

[0055] Fig. 9 is a structural block diagram of a style image generating device in yet another embodiment;

[0056] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] With the continuous development of artificial intelligence technology, style image generation methods have emerged to improve the flexibility of image style transfer. Existing style image generation methods usually perform style transfer on the image to be processed based on the weight value corresponding to the template image.

[0059] However, the existing style image generation method directly performs a simple weighted calculation on the entire image to be processed, which easily ignores the local image information in the image to be processed, thereby reducing the accuracy of style image generation. Based on this, in an exemplary embodiment, a style image generation method is provided, and the method is applied to an image generation device as an example for explanation. Figure 1 As shown, the specific steps include:

[0060] S101, obtaining an image to be processed and style conversion information for the image to be processed.

[0061] The image to be processed refers to a picture with style conversion requirements; the style conversion information is used to describe the image style that needs to be converted to the image to be processed, and may include but is not limited to image description text and image description audio.

[0062] Optionally, after obtaining the image conversion instruction, the information collection interface can be displayed to the user through the terminal; then, the user can upload the image to be processed that needs to convert the image style and the style conversion information for the image to be processed to the information collection interface.

[0063] Furthermore, the user can click the send button in the information collection interface to send the image to be processed and the style conversion information to the image generating device, so that the image generating device obtains the image to be processed and the style conversion information for the image to be processed.

[0064] S102, inputting style conversion information for the image to be processed into a style recognition network in a style model to obtain position encoding information and style encoding information of the position encoding information.

[0065] Among them, the style model refers to a model used to convert the image style of the image to be processed, which can be a neural network model, etc.; the style recognition network refers to a network that can extract position encoding information and style encoding information from the recognition style conversion information. Furthermore, the position encoding information refers to the encoding information corresponding to the position in the image to be processed where style conversion is required, and the style encoding information refers to the encoding information corresponding to the style that needs to be converted.

[0066] Optionally, when the style transfer information is text information, the style transfer information can be directly input into the style recognition network in the style model, and the style recognition network outputs position encoding information and style encoding information of the position encoding information based on position keywords related to the position in the style transfer information and the style keywords associated with each position.

[0067] Alternatively, when the style transfer information is audio information, a conversion device can be used to convert the audio information into text information; then, the style transfer information is input into a style recognition network in the style model, and the style recognition network outputs position encoding information and style encoding information of the position encoding information based on position keywords related to the position in the style transfer information and style keywords associated with each position.

[0068] For example, the style conversion information A is "convert the sun located in the upper left corner of the image into a cartoon style". At this time, the style conversion information A is processed using a style recognition network to obtain the position coding information A1 corresponding to the "upper left corner" and the style coding information A2 corresponding to the "cartoon style".

[0069] It is understandable that in order to improve the flexibility of style conversion, when there are multiple position coding information, the style coding information corresponding to each position coding information may be different, that is, regional images at different positions in the image to be processed may be converted into different styles.

[0070] For example, if the style conversion information B is "convert the sun in the upper left corner of the image into a cartoon style, and convert the trees in the middle of the image into a black and white style", then the style recognition network is used to process the style conversion information B to obtain the position encoding information B corresponding to the "upper left corner"1-1 Style encoding information B corresponding to "comic style" 1-2 , and the position encoding information B corresponding to "middle" 2-1 Style encoding information B corresponding to "black and white style" 2-2 .

[0071] S103, inputting the image to be processed, the position encoding information and the style encoding information of the position encoding information into a style generation network in the style model to obtain a target style image corresponding to the image to be processed.

[0072] Among them, the style generation network refers to a network that can perform style conversion on the processed image based on position encoding information and style encoding information; the target style image refers to the image output by the style generation network.

[0073] Optionally, the image to be processed, the position encoding information and the style encoding information of the position encoding information can be simultaneously input into the style generation network in the style model, and the style generation network outputs a target style image corresponding to the image to be processed based on the image to be processed, the position encoding information, the style encoding information of the position encoding information and the network parameters.

[0074] It is understandable that in order to ensure the uniformity of image processing, the size of the image to be processed can be preprocessed so that the images to be processed input to the style generation network are all images of uniform size. For example, the size of the input image can be set to an image with a length and width of 512×512 pixels. If the size of the image to be processed is larger than 512×512 pixels, the image to be processed is scaled so that the size of the image to be processed reaches 512×512 pixels; if the size of the image to be processed is smaller than 512×512 pixels, the image to be processed is enlarged so that the size of the image to be processed reaches 512×512 pixels.

[0075] In the above-mentioned style image generation method, by obtaining the image to be processed and the style conversion information for the image to be processed, and inputting the style conversion information for the image to be processed into the style recognition network in the style model, the position coding information and the style coding information of the position coding information are obtained; and then the image to be processed, the position coding information and the style coding information of the position coding information are input into the style generation network in the style model to obtain the target style image corresponding to the image to be processed. Compared with the related art, in which a simple weighted calculation is directly performed on the entire image to be processed to perform image style conversion, the above-mentioned method can effectively ensure the accuracy of image style conversion by extracting the local position coding information of the image to be processed and the style coding information corresponding to the position coding information from the style conversion information, thereby ensuring the accuracy of style image generation.

[0076] In actual use, in order to ensure the accuracy of the style generation network in processing the image to be processed, based on the above embodiment, in this embodiment, the style generation network includes an encoder and a decoder. Based on this, an optional method for determining the target style image is provided, such as Figure 2 As shown, the specific steps include:

[0077] S201, inputting the image to be processed into an encoder to obtain image coding information of the image to be processed.

[0078] The encoder is used to convert the input sequence into a context vector of a fixed length, and in this embodiment is used to convert the image to be processed into image coding information; the image coding information refers to the vector information obtained after encoding the image to be processed.

[0079] Optionally, in order to more accurately analyze the image to be processed, the image to be processed may be input into an encoder, and the encoder outputs image coding information of the image to be processed according to image information of the image to be processed.

[0080] For example, the encoder may be composed of two convolutional layers and two pooling layers, wherein the convolutional layers may be used to increase the dimension of the image to be processed, and the pooling layers may be used to downsample the image to be processed, so as to extract image coding information of the image to be processed.

[0081] S202, fusing the image coding information, the position coding information and the style coding information of the position coding information to obtain fused coding information.

[0082] The fused coding information refers to coding information obtained by fusing the image coding information with the position coding information and the style coding information of the position coding information.

[0083] Optionally, the image coding information, position coding information and style coding information of the position coding information can be simultaneously input into a trained fusion model, and the fusion model outputs the fused coding information based on the image coding information, position coding information, style coding information of the position coding information and model parameters.

[0084] Alternatively, the style coding information of the position coding information may be used to fuse the coding information of the region associated with the position coding information in the image coding information, thereby obtaining the fused coding information.

[0085] S203, inputting the fused coding information into a decoder to obtain a target style image corresponding to the image to be processed.

[0086] The decoder is used to convert the context vector into an output sequence, and in this embodiment is used to convert the fused coding information into a target style image.

[0087] Optionally, after obtaining the fused coding information, in order to be able to display the fused coding information again in the form of a picture, the fused coding information can be input into a decoder, and the decoder outputs a target style image corresponding to the image to be processed according to the fused coding information.

[0088] For example, the decoder may be composed of two upsampling layers and two convolutional layers, wherein the image encoding information may be subjected to dimensionality reduction processing through the upsampling layers and the convolutional layers so as to obtain a target style image corresponding to the image to be processed.

[0089] In an embodiment of the present application, by using an encoder to encode the image to be processed, and using a decoder to decode the fused encoding information after fusion processing to obtain a target style image, the image information in the image to be processed can be analyzed more accurately, thereby ensuring the accuracy of the target style image.

[0090] In order to effectively add the style features corresponding to the style coding information into the fused coding information, based on the above embodiment, in this embodiment, an optional method for determining the fused coding information is provided, such as Figure 3 As shown, the specific steps include:

[0091] S301, determining style correction parameters according to style coding information.

[0092] The style correction parameters refer to the parameters used to transform the image style.

[0093] It is understandable that in order to ensure the accuracy of image style conversion, the candidate styles that can be converted by the style generation network can be counted in advance, and at the same time, the style correction parameters corresponding to the encoding information of each candidate style can be determined based on the encoding features of the image corresponding to each candidate style.

[0094] Based on this, the style coding information of the image to be processed may be used as an index to determine the style correction parameter corresponding to the style coding information of the image to be processed from the style correction parameters corresponding to each candidate style coding information.

[0095] S302: Process the image coding information according to the style correction parameter and the position coding information to obtain fused coding information.

[0096] Optionally, the style correction parameter may be used to process the encoding information of the region associated with the position encoding information in the image encoding information, thereby obtaining the fused encoding information. For example, the sum of the style correction parameter and the parameter value corresponding to the encoding information of the region associated with the position encoding information in the image encoding information may be directly used as the fused encoding information.

[0097] In an embodiment of the present application, a style correction parameter is introduced. By processing the image coding information according to the style correction parameter and the position coding information, fused coding information is obtained. The style features corresponding to the style coding information can be effectively added to the fused coding information, thereby ensuring the accuracy of the fused coding information.

[0098] In order to ensure the reliability of the fused coding information, based on the above embodiment, in this embodiment, the image coding information includes the pixel coding information of each pixel in the image to be processed, and the fused coding information includes the conversion coding information and the smoothing coding information; further, another optional method for determining the fused coding information is provided, such as Figure 4 As shown, the specific steps include:

[0099] S401, taking the pixel corresponding to the position coding information as the target pixel, and taking the pixel other than the target pixel in the image to be processed as other pixel.

[0100] The target pixel point refers to the pixel point corresponding to the position coding information in the image to be processed; and the other pixel points refer to the pixel points other than the target pixel point in the image to be processed.

[0101] Optionally, the pixel points at the regional positions corresponding to the position coding information in the image to be processed may be used as target pixel points; then, the pixel points in the image to be processed except the target pixel points may be directly used as other pixel points.

[0102] S402, using the style correction parameter, processing the pixel encoding information corresponding to the target pixel point to obtain the converted encoding information of the target pixel point.

[0103] Among them, pixel coding information refers to the coding feature information that can represent the target pixel point, such as a mask value; conversion coding information refers to the coding information after style conversion processing.

[0104] Optionally, the style correction parameter may be directly used to perform sum processing on the pixel encoding information corresponding to the target pixel point, and the conversion encoding information of the target pixel point may be calculated.

[0105] Alternatively, a mask correction coefficient may be determined based on mask information corresponding to the target pixel point; then, the mask correction coefficient and the style correction parameter are used to process pixel encoding information corresponding to the target pixel point to obtain conversion encoding information of the target pixel point.

[0106] For example, the pixel encoding information E[P n] as the index, and determine the mask correction coefficient j corresponding to the target pixel point P from the corresponding relationship between the pixel encoding information and the mask correction coefficient; then, refer to the following formula (1), according to the style correction parameter S n The pixel coding information corresponding to the target pixel point P is processed by the mask correction coefficient j to obtain the converted coding information E[P new ].

[0107] E[P new ]=E[P n ]×j+S n (1)

[0108] Among them, the corresponding relationship between pixel encoding information and mask correction coefficient E[P n ]:j is, {0-25:1; 26-50:0.5; 51-75:0.3; 76-100:0.25; 101-125:0.2; 126-150:0.15; 151-255:0.1}.

[0109] S403, performing smoothing processing on pixel coding information corresponding to other pixel points according to the pixel distance between other pixel points and the target pixel point, to obtain smoothed coding information of other pixel points.

[0110] The smoothed coding information refers to the coding information obtained by smoothing the pixel coding information corresponding to other pixel points.

[0111] It is understandable that if style conversion is performed only on the target pixel, there will be a problem of image segmentation between the target pixel and other surrounding pixels. Therefore, in order to make the image between the target pixel and other surrounding pixels more peaceful, other pixels need to be smoothed.

[0112] Optionally, based on the pixel distance between the other pixel points and the target pixel point, the pixel coding information corresponding to the other pixel points may be smoothed to different degrees to obtain the smoothed coding information of the other pixel points.

[0113] Exemplarily, for each other pixel point, the pixel distance between the other pixel point and the target pixel point can be compared with a distance threshold; then, based on the comparison result, the smoothing correction parameter corresponding to the other pixel point is determined, and the pixel coding information corresponding to the other pixel point is smoothed using the smoothing correction parameter to obtain the smoothed coding information of the other pixel point.

[0114] For example, referring to the following formula (2), when the distance between other pixel points and the target pixel point P is greater than the distance threshold K, the value of the smoothing correction parameter i is [4, 5, 6]; when the distance between other pixel points and the target pixel point P is less than the distance threshold K, the value of the smoothing correction parameter i is [1, 2, 3].

[0115] E[P new ±i]=E[P n ]×j±S n / (5×i) (2)

[0116] In the embodiment of the present application, by smoothing other pixels according to the pixel distance between other pixels and the target pixel, the problem of image tearing can be effectively avoided, thereby ensuring the reliability of the fused coding information.

[0117] In practical applications, if the number of target pixel points is at least two, the problem of abnormal smoothing processing may occur. Based on this, on the basis of the above embodiment, in this embodiment, an optional method for determining smooth coding information is provided, specifically, for each other pixel point, the pixel distance between the other pixel point and each target pixel point is determined; the target pixel point corresponding to the minimum pixel distance among the pixel distances is used as the associated pixel point of the other pixel points; according to the pixel distance between the other pixel point and the associated pixel point, the pixel coding information corresponding to the other pixel point is smoothed to obtain the smoothed coding information of the other pixel point.

[0118] The pixel distance refers to the distance between two pixels; the associated pixel refers to the target pixel associated with other pixels during smoothing.

[0119] It is understandable that, in order to ensure the accuracy of the smoothing process, for each other pixel point, it is necessary to predetermine, from among the target pixels, a target pixel point to be used when the other pixel point is smoothed.

[0120] Optionally, for each other pixel point, the distance between the other pixel point and each target pixel point in the image to be processed can be used as the pixel distance between the other pixel point and each target pixel point; then, the pixel distances between the other pixel point and each target pixel point can be arranged in order from small to large, and then the minimum pixel distance can be determined, and the target pixel point corresponding to the minimum pixel distance can be used as the associated pixel point of the other pixel point.

[0121] Furthermore, after determining the associated pixel points of the other pixel points, the above step S403 can be referred to, and the pixel coding information corresponding to the other pixel points can be smoothed according to the pixel distance between the other pixel points and the target pixel point, so as to obtain the smoothed coding information of the other pixel points.

[0122] In an embodiment of the present application, when there are multiple target pixel points, the reliability of the smoothing process can be ensured by using the target pixel point corresponding to the minimum pixel distance among the pixel distances as the associated pixel point of other pixel points and then smoothing the other pixel points.

[0123] In order to ensure the accuracy of the style model, based on the above embodiment, in this embodiment, an optional method for training the style model is provided, such as Figure 5A As shown, the specific steps include:

[0124] S501: Input a sample image and style conversion information for the sample image into a style model to obtain a predicted style image.

[0125] Among them, the sample image refers to the picture used to train the style model; the predicted style image refers to the image output by the style model during the training process.

[0126] Optionally, sample data in the text data set can be pre-input into the style recognition network in the style model, and then the style recognition network can be initially trained based on the difference between the position encoding information and style encoding information output by the style recognition network and the standard position encoding information and style encoding information corresponding to the text data.

[0127] Among them, each sample data in the text data set contains one or more position coding information and style coding information corresponding to each position coding information, and the position coding information and the style coding information correspond one to one.

[0128] At the same time, a preset set of standard style images in various styles can be used to perform initial training on the style generation network in the style model, so that the style generation network can perform preliminary image style conversion.

[0129] Furthermore, when the style model has the ability to perform preliminary image style conversion, a picture with a specified size can be randomly selected as a sample picture; then, the sample image and the style conversion information for the sample image are simultaneously input into the style model that has not yet been trained, and the style model outputs a predicted style image.

[0130] S502: training a style recognition network in a style model using the predicted style image, the sample image, and the style conversion information for the sample image.

[0131] Optional, reference Figure 5B , discriminator A can be used to determine the image style difference based on the predicted style image and the sample image, and determine the network output deviation of the style recognition network based on the difference between the image style difference and the style conversion information; then, discriminator A can optimize the style recognition network in the style model based on the network output deviation.

[0132] For example, actual conversion information can be determined based on the predicted style image and the sample image, and the style recognition network in the style model can be trained based on the error value between the actual conversion information and the style conversion information for the sample image, wherein the actual conversion information refers to the style information actually converted by the sample image.

[0133] Optionally, actual conversion information of the sample image can be generated based on the image difference between the predicted style image and the sample image; then, a discriminator can be used to optimize the network parameters of the style recognition network based on the error value between the actual conversion information and the style conversion information for the sample image.

[0134] S503 , training a style generation network in the style model using the predicted style image and a standard style image set corresponding to the style conversion information of the sample image.

[0135] Optionally, to ensure the accuracy of the target style image, refer to Figure 5B , the discriminator B can be used to score the similarity between the predicted style image and the standard style image set corresponding to the style conversion information of the sample image; then, the discriminator B optimizes the decoder in the style generation network according to the scoring result.

[0136] In the embodiment of the present application, by adopting different methods to train the style recognition network and the style generation network in the style model, the rationality of the model training can be ensured, thereby ensuring the accuracy of the style model.

[0137] Figure 6 FIG. 2 is a flow chart of a method for generating a style image in another embodiment. Based on the above embodiment, this embodiment provides an optional example of a method for generating a style image. Figure 6 The specific implementation process is as follows:

[0138] S601: Input a sample image and style conversion information for the sample image into a style model to obtain a predicted style image.

[0139] S602: Determine actual conversion information according to the predicted style image and the sample image.

[0140] S603: training a style recognition network in the style model according to an error value between the actual conversion information and the style conversion information for the sample image.

[0141] S604: training a style generation network in the style model using the predicted style image and a standard style image set corresponding to the style conversion information of the sample image.

[0142] S605: Obtain the image to be processed and style conversion information for the image to be processed.

[0143] S606: Input the style conversion information for the image to be processed into the style recognition network in the style model to obtain position encoding information and style encoding information of the position encoding information.

[0144] S607: Input the image to be processed into the encoder of the style generation network to obtain image encoding information of the image to be processed.

[0145] S608: Determine style correction parameters according to the style coding information.

[0146] S609: taking the pixel corresponding to the position coding information as the target pixel, and taking the pixel other than the target pixel in the image to be processed as other pixel.

[0147] S610, using the style correction parameter, processing the pixel coding information corresponding to the target pixel point to obtain the converted coding information of the target pixel point.

[0148] S611, smoothing pixel coding information corresponding to other pixel points according to the pixel distance between the other pixel points and the target pixel point, to obtain smoothed coding information of the other pixel points.

[0149] Optionally, if the number of target pixel points is at least two, then for each other pixel point, the pixel distance between the other pixel point and each target pixel point is determined; the target pixel point corresponding to the minimum pixel distance among the pixel distances is used as the associated pixel point of the other pixel points; based on the pixel distance between the other pixel point and the associated pixel point, the pixel coding information corresponding to the other pixel point is smoothed to obtain the smoothed coding information of the other pixel point.

[0150] S612: Input the conversion coding information and the smoothing coding information into a decoder of the style generation network to obtain a target style image corresponding to the image to be processed.

[0151] The specific process of S601-S612 can refer to the description of the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0152] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0153] Based on the same inventive concept, the embodiment of the present application also provides a style image generation device for implementing the style image generation method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more style image generation device embodiments provided below can refer to the limitations of the style image generation method above, and will not be repeated here.

[0154] In an exemplary embodiment, Figure 7 As shown, a style image generation device 1 is provided, comprising: an information acquisition module 10, a style recognition module 20 and a style conversion module 30, wherein:

[0155] An information acquisition module 10 is used to acquire an image to be processed and style conversion information for the image to be processed;

[0156] A style recognition module 20, used for inputting style conversion information of the image to be processed into a style recognition network in the style model to obtain position encoding information and style encoding information of the position encoding information;

[0157] The style conversion module 30 is used to input the image to be processed, the position encoding information and the style encoding information of the position encoding information into the style generation network in the style model to obtain a target style image corresponding to the image to be processed.

[0158] In an exemplary embodiment, the style generation network includes an encoder and a decoder, such as Figure 8 As shown, the style conversion module 30 includes:

[0159] The encoding unit 31 is used to input the image to be processed into the encoder to obtain image encoding information of the image to be processed;

[0160] A fusion unit 32, configured to fuse the image coding information, the position coding information and the style coding information of the position coding information to obtain fused coding information;

[0161] The decoding unit 33 is used to input the fusion coding information into a decoder to obtain a target style image corresponding to the image to be processed.

[0162] In an exemplary embodiment, the fusion unit 32 includes:

[0163] A first subunit is used to determine a style correction parameter according to the style coding information;

[0164] The second subunit is used to process the image coding information according to the style correction parameter and the position coding information to obtain fused coding information.

[0165] In an exemplary embodiment, the image coding information includes pixel coding information of each pixel in the image to be processed, and the fusion coding information includes conversion coding information and smoothing coding information; further, the second subunit is specifically used to:

[0166] The pixel point corresponding to the position coding information is taken as the target pixel point, and the pixel points other than the target pixel point in the image to be processed are taken as other pixel points; the pixel coding information corresponding to the target pixel point is processed using the style correction parameter to obtain the converted coding information of the target pixel point; the pixel coding information corresponding to the other pixel points is smoothed according to the pixel distance between the other pixel points and the target pixel point to obtain the smoothed coding information of the other pixel points.

[0167] In an exemplary embodiment, if the number of target pixels is at least two, the second subunit is specifically configured to:

[0168] For each other pixel point, the pixel distance between the other pixel point and each target pixel point is determined; the target pixel point corresponding to the minimum pixel distance among the pixel distances is used as the associated pixel point of the other pixel points; according to the pixel distance between the other pixel point and the associated pixel point, the pixel coding information corresponding to the other pixel point is smoothed to obtain the smoothed coding information of the other pixel point.

[0169] In an exemplary embodiment, the style image generation device 1 further includes a model training module 40, wherein Fig. 9 As shown, the model training module 40 includes:

[0170] A prediction unit 41, configured to input a sample image and style conversion information for the sample image into a style model to obtain a predicted style image;

[0171] A first training unit 42 is used to train a style recognition network in the style model using the predicted style image, the sample image and the style conversion information for the sample image;

[0172] The second training unit 43 is used to train the style generation network in the style model by using the predicted style image and the standard style image set corresponding to the style conversion information of the sample image.

[0173] In an exemplary embodiment, the first training unit 42 is specifically configured to:

[0174] According to the predicted style image and the sample image, actual conversion information is determined; according to the error value between the actual conversion information and the style conversion information for the sample image, the style recognition network in the style model is trained.

[0175] Each module in the above-mentioned style image generation device can be implemented in whole or in part by software, hardware or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0176] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig.10 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store image data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a style image generation method is implemented.

[0177] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0178] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0179] Obtaining an image to be processed and style conversion information for the image to be processed;

[0180] Inputting style conversion information of the image to be processed into a style recognition network in the style model to obtain position encoding information and style encoding information of the position encoding information;

[0181] The image to be processed, the position encoding information and the style encoding information of the position encoding information are input into the style generation network in the style model to obtain a target style image corresponding to the image to be processed.

[0182] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0183] The image to be processed is input into the encoder to obtain image coding information of the image to be processed; the image coding information, the position coding information and the style coding information of the position coding information are fused to obtain fused coding information; the fused coding information is input into the decoder to obtain a target style image corresponding to the image to be processed.

[0184] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0185] According to the style coding information, a style correction parameter is determined; according to the style correction parameter and the position coding information, the image coding information is processed to obtain fused coding information.

[0186] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0187] The pixel point corresponding to the position coding information is taken as the target pixel point, and the pixel points other than the target pixel point in the image to be processed are taken as other pixel points; the pixel coding information corresponding to the target pixel point is processed using the style correction parameter to obtain the converted coding information of the target pixel point; the pixel coding information corresponding to the other pixel points is smoothed according to the pixel distance between the other pixel points and the target pixel point to obtain the smoothed coding information of the other pixel points.

[0188] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0189] For each other pixel point, the pixel distance between the other pixel point and each target pixel point is determined; the target pixel point corresponding to the minimum pixel distance among the pixel distances is used as the associated pixel point of the other pixel points; according to the pixel distance between the other pixel point and the associated pixel point, the pixel coding information corresponding to the other pixel point is smoothed to obtain the smoothed coding information of the other pixel point.

[0190] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0191] The sample image and the style conversion information for the sample image are input into the style model to obtain a predicted style image; the style recognition network in the style model is trained using the predicted style image, the sample image and the style conversion information for the sample image; the style generation network in the style model is trained using a set of standard style images corresponding to the predicted style image and the style conversion information for the sample image.

[0192] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0193] According to the predicted style image and the sample image, actual conversion information is determined; according to the error value between the actual conversion information and the style conversion information for the sample image, the style recognition network in the style model is trained.

[0194] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0195] Obtaining an image to be processed and style conversion information for the image to be processed;

[0196] Inputting style conversion information of the image to be processed into a style recognition network in the style model to obtain position encoding information and style encoding information of the position encoding information;

[0197] The image to be processed, the position encoding information and the style encoding information of the position encoding information are input into the style generation network in the style model to obtain a target style image corresponding to the image to be processed.

[0198] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0199] The image to be processed is input into the encoder to obtain image coding information of the image to be processed; the image coding information, the position coding information and the style coding information of the position coding information are fused to obtain fused coding information; the fused coding information is input into the decoder to obtain a target style image corresponding to the image to be processed.

[0200] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0201] According to the style coding information, a style correction parameter is determined; according to the style correction parameter and the position coding information, the image coding information is processed to obtain fused coding information.

[0202] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0203] The pixel point corresponding to the position coding information is taken as the target pixel point, and the pixel points other than the target pixel point in the image to be processed are taken as other pixel points; the pixel coding information corresponding to the target pixel point is processed using the style correction parameter to obtain the converted coding information of the target pixel point; the pixel coding information corresponding to the other pixel points is smoothed according to the pixel distance between the other pixel points and the target pixel point to obtain the smoothed coding information of the other pixel points.

[0204] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0205] For each other pixel point, the pixel distance between the other pixel point and each target pixel point is determined; the target pixel point corresponding to the minimum pixel distance among the pixel distances is used as the associated pixel point of the other pixel points; according to the pixel distance between the other pixel point and the associated pixel point, the pixel coding information corresponding to the other pixel point is smoothed to obtain the smoothed coding information of the other pixel point.

[0206] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0207] The sample image and the style conversion information for the sample image are input into the style model to obtain a predicted style image; the style recognition network in the style model is trained using the predicted style image, the sample image and the style conversion information for the sample image; the style generation network in the style model is trained using a set of standard style images corresponding to the predicted style image and the style conversion information for the sample image.

[0208] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0209] According to the predicted style image and the sample image, actual conversion information is determined; according to the error value between the actual conversion information and the style conversion information for the sample image, the style recognition network in the style model is trained.

[0210] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0211] Obtaining an image to be processed and style conversion information for the image to be processed;

[0212] Inputting style conversion information of the image to be processed into a style recognition network in the style model to obtain position encoding information and style encoding information of the position encoding information;

[0213] The image to be processed, the position encoding information and the style encoding information of the position encoding information are input into the style generation network in the style model to obtain a target style image corresponding to the image to be processed.

[0214] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0215] The image to be processed is input into the encoder to obtain image coding information of the image to be processed; the image coding information, the position coding information and the style coding information of the position coding information are fused to obtain fused coding information; the fused coding information is input into the decoder to obtain a target style image corresponding to the image to be processed.

[0216] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0217] According to the style coding information, a style correction parameter is determined; according to the style correction parameter and the position coding information, the image coding information is processed to obtain fused coding information.

[0218] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0219] The pixel point corresponding to the position coding information is taken as the target pixel point, and the pixel points other than the target pixel point in the image to be processed are taken as other pixel points; the pixel coding information corresponding to the target pixel point is processed using the style correction parameter to obtain the converted coding information of the target pixel point; the pixel coding information corresponding to the other pixel points is smoothed according to the pixel distance between the other pixel points and the target pixel point to obtain the smoothed coding information of the other pixel points.

[0220] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0221] For each other pixel point, the pixel distance between the other pixel point and each target pixel point is determined; the target pixel point corresponding to the minimum pixel distance among the pixel distances is used as the associated pixel point of the other pixel points; according to the pixel distance between the other pixel point and the associated pixel point, the pixel coding information corresponding to the other pixel point is smoothed to obtain the smoothed coding information of the other pixel point.

[0222] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0223] The sample image and the style conversion information for the sample image are input into the style model to obtain a predicted style image; the style recognition network in the style model is trained using the predicted style image, the sample image and the style conversion information for the sample image; the style generation network in the style model is trained using a set of standard style images corresponding to the predicted style image and the style conversion information for the sample image.

[0224] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0225] According to the predicted style image and the sample image, actual conversion information is determined; according to the error value between the actual conversion information and the style conversion information for the sample image, the style recognition network in the style model is trained.

[0226] It should be noted that the data involved in this application (including but not limited to standard style image set data, etc.) are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0227] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0228] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0229] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A style image generation method, characterized in that: The method comprises: Acquire an image to be processed and style conversion information for the image to be processed; Inputting style conversion information for the image to be processed into a style recognition network in a style model to obtain position encoding information and style encoding information of the position encoding information; The image to be processed, the position encoding information and the style encoding information of the position encoding information are input into a style generation network in the style model to obtain a target style image corresponding to the image to be processed.

2. The method according to claim 1, characterized in that The style generation network includes an encoder and a decoder, and the step of inputting the image to be processed, the position encoding information, and the style encoding information of the position encoding information into the style generation network in the style model to obtain a target style image corresponding to the image to be processed includes: Inputting the image to be processed into the encoder to obtain image coding information of the image to be processed; fusing the image coding information, the position coding information and the style coding information of the position coding information to obtain fused coding information; The fused coding information is input into the decoder to obtain a target style image corresponding to the image to be processed.

3. The method according to claim 2, characterized in that The fusing the image coding information, the position coding information and the style coding information of the position coding information to obtain the fused coding information includes: determining a style correction parameter according to the style encoding information; The image encoding information is processed according to the style correction parameter and the position encoding information to obtain fused encoding information.

4. The method according to claim 3, characterized in that The image coding information includes pixel coding information of each pixel in the image to be processed, and the fusion coding information includes conversion coding information and smoothing coding information; The step of processing the image coding information according to the style correction parameter and the position coding information to obtain fused coding information includes: Taking the pixel point corresponding to the position coding information as the target pixel point, and taking the pixel points in the image to be processed except the target pixel point as other pixel points; Using the style correction parameter, processing the pixel encoding information corresponding to the target pixel point to obtain the converted encoding information of the target pixel point; According to the pixel distance between the other pixel points and the target pixel point, the pixel coding information corresponding to the other pixel points is smoothed to obtain the smoothed coding information of the other pixel points.

5. The method according to claim 4, characterized in that If the number of the target pixel points is at least two, the smoothing of the pixel coding information corresponding to the other pixel points according to the pixel distance between the other pixel points and the target pixel point to obtain the smoothed coding information of the other pixel points includes: For each other pixel point, determining the pixel distance between the other pixel point and each target pixel point; The target pixel point corresponding to the minimum pixel distance among the pixel distances is used as the associated pixel point of the other pixel points; According to the pixel distance between the other pixel points and the associated pixel point, the pixel coding information corresponding to the other pixel points is smoothed to obtain the smoothed coding information of the other pixel points.

6. The method according to claim 1, characterized in that The style model is trained in the following way: Inputting a sample image and style conversion information for the sample image into a style model to obtain a predicted style image; Using the predicted style image, the sample image, and style conversion information for the sample image, training a style recognition network in the style model; The predicted style image and a standard style image set corresponding to the style conversion information of the sample image are used to train the style generation network in the style model.

7. The method according to claim 6, characterized in that The adopting the predicted style image, the sample image and the style conversion information for the sample image to train the style recognition network in the style model includes: Determining actual conversion information according to the predicted style image and the sample image; The style recognition network in the style model is trained according to the error value between the actual conversion information and the style conversion information for the sample image.

8. A style image generating device, characterized in that: The device comprises: An information acquisition module, used to acquire an image to be processed and style conversion information for the image to be processed; A style recognition module, used for inputting the style conversion information of the image to be processed into a style recognition network in the style model to obtain position encoding information and style encoding information of the position encoding information; The style conversion module is used to input the image to be processed, the position encoding information and the style encoding information of the position encoding information into the style generation network in the style model to obtain a target style image corresponding to the image to be processed.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images