Image generation method

Abstract

The invention discloses an image generation method, and relates to the technical field of image processing, and the method comprises the steps: obtaining a first image and image conversion demand information, and the first image comprises a portrait region image; processing the style reference information and the global image information of the first image through a global conversion network of the image processing model to obtain a global style image, and processing the style reference information and the portrait image information of the portrait region image through a portrait conversion network of the image processing model to obtain a portrait style image, the style reference information is determined based on the image conversion demand information; and fusing the global style image and the portrait style image through a fusion network of the image processing model to obtain a second image. According to the method, the problem that the quality of the image obtained based on the original image including the portrait area after style conversion is poor can be solved.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular, relates to an image generation method. Background Art

[0002] Image style transfer refers to transferring the content or structure of one image to the style or expression of another image, that is, combining the content of one image with the style of another image to generate an image with a new style.

[0003] In the related art, after the user selects a conversion style from multiple preset styles, the image style conversion model corresponding to the conversion style is used to convert the image style of the entire original image to obtain an image with the conversion style. However, when the original image includes a portrait area, the image after the style conversion is prone to distortion of the portrait area, resulting in poor quality of the image after the style conversion, affecting the visual effect of the image after the style conversion. Summary of the invention

[0004] The embodiments of the present application provide an image generation method, apparatus, device, medium and product, which can solve the problem in the related art that when the original image includes a portrait area, the image quality after the style conversion based on the original image is poor.

[0005] In a first aspect, an embodiment of the present application provides an image generation method, the image generation method comprising:

[0006] Acquire a first image and image conversion requirement information, where the first image includes a portrait area image;

[0007] Processing the style reference information and the global image information of the first image through a global conversion network of the image processing model to obtain a global style image, and processing the style reference information and the portrait image information of the portrait region image through a portrait conversion network of the image processing model to obtain a portrait style image, wherein the style reference information is determined based on the image conversion requirement information;

[0008] The global style image and the portrait style image are fused through the fusion network of the image processing model to obtain a second image.

[0009] In a second aspect, an embodiment of the present application provides an image generating device, the image generating device comprising:

[0010] A first acquisition module, used to acquire a first image and image conversion requirement information, wherein the first image includes a portrait area image;

[0011] A first processing module is used to process the style reference information and the global image information of the first image through a global conversion network of the image processing model to obtain a global style image, and to process the style reference information and the portrait image information of the portrait region image through a portrait conversion network of the image processing model to obtain a portrait style image;

[0012] The fusion module is used to fuse the global style image and the portrait style image through a fusion network of the image processing model to obtain a second image.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, an image generation method as described in any one of the first aspects is implemented.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, an image generation method as described in any one of the first aspects is implemented.

[0015] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program or instructions, and when the computer program or instructions are executed by a processor, an image generation method as described in any one of the first aspects is implemented.

[0016] The image generation method, device, equipment, medium and product of the embodiments of the present application use the global conversion network of the image processing model to process the first image and the style reference information determined based on the image conversion requirement information, and use the portrait conversion network of the image processing model to process the style reference information and the portrait image information, which effectively ensures that the global conversion network can process the same style reference information, that is, the same style target, in the process of processing the global image information, and the portrait conversion network can process the portrait image information. Then, the second image obtained by fusing the global style image and the portrait style image with the help of the fusion network can combine the global style with the portrait style, and can be coordinated in the overall style. In this way, while controlling the overall conversion style of the image, the characteristics of the portrait area in the image are taken into account, and the image quality of the image after style conversion is effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1A schematic diagram showing a flow chart of an image generation method provided by some embodiments of the present application;

[0019] Figure 2 A flowchart showing a specific implementation of step 140 provided in some embodiments of the present application is shown;

[0020] Figure 3 A flowchart showing a specific implementation of step 1401 provided in some embodiments of the present application is shown;

[0021] Figure 4 A flowchart showing a specific implementation of step 1402 provided in some embodiments of the present application is shown;

[0022] Figure 5 A flowchart showing a specific implementation of step 1403 provided in some embodiments of the present application is shown;

[0023] Figure 6 A schematic flow chart of a method for updating reference noise parameters according to a user's operation on a scoring option in an image generation method provided in some embodiments of the present application is shown;

[0024] Figure 7 A schematic flow chart of a method for updating reference noise parameters according to a user's operation on a download option in an image generation method provided in some embodiments of the present application is shown;

[0025] Figure 8 A schematic flow chart of a method for updating reference noise parameters according to a user's operation on a download option in an image generation method provided in some embodiments of the present application is shown;

[0026] Fig. 9 A schematic diagram of a style conversion interface provided by some embodiments of the present application is shown;

[0027] Fig.10 A schematic diagram showing a method for adjusting a first model parameter of an image processing model in an image generation method provided in some embodiments of the present application;

[0028] Fig.11 A schematic diagram showing a method for adjusting a second model parameter of an image processing model in an image generation method provided in some embodiments of the present application;

[0029] Fig.12 A schematic diagram showing the network architecture of a global conversion network and a portrait conversion network in an image generation method provided in some embodiments of the present application;

[0030] Fig.13 A flowchart showing a specific implementation of step 130 provided in some embodiments of the present application is shown;

[0031] Fig.14 A schematic diagram showing a method for determining a third weight and a fourth weight in an image generation method provided in some embodiments of the present application is shown;

[0032] Fig.15 A schematic diagram of the structure of an image generating device provided in some embodiments of the present application is shown;

[0033] Fig.16 A schematic diagram of the structure of an electronic device provided in some embodiments of the present application is shown. DETAILED DESCRIPTION

[0034] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0035] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0036] It should be noted that the acquisition, storage, use and processing of data in the embodiments of the present application are in compliance with the relevant provisions of national laws and regulations.

[0037] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0038] In order to solve the problems in the above-mentioned related technologies, the embodiments of the present application provide an image generation method, device, equipment, storage medium and program product. Figure 1 To Attachment Fig.14 , the image generation method provided in the embodiment of the present application is described in detail through specific embodiments and their application scenarios.

[0039] Figure 1 FIG. 1 is a flow chart of an image generation method provided by some embodiments of the present application. Figure 1 As shown, the image generating method may include steps 110 to 130.

[0040] Step 110: Acquire a first image and image conversion requirement information, where the first image includes a portrait area image.

[0041] Step 120, the style reference information and the global image information of the first image are processed through the global conversion network of the image processing model to obtain a global style image, and the style reference information and the portrait image information of the portrait area image are processed through the portrait conversion network of the image processing model to obtain a portrait style image, and the style reference information is determined based on the image conversion requirement information.

[0042] Step 130: fuse the global style image and the portrait style image through a fusion network of the image processing model to obtain a second image.

[0043] Therefore, the global conversion network of the image processing model is used to process the first image and the style reference information determined based on the image conversion requirement information, and the portrait conversion network of the image processing model is used to process the style reference information and the portrait image information, which effectively ensures that the global conversion network can process the global image information in the process of processing the global image information, and the portrait conversion network can process the portrait image information in the process of processing the same style reference information, that is, the same style target. Then, the second image obtained by fusing the global style image and the portrait style image with the help of the fusion network can combine the global style with the portrait style, and can be coordinated in the overall style. In this way, while controlling the overall conversion style of the image, the characteristics of the portrait area in the image are taken into account, and the image quality of the image after style conversion is effectively improved.

[0044] The above steps are described in detail below, as shown below.

[0045] First, step 110 is involved. The first image involved in the embodiment of the present application refers to the original image to be converted in the image generation process. The portrait area image is a specific part of the first image that includes a person's image, such as the face, full body or partial body image area of ​​one or more people. The image conversion requirement information is the guidance and constraint conditions for the style conversion operation of the first image, which can be set by the user according to his or her own aesthetic preferences, application scenario requirements, etc.

[0046] The first image can be obtained through a variety of ways. For example, the image to be style-converted can be selected by selecting it in a user file or by uploading it. Specifically, the user can select the image to be style-converted from a plurality of pre-stored images or uploaded images, and then pre-process the selected image such as cutting out a designated area of ​​the image and adjusting the color, and then determine the pre-processed image as the first image. For another example, the first image can be downloaded from a network resource. For another example, the first image can also be directly obtained in real time from an image acquisition device such as a digital camera, a webcam, etc.

[0047] In some embodiments of the present application, in order to improve the accuracy of the style reference information, before the above step 120, the above image generation method may further include step 140.

[0048] Step 140, processing the first image and the image conversion requirement information through the image processing network of the image processing model to obtain style reference information, global image information of the first image and portrait image information of the portrait area image, wherein the style reference information is information related to the expected conversion style for the first image.

[0049] Among them, the image processing model is a comprehensive model architecture that integrates the image processing network, the global conversion network, the portrait conversion network and the fusion network, aiming to achieve the style conversion task of the input image, i.e., the first image. The image processing model processes different levels of information of the first image, such as global image information and local portrait image information, respectively through the collaborative work between various networks.

[0050] The image processing network converts the original first image and abstract image conversion requirement information into specific intermediate information that can be processed by subsequent networks, namely style reference information, separated global image information and portrait image information. The style reference information provides a unified and precise style guidance for the global conversion network and the portrait conversion network, ensuring that the global conversion network can process the global image information and the portrait conversion network can process the portrait image information towards the same style goal.

[0051] In some embodiments of the present application, the image conversion requirement information may include style requirement information, which is used to specify the visual style related information that the first image should present after style conversion. The global image information includes a global sampling image and a global noise image corresponding to the first image, and the portrait image information includes a portrait sampling image and a portrait noise image corresponding to the portrait area image. Based on this, Figure 2 As shown, the above step 140 may specifically include steps 1401 to 1404.

[0052] Step 1401, input the first image and image conversion requirement information into an image processing network, process the style requirement information through the image processing network to obtain style reference information; and perform sampling processing on the first image to obtain a global sampling image.

[0053] Among them, the style requirement information may include a style conversion type or a style reference image. The style conversion type is used to indicate the style type expected to be obtained by performing style conversion processing on the first image. The style reference image provides an intuitive reference example of style conversion for the first image, and is used to guide the first image to be converted in the same or similar style direction as the style reference image, so that the final generated second image has a style similar to the style reference image. It can be understood that the style of the first image and the style corresponding to the style requirement information are at least two different styles. The style reference information is a vectorized representation of the style corresponding to the style requirement information.

[0054] In one example, the style conversion type includes but is not limited to ancient style, animation, aesthetics, fresh style, cartoon, sketch, ink painting, oil painting, quick sketch, printmaking, watercolor, etc. For the method of obtaining the style conversion type, the electronic device may display a style conversion type list, the style conversion type list includes at least two style conversion types, and the user may select a style conversion type according to his / her own needs, and the electronic device determines the selected style conversion type as the style conversion type corresponding to the first image.

[0055] In another example, the style reference image can be a real image obtained by an image acquisition device, or an image captured from a video, or an image searched by a user. The method for obtaining the style reference image can refer to the method for obtaining the first image, which will not be repeated here.

[0056] For the first image I I Perform sampling processing to obtain the global sampling image O I The sampling process can be expressed as follows (1):

[0057]

[0058] Among them, w cis the width scaling factor, h c is the high zoom factor, i, j is the global sampling image O I The coordinates of the pixel in .

[0059] Step 1402: extracting a portrait sampling image from the global sampling image.

[0060] Exemplarily, a portrait detection algorithm can be used to detect the global sampled image. Among them, the portrait detection algorithm can adopt a target detection algorithm based on deep learning such as You Only Look Once (YOLO), Faster Region-based Convolutional Neural Networks (Faster R-CNN), etc. These target detection algorithms can be trained with a large amount of annotated portrait data and can identify the portrait area in the global sampled image. After the portrait area is detected, the corresponding portrait sampled image is cropped from the global sampled image according to the bounding box coordinates of the portrait area.

[0061] Step 1403: Perform noise filling processing on the global sampled image using the style reference information to obtain a global noise image.

[0062] Among them, noise filling processing is the process of adding random or specific pattern noise data to the global sampling image.

[0063] For example, the noise may be Gaussian noise whose pixel values ​​follow Gaussian distribution, or salt and pepper noise of randomly appearing black and white pixels, etc. Here, the global noise image after adding noise increases the randomness and richness of the first image, which helps to simulate the texture and texture effects of different styles in the style conversion process.

[0064] Step 1404: extracting a portrait noise image from the global noise image.

[0065] Exemplarily, according to the portrait region bounding box coordinates obtained in step 1402 , a corresponding region may be cropped from the global noise image as the portrait noise image.

[0066] Therefore, the style reference information output by the image processing network can provide a clear guide for the style conversion of the first image. The global sampling image obtained by sampling processing can effectively reduce the amount of data processing while retaining the key features of the first image, thereby improving the processing efficiency of the image processing model. Next, the portrait sampling image is extracted from the global sampling image to achieve the separation of the portrait area and the global area, avoiding deformation or distortion of the portrait area during the unified processing process, and greatly ensuring the visual quality of the portrait area after style conversion. Then, the global noise image obtained by performing noise filling processing on the global sampling image based on the style reference information can enhance the visual effect of the image. Finally, the portrait noise image is extracted from the global noise image to ensure that the portrait area and the global area are consistent and coordinated at the noise level, further improving the integrity and harmony of the image after style conversion.

[0067] In some embodiments of the present application, when the style requirement information includes a style conversion type, the processing of the style requirement information in the above step 1401 to obtain style reference information may include determining a first style feature vector associated with the style conversion type as the style reference information based on association information between a preset style conversion type and a preset style feature vector.

[0068] Exemplarily, a pre-trained style set including a plurality of different style conversion types may be constructed. The method for obtaining the pre-trained style set may include, for each style conversion type, obtaining a plurality of image samples corresponding to the style conversion type and having typical features of the style conversion type, and inputting these image samples into a convolutional neural network, performing multi-layer convolution, pooling and other operations on these image samples using the convolutional neural network, gradually extracting features of the image samples from low-level to high-level, and converting them into corresponding style feature vectors. Then, according to the user's selection, a search and matching operation is performed in the constructed pre-trained style set, that is, after the user selects the style conversion type, the style feature vector associated with the style conversion type is retrieved in the pre-trained style set.

[0069] Therefore, through the association information between the style conversion type and the style feature vector, the style feature vector associated with the style conversion type can be accurately and quickly obtained, providing a style reference basis for the image style conversion of the first image.

[0070] In some other embodiments of the present application, when the style requirement information includes a style reference image, such as Figure 3 As shown, the processing of the style requirement information in the above step 1401 to obtain the style reference information may include steps 14011 to 14013.

[0071] Step 14011, compress the style reference image according to a preset size to obtain a style compressed image.

[0072] Among them, the preset size is a specific image size standard determined based on the structure and processing capabilities of the global transformation network, portrait transformation network and fusion network. By compressing the style reference image, it can effectively avoid processing errors or inefficiencies caused by size mismatch.

[0073] For example, the preset size is 244×244. If the width and height of the style image S are W and W respectively s ,H s , for the coordinate point (x, y) of the style image S, the pixel extraction can be performed using the following formula (2):

[0074]

[0075] Where x,y∈N * ,x,y≤244,

[0076] Step 14012, vectorize the style compressed image to obtain a second style feature vector.

[0077] Exemplarily, a convolutional neural network may be used to perform multi-layer convolution, pooling and other operations on the style compression image, gradually extracting features of the style compression image from low-level to high-level, and converting them into a corresponding second style feature vector.

[0078] Step 14013: determine the second style feature vector as style reference information.

[0079] Therefore, when processing the style reference image, the compression process ensures that the image size matches the processing capability of the image processing model, thereby improving the processing efficiency of the image processing model. The vectorization process represents the style features of the style reference image in a unified form that is easy for the image processing model to process, so that the subsequent global conversion network and portrait conversion network can accurately perform style conversion on the first image based on these feature information.

[0080] In some embodiments of the present application, Figure 4 As shown, the above step 1402 may specifically include steps 14021 to 14023.

[0081] Step 14021, perform grayscale conversion processing on the global sampling image to obtain a global grayscale image.

[0082] For example, the original RGB values ​​of each pixel point P(x, y) in the global sampling image are R(x,

[0083] y), G(x, y), B(x, y), then the gray value Gray(x, y) of P(x, y) can be expressed by the following formula (3):

[0084] Gray(x,y)=0.299R(x,y)+0.587G(x,y)+0.114B(x,y) (3)

[0085] In this way, all pixels of the entire global sampling image are converted into grayscale values ​​by the above formula (3) to obtain a global grayscale image.

[0086] Step 14022: Based on the preset portrait feature information, perform portrait detection on the global grayscale image to determine the portrait area image.

[0087] The portrait feature information is a set of pre-set features related to the portrait, and specifically, may include the shape, proportion, relative position relationship of facial organs, features of the human body contour, etc. Based on the portrait feature information, the target detection algorithm of step 1402 may be used to construct a portrait detection model or algorithm for identifying the portrait area in the image.

[0088] For example, the global grayscale image is input into the portrait detection model, and the portrait detection result P'={(x'1,h'1,w'1,h'1),(x'2,h'2,w'2,h'2),...,(x' n ,h′ n ,w′ n ,h′ n )}, where x′ i ,h′ i represents the starting coordinates of the i-th portrait, w′ i ,h′ i represents the width and height of the i-th portrait, and n represents the number of detected portraits.

[0089] Step 14023, extracting a portrait sampling image from the global sampling image according to the portrait area image.

[0090] Exemplarily, the portrait region may be determined according to the portrait starting coordinates and the portrait width and height of the portrait detection result, and then, the portrait region is extracted from the global sampling image based on the portrait region to obtain the portrait sampling image.

[0091] Therefore, by performing grayscale conversion on the global sampling image to obtain a global grayscale image, the amount of image processing data can be reduced and the efficiency and accuracy of portrait area detection can be improved. Then, the portrait sampling image is separated from the global sampling image, so that the subsequent image style conversion process can perform refined processing on the portrait area to better preserve the natural features of the portrait.

[0092] In some embodiments of the present application, Figure 5 As shown, the above step 1403 may specifically include step 14031 and step 14032.

[0093] Step 14031, generating a blank image corresponding to the global sampling image according to the size information and channel information of the global sampling image.

[0094] The size information refers to the number of pixels contained in the global sampling image in the horizontal and vertical directions. The channel information refers to the color mode and channel structure of the global sampling image.

[0095] Step 14032: Based on a reference noise parameter, the initial pixel value of the blank image is adjusted to obtain a global noise image. The reference noise parameter is determined based on user feedback information.

[0096] The reference noise parameters are a set of parameters used to control the noise generation and addition process, which can affect the characteristics of the noise, such as the intensity and distribution of the noise.

[0097] Exemplarily, after obtaining the blank image, the initial pixel values ​​of all pixels of the blank image are set to 0. Then, the pixel values ​​of all pixels of the blank image P are traversed, and the pixel value of each pixel is updated by the following formula (4):

[0098]

[0099] Where rand(k) represents a random number generator function that satisfies a normal distribution with an expectation of 1 and a variance of k, generating a value x∈R(0≤x≤1), i, j represents the position of the point with coordinates (i, j) in the blank image P, P amb is the user noise generalization factor, whose value is: P amb ∈N(0≤P amb ≤100), where S l It represents the output obtained after the content image is forward propagated and the style image is propagated to the lth layer where si(l) is 1, and then back-propagated to the content image.

[0100] Here, the user noise generalization factor can adjust the propagation range and intensity variation of noise in the image. Different P amb The value will make the noise present different density and size effects on the image. The k in the rand(k) function is related to the distribution characteristics of the generated random numbers, which in turn affects the randomness and volatility of the noise. By adjusting these reference noise parameters, a suitable global noise image can be generated according to the user's needs and the characteristics of the image to achieve the desired style transfer effect.

[0101] Therefore, the global noise image generated based on the structure of the global sampling image and the reference noise parameters can be better integrated with the image content, avoiding the abruptness of the noise, making the style-converted image more visually natural, and improving the quality and effect of image style conversion. At the same time, the introduction of reference noise parameters meets the needs of different users for diversification and refinement of image styles.

[0102] In some embodiments of the present application, the user feedback information includes a first user feedback value. Based on this, after the step 130, as shown in FIG. Figure 6 As shown, the image generating method may further include steps 210 to 230.

[0103] Step 210: Display the second image in a display area of ​​the style conversion interface. The style conversion interface includes a scoring option, and the scoring option is used to prompt the user to quantify and score the conversion effect of the second image.

[0104] The style conversion interface is an interactive interface that provides users with image style conversion operations and viewing conversion results, i.e., the second image. The scoring option is an interactive component set on the style conversion interface, which is used to guide users to quantitatively evaluate the image conversion effect.

[0105] For example, the style transfer interface integrates various elements related to image style transfer, such as displaying the original image to be converted, i.e., the first image, displaying the converted image, i.e., the second image, and displaying functional options for user operation and feedback, such as scoring options. The scoring options may be a group of radio buttons, such as each button corresponding to a specific score value, or a slider, such as a user dragging a slider to select a score.

[0106] Step 220: In response to a first input of a scoring option, a first user feedback value corresponding to the scoring option is obtained.

[0107] For example, suppose that the user performs a first input, such as a click operation, on a certain rating option. The first input is that the user clicks one or more five-pointed stars to rate. When the user performs such an operation, the system will respond and obtain the first user feedback value corresponding to the rating option. For example, if the rating range is 1 to 5 stars, if the user clicks the third five-pointed star to rate, then the first user feedback value can be 3.

[0108] Step 230: Update the reference noise parameter according to the first user feedback value.

[0109] For example, if the first user feedback value is low, it means that the user is not satisfied with the noise-related part of the current style conversion effect, and may feel that the noise is too strong or too weak, does not meet the expected style texture, etc. At this time, the user-generated noise generalization factor in the reference noise parameter and the parameters related to random number generation can be adjusted.

[0110] Therefore, by dynamically updating the reference noise parameters according to the first user feedback value, adaptive optimization of the image style transfer effect is achieved, that is, the noise-related style transfer features can be continuously adjusted according to the user's evaluation, and a style transfer image that better meets the user's expectations is gradually generated, thereby improving the adaptability and effectiveness of the entire image generation method in different user demand scenarios.

[0111] In some other embodiments of the present application, the user feedback information includes a second user feedback value. Based on this, after the above step 130, as shown in FIG. Figure 7 As shown, the image generating method may further include steps 310 to 330.

[0112] Step 310: Display the second image in a display area of ​​the style conversion interface. The style conversion interface includes a download option, and the download option is used to prompt the user to download the second image.

[0113] Among them, the download option is an interactive component set on the style conversion interface to facilitate users to save the generated second image to a local device such as a computer, mobile phone, etc. It can be presented in a visual form such as a button or link, and accompanied by corresponding text prompt information such as "Download image".

[0114] Step 320: In response to a second input of the download option, a second user feedback value corresponding to the download option is obtained.

[0115] Once the user's operation on the download option is detected, the second user feedback value associated with the download option is acquired and recorded. The second user feedback value may be a pre-set value indicating the user's recognition degree of the second image.

[0116] Exemplarily, the second user feedback value may be set to 1 if downloading indicates satisfaction, and 0 if not downloading indicates dissatisfaction.

[0117] Step 330: Update the reference noise parameter according to the second user feedback value.

[0118] For example, if the second user feedback value indicates that the user is satisfied with the image, it means that the current style transfer effect meets the user's expectations to a certain extent. At this time, the reference noise parameters can be appropriately fine-tuned to maintain or further optimize the current effect. For example, the user noise generalization factor and the parameters related to random number generation can be slightly increased to make the noise more natural. On the contrary, if the second user feedback value indicates that the user is dissatisfied, the reference noise parameters need to be adjusted to a greater extent, such as the user noise generalization factor and the parameters related to random number generation may be significantly changed, so as to change the noise effect in the subsequent second image, so as to generate a style transfer image that better meets the user's preferences.

[0119] Therefore, by dynamically updating the reference noise parameters according to the second user feedback value, adaptive optimization of the image style transfer effect is achieved. That is, based on the satisfaction reflected by the user's actual downloading behavior, the noise-related style transfer features in the image can be continuously adjusted to generate a converted image that meets user needs.

[0120] In some other embodiments of the present application, after the above step 130, as Figure 8 As shown, the image generating method may further include steps 410 to 430.

[0121] Step 410: When the style conversion interface includes a rating option and a download option, a first user feedback value corresponding to the rating option and a second user feedback value corresponding to the download option are obtained.

[0122] For example, Fig. 9 The style conversion interface shown has a first image and three second images corresponding to the first image. The three second images are obtained after performing three style conversions on the first image. A "Download Original" button is configured under each second image, and the button is used for the second image corresponding to the button. A row of rating options is configured under each "Download Original" button, and these rating options are used for users to evaluate the corresponding second image. Users can rate the image conversion results such as the second image by clicking on the five-pointed star, and the image generation system will obtain these ratings as the first user feedback value. The user can click the "Download Original" button to download the second image, and the image generation system will record the user's download operation as the second user feedback value.

[0123] Step 420 , performing weighted summation on the first user feedback value and the second user feedback value according to the first weight of the first user feedback value and the second weight of the second user feedback value to obtain a target feedback value.

[0124] Among them, the first weight corresponds to the first user feedback value, which indicates the proportion of the first user feedback value in determining the updating direction and degree of the reference noise parameter; the second weight corresponds to the second user feedback value, which indicates the proportion of the first user feedback value in determining the updating direction and degree of the reference noise parameter. The values ​​of the first weight and the second weight can be determined based on factors such as the actual application scenario, the analysis of the importance of different feedback channels, and past experience.

[0125] For example, for the same conversion request of the user for the first image, multiple conversions can be performed, and the second images obtained after the multiple conversions are fed back to the user for selection and rating. If three conversions are performed, three corresponding converted images I′ are generated. g = {I g1 ,I g2 ,I g3}, feedback to the user for selection and scoring, the user can select multiple options, and download the selected images to the local through the download option, and the image generation system records the downloaded images I d =

[0126] {(I g1 ,F g1 ),(I g2 ,F g2 ),(I g3 ,F g3 )}, where F gi is the second user feedback value, indicating I gi Whether it is downloaded, when the image is downloaded, I gi =1, otherwise I gi =0, then the user can rate each second image, and the range of rating options is M p ∈N,0≤M s ≤10, get the score result:

[0127] M g ={(I g1 ,M g1 ),(I g2 ,M g2 ),(I g3 ,M g3 )}, where M gi Indicates the user's gi Finally, according to the first user feedback value and the second user feedback value, the target feedback value of the image is calculated: MC g ={(I g1 ,MC g1 ),(I g2 ,MC g2 ),(I g3 ,MC g3)}, where MC gi Indicates I gi The target feedback value can be expressed by the following formula (5):

[0128] MC gi =F g3 *0.3+M g3 *0.7 (5)

[0129] Among them, 0.7 is the first weight and 0.3 is the second weight.

[0130] Step 430: Update the reference noise parameter according to the target feedback value.

[0131] For example, the user noise generalization factor can be updated on a daily basis. First, the user's image conversion records and their corresponding user feedback information records R = {(I1, MC1, k1), (I2, MC2, k2), ...,

[0132] (I n ,MC n ,k n )}, where I n represents the nth second image, MC n Indicates I n The target feedback value, k n Indicates I n The sequence number in the entire conversion sequence can be updated by the following formula (6) to update the user noise generalization factor P amb :

[0133]

[0134] Among them, MC i -5 is used to measure how good the i-th second image is relative to the intermediate feedback level 5.

[0135] Therefore, by integrating the first user feedback value and the second user feedback value in a weighted summation manner, the relative importance of different feedback channels in reflecting user satisfaction is fully considered, so that the final target feedback value can more objectively represent the user's comprehensive evaluation of the image style conversion effect of the second image. Compared with the feedback value that relies solely on a certain feedback channel, this comprehensive consideration method can avoid the problem of inaccurate judgment of the user's true feelings due to the limitations of a single feedback, and provide a reliable basis for more reasonable updating of the reference noise parameters.

[0136] In some embodiments of the present application, the above-mentioned image conversion requirement information includes the degree of style conversion, which refers to the degree to which the first image changes its original style and approaches the target style corresponding to the style reference information during style conversion. For example, when a first image with a realistic style is converted to an oil painting style, when the degree of style conversion is low, the second image only slightly has some color characteristics of the oil painting; and when the degree of style conversion is high, the second image will highly present the typical characteristics of the oil painting, such as rich colors, thick and obvious brushstrokes, etc. Based on this, after the above step 110, if Fig.10 As shown, the image generating method may include steps 510 to 520.

[0137] Step 510: Acquire a first model parameter associated with the style conversion degree according to the association information between the preset style conversion degree and the preset model parameter.

[0138] The preset model parameters are a series of adjustable parameter values ​​involved in the construction and training process of the image processing model, which determine the specific methods and effects of the model's various image processing such as feature extraction, style conversion, noise addition, etc. The first model parameter includes weight parameters and adjustment factors related to the degree of style conversion. The association information between the preset style conversion degree and the preset model parameters can be obtained by learning a large amount of sample data with different degrees of style conversion through the image processing model and summarizing manual experience.

[0139] It is understandable that different combinations of model parameter values ​​will cause the model to output images with different style transfer effects.

[0140] Exemplarily, the degree of style conversion is divided into ten levels from 1 to 10, or 100 levels from 1 to 100, and for each level, the corresponding preset model parameter value is recorded. The image generation system pre-stores a corresponding association relationship table or mapping rules between the degree of style conversion and the preset model parameters. When the image generation system receives the style conversion degree set by the user, such as the user selects the "first level" style conversion degree level through the slider on the image conversion interface, the image generation system will perform a search and matching operation based on the existing association information. Specifically, if the association information is stored in the form of an association relationship table, when obtaining the first model parameter, the preset model parameter value associated with the corresponding style conversion degree "first level" is directly searched in the association relationship table.

[0141] Step 520: Adjust the model parameters related to the degree of style conversion in the image processing model according to the first model parameters.

[0142] Therefore, the model parameters of the image processing model are adjusted in a targeted manner according to the style conversion degree specified by the user, so that the image processing model can process the first image according to the desired style conversion degree, thereby generating a second image whose style conversion effect precisely meets the user's needs, thereby improving the image quality of the second image.

[0143] In some other embodiments of the present application, the above-mentioned image conversion requirement information includes the style abstraction level, which refers to the degree to which the style of the second image deviates from concrete realism and tends to abstract expression. For example, a concrete style image can clearly distinguish the specific shape and details of an object, and has a low level of abstraction; while an abstract style image emphasizes the combination and expression of elements such as color, shape, and line, weakens the presentation of specific objects, and has a high level of abstraction. Based on this, after the above step 110, if Fig.11 As shown, the image generating method may include steps 610 to 620.

[0144] Step 610: Acquire a second model parameter associated with the style abstraction level according to the association information between the preset style abstraction level and the preset model parameters.

[0145] The second model parameters include weight parameters and adjustment factors related to the style abstraction level. The association information between the preset style abstraction level and the preset model parameters can be obtained by learning a large amount of sample data of styles with different abstraction levels through the image processing model and summarizing manual experience.

[0146] It is understandable that different combinations of model parameter values ​​will cause the model to output images with different levels of style abstraction.

[0147] Exemplarily, the style abstraction level is divided into ten levels from 1 to 10, or 100 levels from 1 to 100, and for each style abstraction level, it is associated with a preset model parameter combination matching the style abstraction level. When the image generation system receives the style abstraction level set by the user, the image generation system performs a search and acquisition operation according to the existing associated information.

[0148] Step 620: Adjust the model parameters related to the style abstraction level in the image processing model according to the second model parameters.

[0149] Therefore, the model parameters of the image processing model are adjusted in a targeted manner according to the style abstraction level set by the user, so that the image processing model can accurately generate images of the corresponding style according to the style abstraction level specified by the user, thereby improving the flexibility and effectiveness of the image generation method in dealing with different style abstraction level requirements, thereby improving the image quality of the second image.

[0150] Next, in step 120, both the global conversion network and the portrait conversion network adopt Fig.12 The shown structure is based on the deep convolutional network of the Visual Geometry Group (VGG) with 3 fully connected layers removed.

[0151] The style reference information and global image information are sent to the VGG model with 3 fully connected layers removed for forward propagation, and the style reference information and portrait image information are sent to the VGG model with 3 fully connected layers removed for forward propagation, and then the loss is calculated according to the loss function. In this process, the link parameters, style reference information, global image information and portrait image information of the VGG part with 3 fully connected layers removed are kept unchanged, and then the gradient descent method is used to back propagate to the output image, and the output image is adjusted to complete the model training. In the VGG model with 3 fully connected layers removed here, only the convolution layer is taken to participate in the calculation. Each convolution layer contains all the convolution kernels of the layer and their corresponding feature maps, and contains pooling operations. The fully connected layer contains regularization and residual link processing, and the output layer is normalized. During the model training process, no actual calculation is performed for the fully connected layer and softmax layer after the convolution layer. In addition, after calculating the style transfer abstraction level in the loss function, if the current and subsequent convolutional layers have all losses of 0, the actual propagation calculation will no longer be performed on them.

[0152] Next, the above-mentioned loss functions will be described in detail.

[0153] Image content conversion loss function L c (C, S, l) can be expressed by the following formula (7):

[0154]

[0155] Among them, l is the number of layers, n is the number of portraits detected by portrait detection, and C l represents the features extracted from the global sampling image when it propagates to layer l, P l represents the features extracted from the global noise image at layer l, C pi l represents the features extracted from the i-th portrait sample image at layer l, P pi l Represents the features extracted from the portrait noise image of the i-th portrait at layer l.

[0156] Image style transfer loss function L s (C, S, l) can be expressed by the following formula (8):

[0157]

[0158] Among them, l is the layer where it is located, n is the number of human figures detected through human figure detection, C p is the influence factor adjustment factor of the human figure model configured by the system, P l represents the features extracted from the global noise image at layer l, S l represents the features extracted from the style image at layer l, N l represents the number of features of the style image at layer l, M l represents the number of channels of the style image at layer l, S pi l represents the features extracted from the style image corresponding to the i-th human figure at layer l, P pi l represents the features extracted from the noise image of the i-th human figure at layer l.

[0159] The activation function si(l) at layer l can be represented by the following formula (9):

[0160]

[0161] Among them, P a represents the style conversion abstraction level, and its value is: P a ∈N(0 < x ≤ 100). When si(l) = 0, let L s (C, S, l) = 0, and the actual calculation of the loss function at layer l is no longer executed.

[0162] The loss function of image generation can be represented by the following formula (10):

[0163]

[0164] Then, referring to step 130, the fusion network is used to perform a fusion operation on the global style image and the human figure style image to generate an output image with better perspective effect that combines style features and has coordinated human figure areas, that is, the second image.

[0165] In some embodiments of the present application, as Fig.13 shown, the above step 130 may specifically include steps 1301 to 1303.

[0166] Step 1301, obtain the human figure area corresponding to the human figure area image in the global style image.

[0167] Exemplarily, when processing the first image, the boundary box coordinate information of the human figure area in the first image is determined through the human figure detection algorithm, the human figure area is determined, and then according to these boundary box coordinate information, the corresponding area in the global style image is intercepted or located, that is, the human figure area.

[0168] Step 1302: for each pixel in the portrait area, determine a target pixel value of the pixel according to a first pixel value of the pixel in the portrait style image and a second pixel value of the pixel in the global style image.

[0169] In one example, the target pixel value may be obtained by assigning different weights to the pixel values ​​of the portrait style image and the global style image and then calculating their weighted average.

[0170] In another example, a linear interpolation method may be used to find a new value between the first pixel value and the second pixel value as the target pixel value.

[0171] Step 1303 , mapping the target pixel value of each pixel point to the corresponding position of the portrait area to generate a second image.

[0172] Exemplarily, after calculating the target pixel values ​​of all pixels in the portrait area, the target pixel values ​​are assigned back to the corresponding pixel positions in the global style image one by one according to the original position information of each pixel in the portrait area. For example, if the coordinates of a pixel in the portrait area are (i, j), the calculated target pixel value is assigned to the pixel with coordinates (i, j) in the global style image.

[0173] Therefore, by calculating the target pixel value based on the first pixel value and the second pixel value and then generating the final fused image, it is ensured that the fused portrait area can be accurately embedded in the global style image, so that the generated second image not only retains the overall style characteristics of the global style image, but also shows a unique fusion style in the portrait area, thereby improving the image quality of the second image.

[0174] In some embodiments of the present application, the above step 1302 may specifically include:

[0175] For each pixel point in the portrait area, a weighted sum process is performed on the first pixel value and the second pixel value according to the third weight of the first pixel value and the fourth weight of the second pixel value to obtain a target pixel value of the pixel point.

[0176] Exemplarily, the target pixel value O' of (x+j, y+k) of the second image c (x+j, y+k) can be expressed by the following formula (11):

[0177] O' c (x+j,y+k)=O' c (x+j,y+k)(1-p)+O pi (j,k)p (11)

[0178] Wherein, x represents the starting position coordinate of the portrait style image in the horizontal direction in the style conversion network, y represents the starting position coordinate of the portrait style image in the vertical direction in the style conversion network, j represents the position coordinate of the portrait style image in the horizontal direction relative to the starting position coordinate, k represents the position coordinate of the portrait style image in the vertical direction relative to the starting position coordinate, and p represents the third weight of the first pixel value at the (j, k) position of the portrait style image.

[0179] Here, p represents the fusion ratio coefficient, which is used to control the fusion degree between the output result of the portrait conversion network and the output result of the style conversion network. When p = 1, it means using the result of the portrait conversion network completely; when p = 0, it means using the result of the style conversion network completely; when 0 < p < 1, it means the two are fused in a certain proportion.

[0180] Thus, the effective fusion of the global style image and the portrait style image is achieved, and the second image with high quality and coordinated style is generated.

[0181] In some embodiments of the present application, before the above step 1302, as Fig.14 shown, the above image generation method may further include steps 710 to 730.

[0182] Step 710, for each pixel point in the portrait area, determine the relative distance between the pixel point and the center point according to the position information of the pixel point and the position information of the center point of the portrait area.

[0183] Step 720, according to the association information between the preset relative distance and the preset weight, determine the weight corresponding to the relative distance as the third weight.

[0184] Exemplarily, the association information between the preset relative distance and the preset weight can be implemented through a weight mapping function. The weight mapping function can be represented by the following formula (12):

[0185]

[0186] Wherein, j represents the position coordinate of the portrait style image in the horizontal direction relative to the starting position coordinate, k represents the position coordinate of the portrait style image in the vertical direction relative to the starting position coordinate, w represents the width of the portrait style image in the horizontal direction, and h represents the height of the portrait style image in the vertical direction.

[0187] Here, the magnitude of the weight value p depends on the position of the point (j, k) relative to the center point. When the point (j, k) is close to the center point, the value of p is larger; when the point (j, k) is far from the reference point, the value of p is smaller.

[0188] Step 730, determine the weight corresponding to the third weight as the fourth weight.

[0189] Exemplarily, the sum of the third weight and the fourth weight is 1. After the third weight p is determined, the fourth weight is determined to be 1-p. For example, if the third weight is 0.6, the fourth weight is 0.4.

[0190] Therefore, the relative distance between the pixel point and the center point provides a basis for the weight distribution of pixels at different positions in the portrait area during fusion. Specifically, the pixel points closer to the center point may be more likely to be affected by the portrait style image, while the pixel points farther from the center point are more likely to be fused with the global style image. By determining the third weight based on the relative distance, the influence of the portrait style image in the fusion is dynamically adjusted according to the position of the pixel point in the portrait area, so that the portrait area in the fused second image presents a natural transition effect in style from the center to the edge, avoiding the abrupt transition between the portrait style image and the global style image at the boundary, and improving the visual quality and naturalness of the fusion of the portrait style image and the global image.

[0191] Based on the image generation method provided in the above embodiment, the present application also provides a specific implementation of the image generation device. Please refer to the following embodiment.

[0192] See also Fig.15 , the image generating device 800 provided in the embodiment of the present application includes:

[0193] A first acquisition module 810 is used to acquire a first image and image conversion requirement information, where the first image includes a portrait area image;

[0194] A first processing module 820 is used to process the style reference information and the global image information of the first image through a global conversion network of the image processing model to obtain a global style image, and to process the style reference information and the portrait image information of the portrait region image through a portrait conversion network of the image processing model to obtain a portrait style image;

[0195] The fusion module 830 is used to fuse the global style image and the portrait style image through a fusion network of the image processing model to obtain a second image.

[0196] Thus, the first image acquired by the first acquisition module 810 and the style reference information determined based on the image conversion requirement information are processed using the global conversion network of the image processing model, and the style reference information and the portrait image information are processed using the portrait conversion network of the image processing model, which effectively ensures that the global conversion network of the first processing module 820 can process the global image information in the process of processing the global image information, and the portrait conversion network can process the portrait image information in the process of processing the same style reference information, that is, the same style target. Then, the second image obtained by fusing the global style image and the portrait style image with the help of the fusion network of the fusion module 830 can combine the global style with the portrait style, and can be coordinated in the overall style. In this way, while controlling the overall conversion style of the image, the characteristics of the portrait area in the image are taken into account, and the image quality of the image after style conversion is effectively improved.

[0197] In some embodiments of the present application, the image generating device 800 may further include:

[0198] A second processing module is used to process the first image and the image conversion requirement information through an image processing network of the image processing model to obtain style reference information, global image information of the first image, and portrait image information of the portrait region image, wherein the style reference information is information related to a desired conversion style for the first image;

[0199] In some embodiments of the present application, the second processing module may be specifically used for:

[0200] In the case where the image conversion requirement information includes style requirement information, the global image information includes a global sampled image and a global noise image, and the portrait image information includes a portrait sampled image and a portrait noise image, inputting the first image and the image conversion requirement information into an image processing network, processing the style requirement information through the image processing network to obtain style reference information; and performing sampling processing on the first image to obtain a global sampled image;

[0201] Extracting a portrait sampling image from a global sampling image;

[0202] Using style reference information, the global sampled image is subjected to noise filling processing to obtain a global noise image.

[0203] Extract portrait noise image from global noise image.

[0204] In some embodiments of the present application, the second processing module may be specifically used for:

[0205] In the case where the style requirement information includes a style conversion type, a first style feature vector associated with the style conversion type is determined as the style reference information according to association information between a preset style conversion type and a preset style feature vector.

[0206] In some embodiments of the present application, the second processing module may be specifically used for:

[0207] When the style requirement information includes a style reference image, compressing the style reference image according to a preset size to obtain a style compressed image;

[0208] Vectorizing the style compressed image to obtain a second style feature vector;

[0209] The second style feature vector is determined as the style reference information.

[0210] In some embodiments of the present application, the second processing module may be specifically used for:

[0211] Perform grayscale conversion on the global sampling image to obtain a global grayscale image;

[0212] Based on the preset portrait feature information, portrait detection is performed on the global grayscale image to determine the portrait area image;

[0213] According to the portrait area image, a portrait sampling image is extracted from the global sampling image.

[0214] In some embodiments of the present application, the second processing module may be specifically used for:

[0215] Generate a blank image corresponding to the global sampling image according to the size information and channel information of the global sampling image;

[0216] Based on the reference noise parameter, the initial pixel value of the blank image is adjusted to obtain a global noise image, and the reference noise parameter is determined based on user feedback information.

[0217] In some embodiments of the present application, the above fusion module 830 may be specifically used for:

[0218] After the user feedback information includes the first user feedback value and the global style image and the portrait style image are fused through the fusion network of the image processing model to obtain a second image, the second image is displayed in a display area of ​​the style conversion interface, and the style conversion interface includes a scoring option, and the scoring option is used to prompt the user to quantify and score the conversion effect of the second image;

[0219] In response to a first input of a scoring option, obtaining a first user feedback value corresponding to the scoring option;

[0220] The reference noise parameter is updated according to the first user feedback value.

[0221] In some embodiments of the present application, the image generating device 800 further includes:

[0222] A first display module is used for displaying the second image in a display area of ​​a style conversion interface after the user feedback information includes a second user feedback value and the global style image and the portrait style image are fused through a fusion network of an image processing model to obtain a second image, wherein the style conversion interface includes a download option, and the download option is used to prompt the user to download the second image;

[0223] A second acquisition module, configured to acquire a second user feedback value corresponding to the download option in response to a second input of the download option;

[0224] The first updating module is used to update the reference noise parameter according to the second user feedback value.

[0225] In some embodiments of the present application, the image generating device 800 further includes:

[0226] A third acquisition module is used for, after fusing the global style image and the portrait style image through the fusion network of the image processing model to obtain the second image, and when the style conversion interface includes a rating option and a download option, obtaining a first user feedback value corresponding to the rating option and a second user feedback value corresponding to the download option;

[0227] A first determining module, configured to perform weighted summation of the first user feedback value and the second user feedback value according to a first weight of the first user feedback value and a second weight of the second user feedback value to obtain a target feedback value;

[0228] The second updating module is used to update the reference noise parameter according to the target feedback value.

[0229] In some embodiments of the present application, the image generating device 800 further includes:

[0230] a fourth acquisition module, configured to acquire, after the image conversion requirement information includes the style conversion degree and the first image and the image conversion requirement information are acquired, a first model parameter associated with the style conversion degree according to association information between a preset style conversion degree and a preset model parameter;

[0231] The first adjustment module is used to adjust the model parameters related to the degree of style conversion in the image processing model according to the first model parameters.

[0232] In some embodiments of the present application, the image generating device 800 further includes:

[0233] a fifth acquisition module, configured to acquire, when the image conversion requirement information further includes a style abstraction level and after acquiring the first image and the image conversion requirement information, acquire, according to association information between a preset style abstraction level and a preset model parameter, a second model parameter associated with the style abstraction level;

[0234] The second adjustment module is used to adjust the model parameters related to the style abstraction level in the image processing model according to the second model parameters.

[0235] In some embodiments of the present application, the above fusion module 830 may be specifically used for:

[0236] Obtaining a portrait region corresponding to the portrait region image in the global style image;

[0237] For each pixel in the portrait area, determining a target pixel value of the pixel according to a first pixel value of the pixel in the portrait style image and a second pixel value in the global style image;

[0238] The target pixel value of each pixel is mapped to the corresponding position of the portrait area to generate a second image.

[0239] In some embodiments of the present application, the fusion module 830 may be specifically used for:

[0240] For each pixel point in the portrait area, a weighted sum process is performed on the first pixel value and the second pixel value according to the third weight of the first pixel value and the fourth weight of the second pixel value to obtain a target pixel value of the pixel point.

[0241] In some embodiments of the present application, the fusion module 830 may be specifically used for:

[0242] For each pixel in the portrait area, determining a relative distance between the pixel and the center point according to the position information of the pixel and the position information of the center point of the portrait area;

[0243] According to the association information between the preset relative distance and the preset weight, the weight corresponding to the relative distance is determined as the third weight;

[0244] The weight corresponding to the third weight is determined as the fourth weight.

[0245] The various modules of the image generation device 800 provided in the embodiment of the present application can be implemented Figures 1 to 14 The functions of each step of the image generation method 800 are provided and can achieve the corresponding technical effects, which will not be described in detail here for the sake of brevity.

[0246] Fig.16 A schematic diagram of the hardware structure of an electronic device provided in some embodiments of the present application is shown.

[0247] The electronic device may include a processor 901 and a memory 902 storing computer program instructions.

[0248] Specifically, the processor 901 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0249] The memory 902 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 902 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 902 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 902 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 902 is a non-volatile solid-state memory.

[0250] In a particular embodiment, the memory 902 may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Therefore, generally, the memory 902 includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the data processing method according to the first aspect of the present application.

[0251] The processor 901 implements any one of the image generating methods in the above embodiments by reading and executing computer program instructions stored in the memory 902 .

[0252] In one example, the electronic device may further include a communication interface 909 and a bus 910. Fig. 9 As shown, the processor 901, the memory 902, and the communication interface 909 are connected via a bus 910 and communicate with each other.

[0253] The communication interface 909 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0254] Bus 910 includes hardware, software or both, and the parts of electronic equipment are coupled to each other.For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 910 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.

[0255] The electronic device can execute the image generation method in the embodiment of the present application, thereby realizing the combination Figures 1 to 15 The image generation method and device described.

[0256] In addition, in combination with the image generation method in the above embodiment, the embodiment of the present application may provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any one of the image generation methods in the above embodiment is implemented. Examples of computer-readable storage media include non-transitory computer-readable storage media, such as portable disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, etc.

[0257] In addition, in combination with the image generation method in the above embodiment, the embodiment of the present application may provide a computer program product for implementation. The program product is stored in a storage medium, and may specifically include a computer program or instruction, which implements any of the image generation methods in the above embodiment when executed by a processor. The program product is executed by at least one processor to implement the various processes of the above data processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0258] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0259] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0260] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0261] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0262] The above are only specific implementation methods of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. An image generation method, characterized in that: include: Acquire a first image and image conversion requirement information, wherein the first image includes a portrait area image; Processing the style reference information and the global image information of the first image through a global conversion network of the image processing model to obtain a global style image, and processing the style reference information and the portrait image information of the portrait region image through a portrait conversion network of the image processing model to obtain a portrait style image, wherein the style reference information is determined based on the image conversion requirement information; The global style image and the portrait style image are fused through a fusion network of the image processing model to obtain a second image.

2. The method according to claim 1, characterized in that The method further comprises: The first image and the image conversion requirement information are processed through the image processing network of the image processing model to obtain the style reference information, the global image information of the first image and the portrait image information of the portrait area image.

3. The method according to claim 2, characterized in that The image conversion requirement information includes style requirement information, the global image information includes a global sampling image and a global noise image, and the portrait image information includes a portrait sampling image and a portrait noise image; The step of processing the first image and the image conversion requirement information through the image processing network of the image processing model to obtain the style reference information, the global image information of the first image, and the portrait image information of the portrait region image includes: Inputting the first image and the image conversion requirement information into the image processing network, processing the style requirement information through the image processing network to obtain the style reference information; and performing sampling processing on the first image to obtain the global sampling image; Extracting the portrait sampling image from the global sampling image; Performing noise filling processing on the global sampling image according to the style reference information to obtain the global noise image; The portrait noise image is extracted from the global noise image.

4. The method according to claim 3, characterized in that: The style requirement information includes a style conversion type; the processing of the style requirement information to obtain the style reference information includes: According to association information between a preset style conversion type and a preset style feature vector, a first style feature vector associated with the style conversion type is determined as the style reference information.

5. The method according to claim 3, characterized in that: The style requirement information includes a style reference image; and the processing of the style requirement information to obtain the style reference information includes: Compressing the style reference image according to a preset size to obtain a style compressed image; performing vectorization processing on the style compression image to obtain a second style feature vector; The second style feature vector is determined as the style reference information.

6. The method according to claim 3, characterized in that The step of extracting the portrait sampling image from the global sampling image comprises: Performing grayscale conversion processing on the global sampling image to obtain a global grayscale image; Based on preset portrait feature information, performing portrait detection on the global grayscale image to determine the portrait area image; According to the portrait area image, the portrait sampling image is extracted from the global sampling image.

7. The method according to claim 3, characterized in that The step of performing noise filling processing on the global sampled image by using the style reference information to obtain the global noise image includes: Generate a blank image corresponding to the global sampling image according to the size information and channel information of the global sampling image; The initial pixel value of the blank image is adjusted based on a reference noise parameter to obtain the global noise image, wherein the reference noise parameter is determined based on user feedback information.

8. The method according to claim 7, characterized in that The user feedback information includes a first user feedback value; after fusing the global style image and the portrait style image through the fusion network of the image processing model to obtain a second image, the method further includes: Displaying the second image in a display area of ​​a style conversion interface, wherein the style conversion interface includes a scoring option, wherein the scoring option is used to prompt a user to quantify and score a conversion effect of the second image; In response to a first input of the scoring option, obtaining a first user feedback value corresponding to the scoring option; The reference noise parameter is updated according to the first user feedback value.

9. The method according to claim 7 or 8, characterized in that: The user feedback information includes a second user feedback value; after fusing the global style image and the portrait style image through the fusion network of the image processing model to obtain the second image, the method further includes: Displaying the second image in a display area of ​​a style conversion interface, wherein the style conversion interface includes a download option, and the download option is used to prompt a user to download the second image; In response to a second input of the download option, obtaining a second user feedback value corresponding to the download option; The reference noise parameter is updated according to the second user feedback value.

10. The method according to claim 9, characterized in that After fusing the global style image and the portrait style image through the fusion network of the image processing model to obtain a second image, the method further includes: In a case where the style conversion interface includes a rating option and a download option, obtaining a first user feedback value corresponding to the rating option and a second user feedback value corresponding to the download option; performing a weighted summation of the first user feedback value and the second user feedback value according to a first weight of the first user feedback value and a second weight of the second user feedback value to obtain a target feedback value; The reference noise parameter is updated according to the target feedback value.

11. The method according to any one of claims 1 to 8, characterized in that: The image conversion requirement information includes a style conversion degree; after obtaining the first image and the image conversion requirement information, the method further includes: According to association information between a preset style conversion degree and a preset model parameter, obtaining a first model parameter associated with the style conversion degree; According to the first model parameters, model parameters related to the degree of style conversion in the image processing model are adjusted.

12. The method according to claim 11, characterized in that The image conversion requirement information also includes a style abstraction level; after obtaining the first image and the image conversion requirement information, the method further includes: According to association information between a preset style abstraction level and a preset model parameter, obtaining a second model parameter associated with the style abstraction level; According to the second model parameters, the model parameters related to the style abstraction level in the image processing model are adjusted.

13. The method according to any one of claims 1 to 8, characterized in that: The fusing the global style image and the portrait style image to obtain a second image includes: Acquire a portrait region corresponding to the portrait region image in the global style image; For each pixel in the portrait area, determining a target pixel value of the pixel according to a first pixel value of the pixel in the portrait style image and a second pixel value of the pixel in the global style image; The target pixel value of each pixel is mapped to a corresponding position in the portrait area to generate the second image.

14. The method according to claim 13, characterized in that The step of determining, for each pixel in the portrait area, a target pixel value of the pixel according to a first pixel value of the pixel in the portrait style image and a second pixel value of the pixel in the global style image includes: For each pixel point in the portrait area, determining a relative distance between the pixel point and the center point according to the position information of the pixel point and the position information of the center point of the portrait area; According to association information of a preset relative distance and a preset weight, determining a weight corresponding to the relative distance as a third weight of the first pixel value; Determine a weight corresponding to the third weight as a fourth weight of the second pixel value; According to the third weight of the first pixel value and the fourth weight of the second pixel value, weighted sum processing is performed on the first pixel value and the second pixel value to obtain the target pixel value of the pixel point.

Images