Image drawing method, image drawing apparatus, and storage medium

Abstract

The invention relates to an image drawing method, an image drawing device and a storage medium. The image drawing method comprises the steps of obtaining a first image; in response to the detected operation of selecting a regional image on the first image, calling an image style model, and carrying out stylization processing on the regional image to obtain a second image; and fusing the second image at the regional image of the first image to obtain a target image. According to the invention, the local creation of the image is realized, the local editing style is the same as the original image style, and the preference and demand of the user are met.

Description

Technical Field

[0001] The present disclosure relates to model training, personalized style image editing and generation, cloud / mobile model training and reasoning, and related artificial intelligence technology fields, and in particular to an image drawing method, an image drawing device, and a storage medium. Background Art

[0002] Artificial Intelligence (AI) painting and AI generation technology are one of the hot research directions in the field of artificial intelligence in recent years. AI painting technology refers to the use of artificial intelligence technology to generate new paintings, including hand-painted and digital paintings. This technology can learn the style and skills of artists by training models to generate images with artistic value.

[0003] In general, the research status of AI painting and AI portrait generation technology is still relatively preliminary, but with the continuous development of artificial intelligence technology, I believe that research in this field will become more and more in-depth, and more commercial applications may come out in the future. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides an image drawing method, an image drawing device and a storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an image drawing method, the method comprising acquiring a first image; in response to detecting an operation of selecting a region image on the first image, calling an image style model to perform stylization processing on the region image to obtain a second image; and fusing the second image at the region image of the first image to obtain a target image.

[0006] In one embodiment, calling the image style model to perform stylized processing on the region image to obtain a second image includes: in response to the first image being generated based on the first image style model, calling the first image style model to perform stylized processing on the region image to obtain the second image; or in response to the first image not being generated based on the image style model, training the image style model based on the first image, and performing stylized processing on the region image based on the trained image style model to obtain the second image.

[0007] In one embodiment, the image style model is obtained by training based on the first image, including: cutting the first image according to set edges at a preset ratio, and copying the first image to obtain a style image sample set with a preset number; determining the style image sample set and the first image as a style image set; and obtaining an image style model by training based on the style image set.

[0008] In one embodiment, the training of the image style model based on the style image set includes: calling a first pre-trained model, and training a second pre-trained model based on the style image set; based on the first pre-trained model and the second pre-trained model, repeatedly training each style image in the style image set for a preset number of times to obtain the image style model.

[0009] In one embodiment, the method further includes: in response to detecting an operation of selecting a regional image on the first image, displaying local creation options, the local creation options including image local creation style options and image local creation text description options; the stylized processing of the regional image includes: based on the image local creation style options selected by the user, determining the target image local creation style, and obtaining the target image local creation text description input by the user based on the image local creation text description option; based on the target image local creation style and the target image local creation text description, the stylized processing of the regional image.

[0010] In one embodiment, the local creation option also includes an image creation modification amplitude option, and the image creation modification amplitude option is used to determine the similarity between the second image style and the first image style; the method also includes: based on the image creation modification amplitude corresponding to the image creation modification amplitude option, generating a second image matching the similarity style corresponding to the image creation modification amplitude.

[0011] In one embodiment, before fusing the second image at the regional image of the first image, the method further includes: optimizing the initial target image and displaying the optimized target image; the optimization includes at least one of the following: filtering out target content information; image area repair; redrawing based on a user's redrawing operation; and image enhancement.

[0012] In one embodiment, the fusion of the second image at the area image of the first image to obtain the target image includes: determining the contour edge of the area image in the first image; fusing the second image at the area image of the first image according to the contour edge, and performing fusion optimization processing on the contour edge to obtain the target image; the fusion optimization processing includes at least one of the following: image edge enhancement processing; image edge smoothing processing; image clarity processing.

[0013] According to a second aspect of an embodiment of the present disclosure, there is provided an image drawing device, comprising: an acquisition unit, for acquiring a first image; a calling unit, for invoking an image style model in response to detecting an operation of selecting a region image on the first image, and performing stylization processing on the region image to obtain a second image; and a determination unit, for fusing the second image at the region image of the first image to obtain a target image.

[0014] In one embodiment, the calling unit calls the image style model in the following manner to perform stylized processing on the region image to obtain a second image, including: in response to the first image being generated based on the first image style model, calling the first image style model to perform stylized processing on the region image to obtain the second image; or in response to the first image not being generated based on the image style model, training the image style model based on the first image, and performing stylized processing on the region image based on the trained image style model to obtain the second image.

[0015] In one embodiment, the calling unit obtains the image style model based on the first image training in the following manner, including: cutting the first image along a set edge at a preset ratio, and copying the first image to obtain a style image sample set with a preset number; determining the style image sample set and the first image as a style image set; and training the image style model based on the style image set.

[0016] In one embodiment, the calling unit trains an image style model based on the style image set in the following manner, including: calling a first pre-trained model, and training a second pre-trained model based on the style image set; based on the first pre-trained model and the second pre-trained model, repeatedly training each style image in the style image set for a preset number of times to obtain an image style model.

[0017] In one embodiment, the calling unit is also used for: in response to detecting an operation of selecting a regional image on the first image, displaying local creation options, the local creation options including image local creation style options and image local creation text description options; the stylized processing of the regional image includes: determining the target image local creation style based on the image local creation style option selected by the user, and obtaining the target image local creation text description input by the user based on the image local creation text description option; and stylized processing of the regional image based on the target image local creation style and the target image local creation text description.

[0018] In one embodiment, the local creation option also includes an image creation modification amplitude option, and the image creation modification amplitude option is used to determine the similarity between the second image style and the first image style; the determination unit is also used to: based on the image creation modification amplitude corresponding to the image creation modification amplitude option, generate a second image matching the similarity style corresponding to the image creation modification amplitude.

[0019] In one embodiment, before fusing the second image at the regional image of the first image, the determination unit is also used to: optimize the initial target image and display the optimized target image; the optimization process includes at least one of the following: screening out target content information; image area repair; redrawing based on a user's redrawing operation; image enhancement.

[0020] In one embodiment, the determination unit fuses the second image at the area image of the first image to obtain a target image in the following manner, including: determining a contour edge of the area image in the first image; fusing the second image at the area image of the first image according to the contour edge, and performing fusion optimization processing on the contour edge to obtain a target image; the fusion optimization processing includes at least one of the following: image edge enhancement processing; image edge smoothing processing; image clarity processing.

[0021] According to a third aspect of an embodiment of the present disclosure, there is provided an image drawing device, comprising: a processor; and a memory for storing processor executable instructions; wherein the processor is configured to: execute the image drawing method described in the first aspect or any one of the implementations of the first aspect.

[0022] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided, in which instructions are stored. When the instructions in the storage medium are executed by a processor, the processor is enabled to execute the image drawing method described in the first aspect or any one of the embodiments of the first aspect.

[0023] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: obtaining a first image, in response to detecting an operation of selecting a regional image in the first image, calling an image style model, stylizing the regional image to obtain a second image. Fusing the regional image of the first image with the second image to determine it as a target image. Through the present disclosure, local editing and creation of images are achieved.

[0024] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0026] Figure 1 The figure is a flowchart of an image drawing method according to an exemplary embodiment.

[0027] Figure 2 The figure is a flowchart showing a method for determining a second image according to an exemplary embodiment.

[0028] Figure 3 The figure is a flowchart of a method for determining an image style model according to an exemplary embodiment.

[0029] Figure 4 The figure is a flowchart of a method for obtaining an image style model according to an exemplary embodiment.

[0030] Figure 5 It is a schematic diagram of a model training method according to an exemplary embodiment.

[0031] Figure 6 The figure is a flow chart of a method for processing a regional image according to an exemplary embodiment.

[0032] Figure 7 The invention is a flowchart of a method for modifying an image creation range according to an exemplary embodiment.

[0033] Figure 8 The figure is a flow chart of an image fusion method according to an exemplary embodiment.

[0034] Fig. 9 The figure is a flowchart of an image drawing method according to an exemplary embodiment.

[0035] Fig.10 The figure is a block diagram of an image drawing device according to an exemplary embodiment.

[0036] Fig.11 It is a block diagram of a device for image drawing according to an exemplary embodiment. DETAILED DESCRIPTION

[0037] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure.

[0038] AI image generation technology refers to the method of generating images using artificial intelligence technology. At present, AI image generation technology has made great progress. In the field of AI image generation, you can choose GAN-based image generation technology: GAN is a generative model that consists of a generator and a discriminator. The generator is used to generate images, and the discriminator is used to determine whether the generated image is real. By continuously training the generator and the discriminator, GAN can generate high-quality images. Or VAE-based image generation technology: VAE (Variational Autoencoder) is an autoencoder that can encode the input image into a latent vector and then decode the latent vector into an image. By adjusting the value of the latent vector, VAE can generate different images. Images can also be generated based on deep learning image generation technology and transfer learning image generation technology.

[0039] Among the related technologies, the research status of AI image editing and AI partial creation technology is still relatively preliminary. It does not allow users to upload personal images for secondary creation, and only allows online graffiti for creation. It does not support text-guided drawing, nor does it support partial editing of images.

[0040] Furthermore, in the related technologies, it is impossible to support the consistency of the local creation effect with the style of the overall image, and the actual raw image effect cannot meet practical needs. The generated image may lack details and realism. In local creation or editing of local areas of the image, the connection and transition of the image editing edges cannot be met, the details are poorly rendered, and the generated part has a false feeling. The generated image is distorted and cannot be integrated with the original image. When creating locally and processing relatively complex scenes or details, there is a problem of unreasonable generated images.

[0041] In view of this, the present disclosure proposes an image drawing method.

[0042] The following describes an image drawing method in an embodiment of the present disclosure.

[0043] Figure 1 is a flowchart of an image drawing method according to an exemplary embodiment. Figure 1 As shown, the method includes the following.

[0044] In step S11, a first image is acquired.

[0045] In the embodiment of the present disclosure, a first image input by a user is acquired, wherein the first image may be a graffiti image of the user or an image locally uploaded by the user.

[0046] In step S12, in response to detecting an operation of selecting a region image on the first image, an image style model is called to perform stylization processing on the region image to obtain a second image.

[0047] In the embodiment of the present disclosure, in response to detecting that a user has made a partial selection on a first image using a brush, where the partially selected area can be understood as a regional image, when performing an operation of selecting a regional image, the mobile terminal calls an image style model to perform stylization processing on the regional image selected by the user, and the obtained stylized regional image is determined as the second image. The stylization processing can be understood as performing style creation.

[0048] In step S13, the second image is fused at the region image of the first image to obtain a target image.

[0049] In the disclosed embodiment, the target image can be understood as fusing the second image obtained by stylizing the area image selected by the user with the area image of the first image, and determining the image obtained by fusing with the area image of the first image as the target image.

[0050] In one example, a first image is obtained, and it is detected that the user has performed secondary creation on the first image. For example, if you upload an image with a person, and you want to draw a gun in the person's hand, you select the person's hand part on the first image, and the person's hand part can be understood as a regional image. Because the selected regional image, that is, the gun in the person's hand, is different in style from the first image, the image style model can be called to perform stylized processing on the regional image to obtain a second image, so that the regional image part has the same style as the first image. The second image is fused with the first image to obtain the target image of a person holding a gun.

[0051] In the disclosed embodiment, a second image is obtained based on stylized processing of a regional image, and the second image is fused to the regional image of the first image, so that a local editing effect having the same style as the first image can be achieved, thereby ensuring high-quality editing effects and realizing local creation.

[0052] The method for determining the second image is described below in the embodiment of the present disclosure.

[0053] Figure 2 is a flowchart of a method for determining a second image according to an exemplary embodiment. Figure 2 As shown, the method includes the following.

[0054] In step S21, the image style model is called to perform stylization processing on the regional image.

[0055] In the disclosed embodiment, based on the region image selected by the user on the first image, an image style model is called to perform stylized processing on the region image, wherein the image style model may be a default image style model or a trained image style model.

[0056] In step S22a, in response to the first image being generated based on the first image style model, the first image style model is called to perform stylization processing on the regional image.

[0057] In the disclosed embodiment, if the user's first image is stylized based on a graffiti work, since the graffiti work has achieved model training for style creation, or a default style model is used, the first image style model can be directly called to stylize the regional image.

[0058] In step S22b, in response to the first image not being generated based on the image style model, an image style model is trained based on the first image, and stylization processing is performed on the regional image based on the trained image style model.

[0059] In the disclosed embodiment, if the first image is an image uploaded locally by a user or obtained by other means, that is, it is not generated based on an image style model, an image style model is trained based on the first image, and the regional image is stylized based on the trained image style model.

[0060] In step S23, a second image is obtained.

[0061] In the disclosed embodiment, different image style models are selected based on different first images to perform stylized processing on regional images, thereby implementing an adaptive style algorithm, satisfying user needs for local image drawing, and improving user creation efficiency and quality.

[0062] The following is an explanation of determining an image style model in the disclosed embodiment.

[0063] Figure 3 is a flowchart of a method for determining an image style model according to an exemplary embodiment. Figure 3 As shown, the method includes the following.

[0064] In step S31, the first image is cut along set edges at a preset ratio, and the first image is copied to obtain a style image sample set having a preset number.

[0065] In the disclosed embodiment, the first image input by the user is obtained as the original data to obtain an image sample set. Data preprocessing is performed on the first image, the first image is cut at a set edge according to a preset ratio, and the first image is copied, where copying can be understood as repetition. The preset number of images obtained after cutting and copying are determined as the style image sample set. In one example, for the first image, the set edge can be 1 / 2, 1 / 3, 1 / 4, 1 / 8, and the preset ratio can be 2, 4, 4, 8. That is, random cutting is performed at the ratio of 1 / 2, 1 / 3, 1 / 4, 1 / 8 of the set edge, and 2, 4, 4, and 8 images are generated respectively, so there are a total of 18 images. And the first image is copied once to obtain a style image sample set. The data involved above are only exemplary descriptions and are not limited to the specific data.

[0066] In step S32, the style image sample set and the first image are determined as a style image set.

[0067] In the disclosed embodiment, based on cutting the first image along set edges at a preset ratio and copying the first image, an image set consisting of a preset number of style image sample sets and the first image is determined as a style image set.

[0068] In step S33, an image style model is trained based on the style image set.

[0069] In the disclosed embodiment, the first image uploaded by the user is used as the original data to generate a style image set, which reduces the training time of the model and can complete the image style model training at a faster speed.

[0070] The following is an explanation of how to obtain an image style model in the disclosed embodiment.

[0071] Figure 4 is a flowchart of a method for obtaining an image style model according to an exemplary embodiment. Figure 4 As shown, the method includes the following.

[0072] In step S41, the first pre-trained model is called, and the second pre-trained model is trained based on the style image set.

[0073] In the embodiments of the present disclosure, the first pre-trained model can be understood as a basic model. In one example, the basic model can be an open source model, and the open source model is fine-tuned on a large scale through a style image set, and the fine-tuned model is used as the basic model. The second pre-trained model is obtained by training based on the style image set. Among them, the second pre-trained model can be understood as a custom style model.

[0074] In step S42, based on the first pre-trained model and the second pre-trained model, each style image in the style image set is repeatedly trained for a preset number of times to obtain an image style model.

[0075] In one example, based on the first pre-trained model and the second pre-trained model, the style image set is repeatedly trained for a preset number of times. Among them, the settings of the preset parameters for training can be the following values: the number of repetitions for each image in the style image set is 8, the number of samples used for training in a single pass is set to 1, the rank of the training model is 32, the number of model training rounds is 4, and after the training is completed, the model with the smallest loss function (loss) after the last three rounds of training is compared as the final model. In order to further speed up the image generation speed on the terminal side without affecting the timeliness of reasoning, the second pre-trained model and the first pre-trained model after training are quantized into integer (Int) 8 and floating point (Float) 16, and the quantized models are sent to the mobile terminal. The above data involved are only exemplary descriptions and are not limited to specific data.

[0076] In one example, Figure 5 FIG. 1 is a schematic diagram of a model training method according to an exemplary embodiment. Figure 5 As shown, a first pre-trained model and a second pre-trained model are obtained based on a style image set, and the first pre-trained model and the second pre-trained model are trained based on preset parameter settings to obtain an image style model.

[0077] In the disclosed embodiment, it is estimated that the training time is shortened when a single process is not queued. That is, after the user selects this function and uploads the first image, the data is automatically uploaded to the cloud and training is started. When the user is familiar with the user interface and the tutorial prompts, the training of the image style model is basically completed. And through quantization, the amount of calculation is reduced, the calculation speed is accelerated, and the training time is reduced. The prompt can continuously train the model, and the quality and diversity of the generated images can be improved based on the continuous training model, which has strong scalability.

[0078] The following is an explanation of the processing area image in the embodiment of the present disclosure.

[0079] Figure 6 is a flowchart of a method for processing a regional image according to an exemplary embodiment. Figure 6 As shown, the method includes the following.

[0080] In step S51, in response to detecting an operation of selecting an area image on a first image, local creation options are displayed, the local creation options including image local creation style options and image local creation text description options.

[0081] In the embodiment of the present disclosure, when the user selects a region image on the first image, the mobile terminal displays a local creation option. Local creation can be understood as the user selecting a partial region on the first image as a region image with a brush and creating the region. The local creation option includes an image local creation style option and an image local creation text description option. It can be understood that the local creation option supports style generation and text-guided generation.

[0082] In step S52, stylization processing is performed on the region image.

[0083] In the disclosed embodiment, local editing and local creation of an image are implemented based on a local creation style option and a local creation text description option, and the regional image is stylized so that the regional image adapts to the style of the first image.

[0084] In step S53, based on the image local creation style option selected by the user, the target image local creation style is determined, and the target image local creation text description input by the user based on the image local creation text description option is obtained.

[0085] In the embodiment of the present disclosure, the local creation style of the image selected by the user is used as the local creation style of the target image. For example, the local creation style options include animation, cartoon, oil painting and other local creation styles. If the user selects the oil painting option, the oil painting style is used as the local creation style of the target image. The local creation text description of the image input by the user is used as the local creation text description of the target image. For example, the user can input "black kitten" as the local creation text description of the target image.

[0086] In step S54, the regional image is stylized based on the local creation style of the target image and the local creation text description of the target image.

[0087] In the disclosed embodiment, local creation is implemented based on the selected local creation style of the target image and the text description of the local creation of the target image, so that the regional image is adapted to the first image style. For example, based on the selected local creation style of the target image and the text description of "black kitten", a black kitten with the local creation style of the target image can be generated.

[0088] In the disclosed embodiment, the regional image can be stylized by supporting text guidance, and the regional image can be stylized by combining the local creation style of the target image and the local creation text description of the target image, and AI creation, modification, and repair can be performed on the local area until the user is satisfied with the painting effect. The practical needs of users are met, and the efficiency and quality of user creation are improved.

[0089] The following is an explanation of the modification range of image creation in the embodiment of the present disclosure.

[0090] Figure 7 is a flow chart of a method for modifying the amplitude of image creation according to an exemplary embodiment. Figure 7 As shown, the method includes the following.

[0091] In step S61, the local creation options also include image creation modification range options.

[0092] In the disclosed embodiment, the local creation options include, in addition to the image local creation style options and the image local creation text description options, an image creation modification range option may also be included.

[0093] In step S62, based on the image creation modification amplitude corresponding to the image creation modification amplitude option, a second image matching the similarity style corresponding to the image creation modification amplitude is generated.

[0094] In the disclosed embodiment, during the process of generating the target image on the mobile terminal, the user is allowed to perform multiple repeated image generation operations, and the image creation modification range option can also provide the user with controllable AI creation capabilities. Among them, the image creation modification range option is mainly used to modify the regional image and generate a second image that matches the style of the image creation modification range corresponding to the similarity.

[0095] In one example, if the first image is a landscape image of a grassland, if the grassland is selected as the regional image, and several horses are drawn on the regional image. Based on the horses drawn by the user, if the user clicks the image creation modification amplitude option at this time to change the amplitude, the lower the amplitude value, the closer it is to the style of the horses drawn by the user, and the higher the amplitude value, the closer the style of the regional image part is to the style of the first image, and the similarity changes with the change of the image creation modification amplitude.

[0096] In the disclosed embodiment, the needs of AI painting creation are met by controlling the image creation modification amplitude, and the system is customizable, so that users can achieve the effect of AI painting by adjusting the image creation modification amplitude parameters.

[0097] The optimization process is described below in the embodiments of the present disclosure.

[0098] In the disclosed embodiment, before fusing the second image at the region image of the first image, an initial target image is generated, wherein the initial target image can be understood as the style image generated for the first time, and the initial target image will not be directly displayed to the user, and the initial target image needs to be optimized and displayed after the optimization. The optimization includes at least one of the following: filtering out target content information, image region repair, redrawing based on the user's redrawing operation, and image enhancement.

[0099] In the disclosed embodiment, the initial target image needs to be screened for target content information, i.e., risk detection. The content information may be privacy, violence, pornography, and other related information. The screening of target content information is mainly controlled by the prompt words in the target image generation process. The mobile terminal will detect the text description content of the user-defined prompt words, and identify it through the image recognition of the current step. After ensuring that the generated target image is a regular image, it is displayed to the user. It is understandable that if the user is not satisfied with the generation effect, the user can return to regenerate the target image.

[0100] In the disclosed embodiment, the optimization processing of the initial target image also includes image area repair, and the mobile terminal repairs the local area of ​​the initial target image, wherein the local area repair includes edge repair and proportion repair. In edge repair, the edge position of the image can be repaired using but not limited to interpolation methods. It can also be redrawn by selecting the area, and the area repair depends on the generation capability of the basic model. In proportion repair, the proportion problem of the generated target image object is mainly repaired, and the parts that are too blurred and have too much size difference are regenerated.

[0101] In the disclosed embodiment, the optimization process of the initial target image also includes redrawing based on the user's redrawing operation. If the generated target image is not satisfactory, the user can return to the previous version for repair, or directly return to the target image generation stage.

[0102] In the disclosed embodiment, the optimization process of the initial target image also includes image enhancement process. The image quality enhancement algorithm is used to improve the image quality, and the image can be smoothed, noise and details can be removed, the brightness and contrast of the image can be adjusted, and the edges and details of the initial target image can be enhanced, thereby improving the clarity of the initial target image.

[0103] In the disclosed embodiment, for a result that satisfies the user, high-definition restoration will be performed, that is, a low-resolution image will be upsampled to generate a high-resolution image.

[0104] In the disclosed embodiment, the target image displayed after the initial target image is optimized automatically repairs the defects in the image, and through prompt words and screening target content information, effective privacy protection can be performed to prevent the generated target image from being used for illegal purposes, and the copyright of AI creation can be protected. At the same time, high-definition images can be obtained after optimization, which improves user satisfaction.

[0105] The following is an explanation of image fusion in the embodiments of the present disclosure.

[0106] Figure 8 is a flow chart of an image fusion method according to an exemplary embodiment. Figure 8 As shown, the method includes the following.

[0107] In step S71 , a contour edge of the area image in the first image is determined.

[0108] In the embodiment of the present disclosure, the contour of the regional image is determined according to the contour of the first image generation area to generate the image, so as to ensure that the edge of the image contour in the image generation process fits into the first image area.

[0109] In step S72, the second image is fused at the region image of the first image according to the contour edge, and the contour edge is subjected to fusion optimization processing to obtain a target image.

[0110] In the disclosed embodiment, the second image is fused at the regional image of the first image according to the contour edge, and the contour edge is subjected to fusion optimization processing. Among them, the fusion optimization processing includes at least one of the following: image edge enhancement processing, image edge smoothing processing, and image clarity processing. In the image edge enhancement processing, an image quality enhancement algorithm is used to improve the image quality. In the image edge smoothing processing, for the edge of the image fusion, the image is smoothed by a mean filter. In the image clarity processing, noise and details are removed, and the fused edges and details are optimized by a sharpening filter to make the image clearer. The whole image is subjected to a histogram equalization method to adjust the brightness and contrast of the image to make the image clearer and more vivid. At the same time, super-resolution reconstruction is performed, and the information of the high-resolution image is used to enhance the details of the low-resolution image, thereby improving the clarity of the image. The above-mentioned methods are only exemplary descriptions and are not limited to the specific methods.

[0111] In the disclosed embodiment, in edge fusion, pixel-level mean fusion and gradient domain fusion can be used to fully retain the detail information of the image. After the fusion is completed, a high-definition style-edited image can be obtained. The style effect of the regional image is the same as the style of the first image, ensuring a high-quality editing effect.

[0112] The following describes an image drawing method in an embodiment of the present disclosure.

[0113] Fig. 9 is a flowchart of an image drawing method according to an exemplary embodiment. Fig. 9 As shown, the method includes the following.

[0114] In the disclosed embodiments, based on the user's graffiti or pictures, local editing of the image and local AI creation are achieved through the technology based on text guidance and AI creation drawing, and the style of AI creation is adapted to the style of the original picture through algorithm technology, so that the consistency of the image style after local creation and the style of the original image is achieved, and the AI ​​creation image can be integrated with the original image.

[0115] In the disclosed embodiment, a first image input by a user is obtained. If the user creates based on the original graffiti, the style of the graffiti is generated based on the self-developed AI creation technology. If the user edits an existing picture, the user is allowed to upload an image. For images uploaded in any way, local selection is made with a brush to select the area image that needs to be created or modified by AI. For images generated based on graffiti, the model training for style creation has been implemented, or the default style model is used, that is, there is no need to train the image style model separately. However, for data of unknown style uploaded by users, style adaptation needs to be performed quickly, that is, the style model of a single image needs to be trained. The style model trained and optimized in the cloud is sent to the mobile terminal for use.

[0116] Among them, for the user's current image, data preprocessing is first performed, and the focus of preprocessing is to cut the data to generate more training data. First, the user's data is segmented, and randomly segmented according to the ratio of 1 / 2, 1 / 3, 1 / 4, and 1 / 8 of the long side of the image, generating 2, 4, 4, and 8 pictures respectively, so there are a total of 18 pictures, and the original picture is repeated once, that is, a total of 20 training data are generated. Setting of training parameters: For each image of the style data set, the number of repetitions is 8, the batch size is set to 1, the rank of the LoRa model is 32, and the number of training rounds is 4. It is estimated that the training time is 1-2 minutes in the case of a single process without queuing, that is, when the user selects this function and uploads the picture, the data is automatically uploaded to the cloud and training is carried out. When the user is familiar with the user interface and the tutorial prompts, the training of the model is basically completed. In order to further speed up the image generation speed on the terminal side and not affect the timeliness of reasoning, the personalized style model and the basic model after training are quantized by Int8 and Float16, and the quantized model is sent to the mobile terminal. Then, based on the area image selected by the user, the style editing is performed to create the original image. This process can be repeated continuously and allows the user to adjust the extent of AI modification. It also supports text-guided AI editing and creation.

[0117] In the disclosed embodiment, the image is also subjected to optimization processing operations including privacy filtering, illegal image filtering, image resolution adaptation, etc., and then merged with the original image, i.e., the first image, and the final editing result, i.e., the target image, is output.

[0118] In the disclosed embodiment, the generated image needs to be fused with the original image in the image fusion part. In order to achieve better effects, three methods are mainly used to improve the final effect. First, the image quality enhancement algorithm is used to improve the image quality. For the edges of image fusion, the mean filter is used to smooth the image to remove noise and details. Then, the sharpening filter is used to optimize the fused edges and details to make the image clearer. Secondly, the histogram equalization method is used for the entire image to adjust the brightness and contrast of the image to make the image clearer and vivid. Finally, super-resolution reconstruction is performed to use the information of the high-resolution image to enhance the details of the low-resolution image, thereby improving the clarity of the image.

[0119] Furthermore, in the disclosed embodiments, the difficulty of image fusion is that in image edge fusion, it is relatively difficult to align the image edges, but in the image editing and generation process, the "image-to-image" generation method is used, that is, the contour of the generated image is generated according to the contour of the original image area, which ensures that the contour edge of the generated image can be better fitted into the original image area. In edge fusion, the detail information of the image can be fully retained through pixel-level mean fusion and gradient domain fusion. After the fusion is completed, the user can obtain a high-definition style-edited image.

[0120] In the disclosed embodiment, the image drawing method is applied to local image editing, photo editing applications, and painting and retouching applications, and supports multiple creation and editing, and can perform secondary or multiple creation of raw images based on generated images. Supports the generation of images in the form of style images and text guidance, so that the local style effect is the same as the style of the first image, and ensures that the second image generated by the regional image is fused with the first image. The real and natural image convenience effect also ensures high-quality editing effects. At the same time, an adaptive style algorithm is implemented, which can quickly realize style adaptive model training and send the model for editing in time. More efficient training and style adaptive algorithms, unlike the long-term training of other technologies, can complete the training of style models at a faster speed and send them for local use. Based on the training of image style models, through the combination of cloud and local reasoning, high-quality images can be generated in a short time, while saving computing resources and environmental resources, making full use of the edge computing capabilities of the mobile terminal, and improving resource utilization. And meet the needs of users, generate target images based on user preferences, and improve user creation efficiency and quality.

[0121] Based on the same concept, the embodiment of the present disclosure further provides an image drawing device 100 .

[0122] It is understandable that the image drawing device 100 provided in the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function in order to realize the above functions. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.

[0123] Fig.10 1 is a block diagram of an image drawing apparatus 100 according to an exemplary embodiment. Fig.10 The device includes an acquisition unit 101, a calling unit 102 and a determination unit 103.

[0124] The acquisition unit 101 is configured to acquire a first image.

[0125] The calling unit 102 is used for calling the image style model in response to detecting an operation of selecting a region image on the first image, and performing stylization processing on the region image to obtain a second image.

[0126] The determination unit 103 is configured to fuse the second image with the region image of the first image to obtain a target image.

[0127] In one implementation, the calling unit 102 calls the image style model in the following manner to perform stylized processing on the regional image to obtain a second image, including: in response to the first image being generated based on the first image style model, calling the first image style model to perform stylized processing on the regional image to obtain the second image; or in response to the first image not being generated based on the image style model, training an image style model based on the first image, and performing stylized processing on the regional image based on the trained image style model to obtain the second image.

[0128] In one implementation, the calling unit 102 obtains an image style model based on the first image training in the following manner, including: cutting the first image according to a set edge at a preset ratio, and copying the first image to obtain a style image sample set with a preset number; determining the style image sample set and the first image as a style image set; and training the image style model based on the style image set.

[0129] In one implementation, the calling unit 102 trains an image style model based on a style image set in the following manner, including: calling a first pre-trained model, and training a second pre-trained model based on the style image set; based on the first pre-trained model and the second pre-trained model, repeatedly training each style image in the style image set for a preset number of times to obtain an image style model.

[0130] In one implementation, the calling unit 102 is further used to: in response to detecting an operation of selecting a regional image on a first image, display local creation options, the local creation options including image local creation style options and image local creation text description options; stylize the regional image, including: determining the target image local creation style based on the image local creation style option selected by the user, and obtaining the target image local creation text description input by the user based on the image local creation text description option; and stylize the regional image based on the target image local creation style and the target image local creation text description.

[0131] In one implementation, the local creation option also includes an image creation modification amplitude option, and the image creation modification amplitude option is used to determine the similarity between the second image style and the first image style; the determination unit 103 is also used to: based on the image creation modification amplitude corresponding to the image creation modification amplitude option, generate a second image that matches the similarity style corresponding to the image creation modification amplitude.

[0132] In one implementation, before fusing the second image at the regional image of the first image, the determination unit 103 is also used to: optimize the initial target image and display the optimized target image; the optimization process includes at least one of the following: filtering out target content information; image area repair; redrawing based on the user's redrawing operation; image enhancement.

[0133] In one implementation, the determination unit 103 fuses the second image at the regional image of the first image to obtain a target image in the following manner, including: determining the contour edge of the regional image in the first image; fusing the second image at the regional image of the first image according to the contour edge, and performing fusion optimization processing on the contour edge to obtain the target image; the fusion optimization processing includes at least one of the following: image edge enhancement processing; image edge smoothing processing; image clarity processing.

[0134] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0135] Fig.111 is a block diagram of a device 200 for image drawing according to an exemplary embodiment. The device 200 may be provided as a terminal. For example, the device 200 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0136] Reference Fig.11 , the device 200 may include one or more of the following components: a processing component 202 , a memory 204 , a power component 206 , a multimedia component 208 , an audio component 210 , an input / output (I / O) interface 212 , a sensor component 214 , and a communication component 216 .

[0137] The processing component 202 generally controls the overall operation of the device 200, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 202 may include one or more modules to facilitate interaction between the processing component 202 and other components. For example, the processing component 202 may include a multimedia module to facilitate interaction between the multimedia component 208 and the processing component 202.

[0138] The memory 204 is configured to store various types of data to support operations on the device 200. Examples of such data include instructions for any application or method operating on the device 200, contact data, phone book data, messages, pictures, videos, etc. The memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0139] The power component 206 provides power to the various components of the device 200. The power component 206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 200.

[0140] The multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 208 includes a front camera and / or a rear camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0141] The audio component 210 is configured to output and / or input audio signals. For example, the audio component 210 includes a microphone (MIC), and when the device 200 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 204 or sent via the communication component 216. In some embodiments, the audio component 210 also includes a speaker for outputting audio signals.

[0142] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0143] The sensor assembly 214 includes one or more sensors for providing various aspects of the status assessment of the device 200. For example, the sensor assembly 214 can detect the open / closed state of the device 200, the relative positioning of components, such as the display and keypad of the device 200, the sensor assembly 214 can also detect the position change of the device 200 or a component of the device 200, the presence or absence of user contact with the device 200, the orientation or acceleration / deceleration of the device 200 and the temperature change of the device 200. The sensor assembly 214 can include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 214 can also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 214 can also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

[0144] The communication component 216 is configured to facilitate wired or wireless communication between the device 200 and other devices. The device 200 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 216 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 216 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0145] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0146] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, and the instructions can be executed by the processor 220 of the device 200 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

Claims

1. An image drawing method, characterized in that: include: acquiring a first image; In response to detecting an operation of selecting a regional image on the first image, calling an image style model to perform stylization processing on the regional image to obtain a second image; The second image is fused at the region image of the first image to obtain a target image.

2. The method according to claim 1, characterized in that The calling of the image style model to perform stylization processing on the region image to obtain a second image includes: In response to the first image being generated based on a first image style model, calling the first image style model to perform stylization processing on the region image to obtain a second image; or In response to the first image not being generated based on an image style model, the image style model is trained based on the first image, and based on the trained image style model, stylization processing is performed on the regional image to obtain a second image.

3. The method according to claim 2, characterized in that The step of obtaining the image style model based on the first image training includes: Cutting the first image along set edges at a preset ratio and copying the first image to obtain a style image sample set having a preset number; Determine the style image sample set and the first image as a style image set; Based on the style image set, an image style model is trained.

4. The method according to claim 3, characterized in that The step of training an image style model based on the style image set includes: Calling the first pre-trained model, and training the second pre-trained model based on the style image set; Based on the first pre-trained model and the second pre-trained model, each style image in the style image set is repeatedly trained for a preset number of times to obtain an image style model.

5. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: In response to detecting an operation of selecting a region image on the first image, displaying local creation options, the local creation options including image local creation style options and image local creation text description options; The stylizing the regional image includes: Determine the local creation style of the target image based on the image local creation style option selected by the user, and obtain the target image local creation text description input by the user based on the image local creation text description option; Based on the local creation style of the target image and the local creation text description of the target image, the regional image is stylized.

6. The method according to claim 5, characterized in that The local creation options also include an image creation modification range option, and the image creation modification range option is used to determine the similarity between the second image style and the first image style; The method further comprises: Based on the image creation modification amplitude corresponding to the image creation modification amplitude option, a second image matching the similarity style corresponding to the image creation modification amplitude is generated.

7. The method according to claim 1, characterized in that Before fusing the second image at the regional image of the first image, the method further includes: Optimizing the initial target image and displaying the optimized target image; The optimization process includes at least one of the following: Filter out target content information; Image region restoration; Perform redrawing based on the user's redrawing operation; Image enhancement processing.

8. The method according to claim 1, characterized in that The step of fusing the second image at the region image of the first image to obtain a target image comprises: Determine a contour edge of the regional image in the first image; According to the contour edge, the second image is fused at the area image of the first image, and the contour edge is subjected to fusion optimization processing to obtain a target image; The fusion optimization process includes at least one of the following: Image edge enhancement processing; Image edge smoothing; Image clarity processing.

9. An image drawing device, characterized in that: include: An acquisition unit, configured to acquire a first image; a calling unit, configured to call an image style model in response to detecting an operation of selecting a regional image on the first image, and perform stylization processing on the regional image to obtain a second image; A determination unit is used to fuse the second image at the area image of the first image to obtain a target image.

10. The device according to claim 9, characterized in that The calling unit calls the image style model in the following manner to perform stylization processing on the region image to obtain a second image, including: In response to the first image being generated based on a first image style model, calling the first image style model to perform stylization processing on the region image to obtain a second image; or In response to the first image not being generated based on an image style model, the image style model is trained based on the first image, and based on the trained image style model, stylization processing is performed on the regional image to obtain a second image.

11. The device according to claim 10, characterized in that The calling unit obtains the image style model based on the first image training in the following manner, including: Cutting the first image along set edges at a preset ratio and copying the first image to obtain a style image sample set having a preset number; Determine the style image sample set and the first image as a style image set; Based on the style image set, an image style model is trained.

12. The device according to claim 11, characterized in that The calling unit trains an image style model based on the style image set in the following manner, including: Calling the first pre-trained model, and training the second pre-trained model based on the style image set; Based on the first pre-trained model and the second pre-trained model, each style image in the style image set is repeatedly trained for a preset number of times to obtain an image style model.

13. The device according to any one of claims 9 to 11, characterized in that The calling unit is also used for: In response to detecting an operation of selecting a region image on the first image, displaying local creation options, the local creation options including image local creation style options and image local creation text description options; The stylizing the regional image includes: Determine the local creation style of the target image based on the image local creation style option selected by the user, and obtain the target image local creation text description input by the user based on the image local creation text description option; Based on the local creation style of the target image and the local creation text description of the target image, the regional image is stylized.

14. The device according to claim 13, characterized in that The local creation options also include an image creation modification range option, and the image creation modification range option is used to determine the similarity between the second image style and the first image style; The determining unit is further configured to: Based on the image creation modification amplitude corresponding to the image creation modification amplitude option, a second image matching the similarity style corresponding to the image creation modification amplitude is generated.

15. The device according to claim 9, characterized in that Before fusing the second image at the regional image of the first image, the determining unit is further used for: Optimizing the initial target image and displaying the optimized target image; The optimization process includes at least one of the following: Filter out target content information; Image region restoration; Perform redrawing based on the user's redrawing operation; Image enhancement processing.

16. The device according to claim 9, characterized in that The determining unit fuses the second image at the region image of the first image to obtain a target image in the following manner, including: Determine a contour edge of the regional image in the first image; According to the contour edge, the second image is fused at the area image of the first image, and the contour edge is subjected to fusion optimization processing to obtain a target image; The fusion optimization process includes at least one of the following: Image edge enhancement processing; Image edge smoothing; Image clarity processing.

17. An image drawing device, characterized in that: include: processor: a memory for storing processor-executable instructions; Wherein, the processor is configured to: execute the image drawing method according to any one of claims 1 to 8.

18. A storage medium, characterized in that: The storage medium stores instructions, and when the instructions in the storage medium are executed by a processor, the processor is enabled to execute the image drawing method according to any one of claims 1 to 8.

Images