Image processing method and device, computer equipment and storage medium

Abstract

The invention provides an image processing method and device, computer equipment and a storage medium. The method comprises the steps that multiple first reference images and style description information and content description information corresponding to the first reference images are acquired; wherein the style description information of the plurality of first reference images is the same; based on the style description information and the content description information, constructing first prompt information corresponding to the first reference image; the first prompt information is input into an initial stylized model, a first prediction image generated by the initial stylized model is determined, and the first prediction image is used for adjusting model parameters of the initial stylized model to determine a target stylized model; wherein the target stylized model is used for generating an image conforming to the style description information. Through the method, the display effect of the generated stylized image can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an image processing method, apparatus, computer equipment and storage medium. Background Art

[0002] With the development of artificial intelligence generated content (AIGC) technology, more and more people are beginning to generate images through AIGC. Generally, users can input image description information into the model, and the model can generate the corresponding target image based on the image description information. However, since the model is trained based on a large number of images, the stylized features of the target images generated by the trained model are not prominent and may not meet the user's image needs. Summary of the invention

[0003] The embodiments of the present disclosure at least provide an image processing method, apparatus, computer equipment, and storage medium.

[0004] In a first aspect, an embodiment of the present disclosure provides an image processing method, including:

[0005] Acquire a plurality of first reference images, and style description information and content description information corresponding to the first reference images; wherein the style description information of the plurality of first reference images is the same;

[0006] Based on the style description information and the content description information, construct first prompt information corresponding to the first reference image;

[0007] The first prompt information is input into an initial stylization model to determine a first predicted image generated by the initial stylization model, wherein the first predicted image is used to adjust model parameters of the initial stylization model to determine a target stylization model; wherein the target stylization model is used to generate an image that conforms to the style description information.

[0008] In a possible implementation manner, inputting the first prompt information into an initial stylization model to determine a first predicted image generated by the initial stylization model includes:

[0009] The first reference image and first prompt information corresponding to the first reference image are input into the initial stylization model to determine a first predicted image generated by the initial stylization model.

[0010] In a possible implementation manner, acquiring a plurality of first reference images includes:

[0011] Determining the entity category to be optimized corresponding to the initial stylized model;

[0012] A first reference image containing entities corresponding to the entity category to be optimized is obtained.

[0013] In a possible implementation manner, determining the entity category to be optimized corresponding to the initial stylized model includes:

[0014] Acquire content description information of a plurality of first images to be generated; wherein the content description information includes entity categories;

[0015] Inputting the content description information into the initial stylization model to determine a second predicted image;

[0016] The entity category included in the content description information corresponding to the second predicted image that does not meet the preset rule is used as the entity category to be optimized.

[0017] In a possible implementation manner, the method further includes determining first prompt information corresponding to the first reference image according to the following method:

[0018] The category information corresponding to the entity category to be optimized is used as the first prompt information corresponding to the first reference image.

[0019] In a possible implementation manner, the method further includes:

[0020] Obtaining content description information of a second image to be generated;

[0021] Constructing second prompt information based on the content description information and the style description information;

[0022] The second prompt information is input into the target stylized model to generate the second image to be generated.

[0023] In a possible implementation, when there are multiple target stylized models to be trained, obtaining content description information of the second image to be generated includes:

[0024] Obtaining content description information of a second image to be generated and target style description information;

[0025] The constructing the second prompt information based on the content description information and the style description information includes:

[0026] Constructing second prompt information based on the content description information and the target style description information;

[0027] The step of inputting the second prompt information into the target stylized model to generate the second image to be generated includes:

[0028] The second prompt information is input into a target stylization model corresponding to the target style description information to generate the second image to be generated; wherein different target stylization models match different style description information.

[0029] In a possible implementation manner, obtaining content description information of the second image to be generated includes:

[0030] Acquire content description information of a second image to be generated and a second reference image corresponding to the second image to be generated;

[0031] The step of inputting the second prompt information into the target stylized model to generate the second image to be generated includes:

[0032] The second prompt information and the second reference image are input into the target stylization model to generate the second image to be generated.

[0033] In a second aspect, the present disclosure also provides an image processing device, including:

[0034] An acquisition module, used to acquire a plurality of first reference images, and style description information and content description information corresponding to the first reference images; wherein the style description information of the plurality of first reference images is the same;

[0035] A construction module, configured to construct first prompt information corresponding to the first reference image based on the style description information and the content description information;

[0036] A determination module is used to input the first prompt information into an initial stylization model to determine a first predicted image generated by the initial stylization model, wherein the first predicted image is used to adjust model parameters of the initial stylization model to determine a target stylization model; wherein the target stylization model is used to generate an image that conforms to the style description information.

[0037] In a possible implementation manner, the determination module, when inputting the first prompt information into the initial stylization model to determine the first predicted image generated by the initial stylization model, is configured to:

[0038] The first reference image and first prompt information corresponding to the first reference image are input into the initial stylization model to determine a first predicted image generated by the initial stylization model.

[0039] In a possible implementation manner, when acquiring the plurality of first reference images, the acquisition module is used to:

[0040] Determining the entity category to be optimized corresponding to the initial stylized model;

[0041] A first reference image containing entities corresponding to the entity category to be optimized is obtained.

[0042] In a possible implementation, the acquisition module, when determining the entity category to be optimized corresponding to the initial stylized model, is used to:

[0043] Acquire content description information of a plurality of first images to be generated; wherein the content description information includes entity categories;

[0044] Inputting the content description information into the initial stylization model to determine a second predicted image;

[0045] The entity category included in the content description information corresponding to the second predicted image that does not meet the preset rule is used as the entity category to be optimized.

[0046] In a possible implementation manner, the acquisition module is further configured to determine the first prompt information corresponding to the first reference image according to the following method:

[0047] The category information corresponding to the entity category to be optimized is used as the first prompt information corresponding to the first reference image.

[0048] In a possible implementation manner, the device further includes a reasoning module, configured to:

[0049] Obtaining content description information of a second image to be generated;

[0050] Constructing second prompt information based on the content description information and the style description information;

[0051] The second prompt information is input into the target stylized model to generate the second image to be generated.

[0052] In a possible implementation, when there are multiple target stylized models to be trained, the inference module, when acquiring the content description information of the second image to be generated, is used to:

[0053] Obtaining content description information of a second image to be generated and target style description information;

[0054] The inference module, when constructing the second prompt information based on the content description information and the style description information, is used to:

[0055] Constructing second prompt information based on the content description information and the target style description information;

[0056] The inference module, when inputting the second prompt information into the target stylized model to generate the second image to be generated, is used to:

[0057] The second prompt information is input into a target stylization model corresponding to the target style description information to generate the second image to be generated; wherein different target stylization models match different style description information.

[0058] In a possible implementation manner, the inference module, when acquiring the content description information of the second image to be generated, is used to:

[0059] Acquire content description information of a second image to be generated and a second reference image corresponding to the second image to be generated;

[0060] The inference module, when inputting the second prompt information into the target stylized model to generate the second image to be generated, is used to:

[0061] The second prompt information and the second reference image are input into the target stylization model to generate the second image to be generated.

[0062] In a third aspect, an embodiment of the present disclosure further provides a computer device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the above-mentioned first aspect, or any possible implementation of the first aspect are performed.

[0063] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned first aspect or any possible implementation of the first aspect are executed.

[0064] In the image processing method, apparatus, computer device and storage medium provided by the embodiments of the present disclosure, the first prompt information corresponding to the first reference image includes style description information of the first reference image. In this way, when determining the target stylization model based on the first reference image and the first prompt information of the first reference image, the style that needs to be learned by the target stylization model can be explicitly constrained, so that the image generated by the target stylization model can be more focused on the style corresponding to the style description information, thereby improving the stylization processing effect of the image.

[0065] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0067] Figure 1 A flowchart of an image processing method provided by an embodiment of the present disclosure is shown;

[0068] Figure 2 A flowchart of another image processing method provided by an embodiment of the present disclosure is shown;

[0069] Figure 3 A flowchart of another image processing method provided by an embodiment of the present disclosure is shown;

[0070] Figure 4 A schematic diagram of the architecture of an image processing device provided by an embodiment of the present disclosure is shown;

[0071] Figure 5 A schematic diagram of the structure of a computer device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0073] In the related technology, when training an image generation model, it is necessary to base it on a massive amount of images, which include images of various styles, such as ancient style images, comic style images, etc.; this results in the trained image generation model being unable to focus on a certain style, so when the model is performing inference, the stylized features of the generated images are not prominent and may not meet the user's image needs.

[0074] Based on the above research, the present disclosure provides an image processing method, apparatus, computer device and storage medium, wherein the first prompt information corresponding to the first reference image includes style description information of the first reference image. In this way, when determining a target stylization model based on the first reference image and the first prompt information of the first reference image, the style that the target stylization model needs to learn can be explicitly constrained, so that the image generated by the target stylization model can be more focused on the style corresponding to the style description information, thereby improving the stylization processing effect of the image.

[0075] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0076] The term "and / or" herein only describes an association relationship, indicating that three relationships may exist. For example, A and / or B may represent the following three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C may represent including any one or more elements selected from the set consisting of A, B, and C.

[0077] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0078] For example, in response to receiving an active request from a user, a first prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the first prompt message.

[0079] As an optional but non-limiting implementation, in response to receiving an active request from the user, the first prompt information may be sent to the user in the form of a pop-up window, in which the first prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0080] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0081] To facilitate understanding of this embodiment, an image processing method disclosed in the embodiment of the present disclosure is first introduced in detail. The executor of the image processing method provided in the embodiment of the present disclosure is generally a computer device with certain computing capabilities, for example, the computer device may be a server.

[0082] See also Figure 1 FIG. 1 is a flowchart of an image processing method provided by an embodiment of the present disclosure, wherein the method includes steps 101 to 103, wherein:

[0083] Step 101: Acquire multiple first reference images, and style description information and content description information corresponding to the first reference images; wherein the style description information of the multiple first reference images is the same;

[0084] Step 102: construct first prompt information corresponding to the first reference image based on the style description information and the content description information;

[0085] Step 103: input the first prompt information into an initial stylization model to determine a first predicted image generated by the initial stylization model, wherein the first predicted image is used to adjust model parameters of the initial stylization model to determine a target stylization model; wherein the target stylization model is used to generate an image that conforms to the style description information.

[0086] The following is a detailed description of the above steps.

[0087] For step 101,

[0088] The first reference image can be understood as a sample image. Since the determined target stylization model is a model for generating images of a target style, the first reference image is an image used to train the target stylization model. Therefore, the image styles of the first reference images can be the same, both being the target style. Here, the target style can be the style described by the style description information.

[0089] The style description information may be, for example, a style name, etc. Optionally, the style description information may also carry a keyword for identifying a style, such as "artstyle", and the style description information may be, for example, "A artstyle", "B artstyle", etc.

[0090] Here, since the target stylization model needs to learn the image style of the first reference image, and the role of the style description information is to explicitly limit the image style of the first reference image, the style description information of the multiple first reference images needs to be consistent.

[0091] It should be noted that after the style description information is input into the stylized model (including the initial stylized model and the target stylized model), its main function is to limit the image style and does not contain other special semantic information. Therefore, the style description information can be set to any text information, for example, it can be set to A, B, C, D, X, etc.

[0092] The content description information is information used to describe the content contained in the first reference image. In a possible implementation, the content description information may include the names of entities contained in the first reference image and the association relationships between the entities. Exemplarily, the content description information may be "two people walking on the beach."

[0093] For step 102 and step 103,

[0094] In a possible implementation, when constructing the first prompt information corresponding to the first reference image based on the style description information and the content description information, the style description information and the content description information may be concatenated to form the first prompt information.

[0095] Exemplarily, if the style description information is A artstyle and the content description information is “On a cloudy day, two people are walking on the beach”, the first prompt information may be “A artstyle, two people are walking on the beach”.

[0096] Optionally, when concatenating the style description information and the content description information, they may be concatenated in a uniform preset order. For example, the content description information may be placed after the style description information, and the content description information and the style description information may be separated by a comma; or, the style description information may be placed after the content description information, and the style description information and the content description information may be separated by a semicolon.

[0097] The initial stylized model may be a model pre-trained based on massive data, that is, before adjusting the model parameters of the initial stylized model, the prompt information is input into the initial stylized model, and the initial stylized model can also generate a corresponding image, but the stylization of the generated image is not prominent.

[0098] In the method provided in the present disclosure, fine-tuning training is performed on the basis of the initial stylized model. Since the initial stylized model has been trained with massive data, when the model parameters of the initial stylized model are adjusted based on the first reference image and the first prompt information, the number of the first reference images can be small, that is, the application scenario of the method provided in the present disclosure is small sample fine-tuning training.

[0099] The method provided by the present disclosure can be applied to a variety of AIGC scenarios, for example, it can be applied to a Wensheng picture scenario. In this scenario, the first prompt information can be directly input into the initial stylization model to determine the first predicted image generated by the initial stylization model.

[0100] In another scenario, the method provided by the present disclosure can be applied to a picture-to-picture scenario. In this scenario, the first reference image and the first prompt information corresponding to the first reference image can be input into the initial stylization model to determine the first predicted image generated by the initial stylization model.

[0101] In a possible implementation, when adjusting the model parameters of the initial stylization model based on the first reference image and the first predicted image, the loss value of this training can be determined based on the first reference image and the first predicted image, and the model parameters of the initialized style model can be adjusted based on the loss value.

[0102] Exemplarily, when determining the loss value based on the first reference image and the first predicted image, image features of the first reference image and the first predicted image can be extracted respectively, and then the feature distance between the two image features is calculated, and the feature distance is used as the loss value.

[0103] In another possible implementation, when determining the loss value, it may also be based on content description information of the first predicted image and the first reference image.

[0104] Exemplarily, after determining the first predicted image, content extraction may be performed on the first predicted image to determine content description information of the first predicted image, and then based on the content description information of the first reference image and the content description information of the first predicted image, the loss value of this training may be determined.

[0105] Theoretically, after the model generates the first predicted image based on the content description information (the first prompt information includes the content description information, and the model generates the predicted image mainly based on the content description information), theoretically, after content extraction is performed on the first predicted image, the extracted content description information should be the same as the content description information of the first reference image. Therefore, the difference between the content description information of the first reference content and the content description information of the first predicted image can reflect the performance of the initial stylization model.

[0106] When determining the loss value of this training based on the content description information of the first reference image and the content description information of the first predicted image, exemplarily, the first semantic information of the content description information of the first reference image can be extracted, and the second semantic information of the content description information of the first predicted image can be extracted, and then the information difference between the first semantic information and the second semantic information can be calculated, and the calculated information difference can be used as the loss value of this training.

[0107] In the above method, the first prompt information corresponding to the first reference image includes style description information of the first reference image. In this way, when determining the target stylization model based on the first reference image and the first prompt information of the first reference image, the style that the target stylization model needs to learn can be explicitly constrained, so that the image generated by the target stylization model can be more focused on the style corresponding to the style description information, thereby improving the stylization processing effect of the image.

[0108] In addition, since the style description information and content description information are distinguished in the process of adjusting the initial stylization model, compared with using the style description information alone to adjust the initial stylization model, the decoupling of style features and content features can be achieved, avoiding the initial stylization model from associating the image content during training with the style features, which affects the image generation effect.

[0109] Optionally, the sample data of the initial stylized model during pre-training may not be rich enough, so the quality of the generated images including some entities may be poor. For example, if the number of sample images containing the entity "candle" during pre-training of the initial stylized model is small, then when the image is subsequently generated based on the content description information, if the content description information contains "candle", the display effect of the entity "candle" in the generated predicted image may be poor.

[0110] Therefore, based on this situation, the method provided by the present disclosure can also optimize the generation effect of the initial stylization model for entity categories with a small number of sample images in the pre-training stage.

[0111] See also Figure 2 FIG. 1 is a flowchart of another image processing method provided by the present disclosure, comprising the following steps:

[0112] Step 201: determining the entity category to be optimized corresponding to the initial stylized model, and obtaining a first reference image containing the entity corresponding to the entity category to be optimized;

[0113] Here, the entity categories to be optimized may include multiple categories, and the first reference image may be an image containing entities corresponding to at least one category among the multiple entity categories to be optimized.

[0114] Step 202: Obtain content description information corresponding to the first reference image;

[0115] Here, the content description information corresponding to the first reference image may refer to category information of the entity category to be optimized contained in the first reference image, for example, may be an entity name.

[0116] Step 203: Determine first prompt information corresponding to the first reference image based on the content description information;

[0117] The purpose of optimizing the initial stylized model here is to improve the generation effect of the initial stylized model for entities corresponding to the entity category to be optimized, which is independent of the image style of the first reference image. Therefore, only the content description information corresponding to the first reference image can be obtained in step 202.

[0118] In a possible implementation manner, when determining the first prompt information corresponding to the first reference image based on the content description information, the content description information may be directly used as the first prompt information corresponding to the first reference image.

[0119] Exemplarily, if the entity category to be optimized is a violin, the first reference image is an image containing a violin, and the content description information corresponding to the first reference image is “violin”.

[0120] Step 204: input the first prompt information into an initial stylization model to determine a first predicted image generated by the initial stylization model;

[0121] Step 205: Adjust model parameters of the initial stylized model based on the first reference image and the first predicted image.

[0122] Here, the specific execution process of adjusting the model parameters of the initial stylized model based on the first reference image and the first predicted image can be: Figure 1 The steps are the same as those described in , and will not be repeated here.

[0123] In a possible implementation manner, the entity category to be optimized may be determined in the following manner:

[0124] Step a1, obtaining content description information of a plurality of first images to be generated; wherein the content description information includes entity categories;

[0125] Here, the content description information may include at least one entity category.

[0126] Step a2: inputting the content description information into the initial stylization model to determine a second predicted image;

[0127] Step a3: taking the entity category contained in the content description information corresponding to the second predicted image that does not meet the preset rule as the entity category to be optimized.

[0128] The second predicted image not meeting the preset rule may refer to a second predicted image not meeting the user's needs. The second predicted image not meeting the user's needs here does not refer to a specific second predicted image not meeting the user's needs, but refers to a certain type of second predicted image, that is, a second predicted image containing the same entity category. For example, not meeting the preset rule may refer to receiving negative feedback for a second predicted image containing a certain entity for more than a preset number of times.

[0129] The process of determining the category of the examples to be optimized may be determined during the process of model reasoning after the initial stylization model has completed pre-training. The content description information of the plurality of first images to be generated may be sent by a plurality of user terminals.

[0130] In another possible implementation, the first reference image obtained in step 201 and the first reference image obtained in step 101 may be mixed to adjust the initial stylized image.

[0131] Specifically, refer to Figure 3 As shown, another image processing method provided by the embodiment of the present disclosure includes the following steps:

[0132] Step 301: Acquire multiple reference images, and determine prompt information corresponding to the reference images;

[0133] Here, the reference image includes a plurality of first reference images, and the style description information of the plurality of first reference images is the same; the prompt information corresponding to the first reference image is determined based on the style description information and content description information corresponding to the first reference image.

[0134] The reference image also includes a plurality of third reference images, wherein the third reference image is an image containing entities corresponding to the entity category to be optimized; the prompt information corresponding to the third reference image is determined based on the content description information of the third reference image. The third reference image here is equivalent to the first reference image in the above step 201.

[0135] Here, in order to avoid the initial stylization model being unable to fully learn the style features of the first reference image due to too many third reference images, the first reference image and the third reference image may meet a preset ratio, for example, the ratio of the first reference image to the third reference image may be 3: 1. The specific ratio may be adjusted according to actual conditions.

[0136] Step 302: input the prompt information into an initial stylization model to determine a predicted image generated by the initial stylization model;

[0137] Step 303: Based on the reference image and the predicted image, adjust the model parameters of the initial stylization model to determine a target stylization model; wherein the target stylization model is used to generate an image that conforms to the style description information.

[0138] Optionally, in another possible implementation, the model parameters of the initial stylized model may be adjusted based on the first reference image, and then based on the third reference image, the model parameters of the initial stylized model may be adjusted again on the basis of the adjustment.

[0139] Alternatively, the model parameters of the initial stylized model may be adjusted based on the third reference image first, and then based on the adjustment, the model parameters of the initial stylized model may be adjusted again based on the first reference image.

[0140] After the adjustment of the target stylized model is completed, the following steps can be used as an example for inference:

[0141] Step b1, obtaining content description information of the second image to be generated;

[0142] Exemplarily, the acquiring content description information of the second image to be generated may refer to acquiring content description information sent by the user terminal.

[0143] Step b2: constructing second prompt information based on the content description information and the style description information;

[0144] In a possible implementation manner, the style description information may be pre-set, that is, fixed style description information. After acquiring the content description information, the pre-set style description information may be directly concatenated with the content description information to construct the second prompt information.

[0145] In another possible implementation, different target stylized models can be pre-trained based on first reference images of different styles. In this implementation, when acquiring content description information, target style description information can be acquired simultaneously, and then second prompt information can be constructed based on the acquired content description information and the acquired target style description information. Accordingly, a target stylized model that matches the target style description information can also be selected based on the acquired target style description information; different target stylized models match different style description information, and the style description information that matches the target stylized model is the style description information of the reference image in the training process of the target stylized model.

[0146] Step b3: input the second prompt information into the target stylized model to generate the second image to be generated.

[0147] Correspondingly, in the scenario of graph-generated graphs, the process of model reasoning is:

[0148] Step c1, obtaining content description information of a second image to be generated and a second reference image corresponding to the second image to be generated;

[0149] Step c2: constructing second prompt information based on the content description information and the style description information;

[0150] Step c3: input the second prompt information and the second reference image into the target stylized model to generate the second image to be generated.

[0151] The specific implementation process of the above process is Figure 1 and Figure 2 The specific execution process in the method shown is similar and will not be repeated here.

[0152] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0153] Based on the same inventive concept, an image processing device corresponding to the image processing method is also provided in the embodiment of the present disclosure. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned image processing method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0154] Reference Figure 4 , which is a schematic diagram of the architecture of an image processing device provided by an embodiment of the present disclosure, the device includes: an acquisition module 401, a construction module 402, a determination module 403, and an inference module 404; wherein,

[0155] The acquisition module 401 is used to acquire a plurality of first reference images, and style description information and content description information corresponding to the first reference images; wherein the style description information of the plurality of first reference images is the same;

[0156] A construction module 402, configured to construct first prompt information corresponding to the first reference image based on the style description information and the content description information;

[0157] The determination module 403 is used to input the first prompt information into the initial stylization model, determine the first predicted image generated by the initial stylization model, and the first predicted image is used to adjust the model parameters of the initial stylization model to determine the target stylization model; wherein the target stylization model is used to generate an image that conforms to the style description information.

[0158] In a possible implementation manner, the determination module 403, when inputting the first prompt information into the initial stylization model to determine the first predicted image generated by the initial stylization model, is configured to:

[0159] The first reference image and first prompt information corresponding to the first reference image are input into the initial stylization model to determine a first predicted image generated by the initial stylization model.

[0160] In a possible implementation manner, when acquiring a plurality of first reference images, the acquisition module 401 is configured to:

[0161] Determining the entity category to be optimized corresponding to the initial stylized model;

[0162] A first reference image containing entities corresponding to the entity category to be optimized is obtained.

[0163] In a possible implementation, the acquisition module 401, when determining the entity category to be optimized corresponding to the initial stylized model, is used to:

[0164] Acquire content description information of a plurality of first images to be generated; wherein the content description information includes entity categories;

[0165] Inputting the content description information into the initial stylization model to determine a second predicted image;

[0166] The entity category included in the content description information corresponding to the second predicted image that does not meet the preset rule is used as the entity category to be optimized.

[0167] In a possible implementation manner, the acquisition module 401 is further configured to determine the first prompt information corresponding to the first reference image according to the following method:

[0168] The category information corresponding to the entity category to be optimized is used as the first prompt information corresponding to the first reference image.

[0169] In a possible implementation manner, the device further includes a reasoning module 404, configured to:

[0170] Obtaining content description information of a second image to be generated;

[0171] Constructing second prompt information based on the content description information and the style description information;

[0172] The second prompt information is input into the target stylized model to generate the second image to be generated.

[0173] In a possible implementation, when there are multiple target stylized models to be trained, the reasoning module 404, when acquiring the content description information of the second image to be generated, is used to:

[0174] Obtaining content description information of a second image to be generated and target style description information;

[0175] The reasoning module 404, when constructing the second prompt information based on the content description information and the style description information, is used to:

[0176] Constructing second prompt information based on the content description information and the target style description information;

[0177] The inference module 404, when inputting the second prompt information into the target stylized model to generate the second image to be generated, is used to:

[0178] The second prompt information is input into a target stylization model corresponding to the target style description information to generate the second image to be generated; wherein different target stylization models match different style description information.

[0179] In a possible implementation, the reasoning module 404, when acquiring the content description information of the second image to be generated, is used to:

[0180] Acquire content description information of a second image to be generated and a second reference image corresponding to the second image to be generated;

[0181] The inference module 404, when inputting the second prompt information into the target stylized model to generate the second image to be generated, is used to:

[0182] The second prompt information and the second reference image are input into the target stylization model to generate the second image to be generated.

[0183] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0184] Based on the same technical concept, the embodiment of the present disclosure also provides a computer device. Figure 5, which is a schematic diagram of the structure of a computer device 500 provided in an embodiment of the present disclosure. The computer device in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc., or various forms of servers, such as independent servers or server clusters. Figure 5 The computer device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0185] like Figure 5 As shown, the computer device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only storage device (ROM) 502 or a program loaded from a storage device 505 to a random access storage device (RAM) 503. Various programs and data required for the operation of the computer device 500 are also stored in the RAM 503. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0186] Typically, the following devices may be connected to the I / O interface 505: input devices 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 508 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 509. The communication devices 509 may allow the computer device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 The computer device 500 is shown with various devices, but it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have instead.

[0187] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing a method for recommending words. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0188] The present disclosure also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the steps of the image processing method described in the above method embodiment are executed. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0189] The present disclosure also provides a computer program product that carries a program code. The program code includes instructions that can be used to execute the steps of the image processing method described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.

[0190] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is implemented as a computer storage medium. In another optional embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0191] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0192] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0193] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0194] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0195] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: include: Acquire a plurality of first reference images, and style description information and content description information corresponding to the first reference images; wherein the style description information of the plurality of first reference images is the same; Based on the style description information and the content description information, construct first prompt information corresponding to the first reference image; The first prompt information is input into an initial stylization model to determine a first predicted image generated by the initial stylization model, wherein the first predicted image is used to adjust model parameters of the initial stylization model to determine a target stylization model; wherein the target stylization model is used to generate an image that conforms to the style description information.

2. The method according to claim 1, characterized in that The step of inputting the first prompt information into an initial stylization model to determine a first predicted image generated by the initial stylization model includes: The first reference image and first prompt information corresponding to the first reference image are input into the initial stylization model to determine a first predicted image generated by the initial stylization model.

3. The method according to claim 1, characterized in that The acquiring of a plurality of first reference images comprises: Determining the entity category to be optimized corresponding to the initial stylized model; A first reference image containing entities corresponding to the entity category to be optimized is obtained.

4. The method according to claim 3, characterized in that: The step of determining the entity category to be optimized corresponding to the initial stylized model includes: Acquire content description information of a plurality of first images to be generated; wherein the content description information includes entity categories; Inputting the content description information into the initial stylization model to determine a second predicted image; The entity category included in the content description information corresponding to the second predicted image that does not meet the preset rule is used as the entity category to be optimized.

5. The method according to claim 3, characterized in that: The method further includes determining first prompt information corresponding to the first reference image according to the following method: The category information corresponding to the entity category to be optimized is used as the first prompt information corresponding to the first reference image.

6. The method according to claim 1, characterized in that The method further comprises: Obtaining content description information of a second image to be generated; Constructing second prompt information based on the content description information and the style description information; The second prompt information is input into the target stylized model to generate the second image to be generated.

7. The method according to claim 6, characterized in that In the case where there are multiple target stylized models to be trained, the step of obtaining content description information of the second image to be generated includes: Obtaining content description information of a second image to be generated and target style description information; The constructing the second prompt information based on the content description information and the style description information includes: Constructing second prompt information based on the content description information and the target style description information; The step of inputting the second prompt information into the target stylized model to generate the second image to be generated includes: The second prompt information is input into a target stylization model corresponding to the target style description information to generate the second image to be generated; wherein different target stylization models match different style description information.

8. The method according to claim 6, characterized in that The obtaining content description information of the second image to be generated includes: Acquire content description information of a second image to be generated and a second reference image corresponding to the second image to be generated; The step of inputting the second prompt information into the target stylized model to generate the second image to be generated includes: The second prompt information and the second reference image are input into the target stylization model to generate the second image to be generated.

9. An image processing device, characterized in that: include: An acquisition module, used to acquire a plurality of first reference images, and style description information and content description information corresponding to the first reference images; wherein the style description information of the plurality of first reference images is the same; A construction module, configured to construct first prompt information corresponding to the first reference image based on the style description information and the content description information; A determination module is used to input the first prompt information into an initial stylization model to determine a first predicted image generated by the initial stylization model, wherein the first predicted image is used to adjust model parameters of the initial stylization model to determine a target stylization model; wherein the target stylization model is used to generate an image that conforms to the style description information.

10. A computer device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the image processing method according to any one of claims 1 to 8 are performed.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the image processing method according to any one of claims 1 to 8 are executed.

Images