Image processing method and apparatus, model training method and apparatus, medium, and program product

Abstract

An image processing method and apparatus, a model training method and apparatus, a medium, and a program product. The image processing method comprises: acquiring an original image and a tag which is used for indicating a deformability, wherein the original image comprises an object that is to deform according to the deformability (101); acquiring image description text of the original image (102); generating target description text on the basis of the image description text and the tag (103); and generating a target image on the basis of the original image and the target description text, wherein the target image comprises the object that has deformed according to the deformability (104). The deformation effect after processing according to the deformability can be obtained by means of the original image and the tag which is used for indicating the deformability, so that reliable technical support can be provided for human body beautification, facilitating the further improvement of the human body beautification effect, and the convenience of image processing can also be effectively improved.

Description

Image processing methods, model training methods, devices, media and program products

[0001] This application claims priority to Chinese Patent Application No. 202410495412.0, filed on April 23, 2024, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0002] This disclosure relates to an image processing method, a model training method, an apparatus, a medium, and a program product. Background Technology

[0003] Human body enhancement has always been a key focus for photo editing software. However, compared to facial enhancement, human body postures are more complex and clothing and accessories have more diverse textures, making it more difficult to use neural networks for human body enhancement. As a result, there is currently no mature body enhancement solution, and existing body enhancement algorithms in related technologies mostly suffer from distortion during deformation processing. Summary of the Invention

[0004] This summary section is provided to briefly introduce the concepts, which will be described in detail in the subsequent detailed description section. This summary section is not intended to identify key or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0005] In a first aspect, this disclosure provides an image processing method, the method comprising:

[0006] Acquire an original image, and acquire the original image, and a label for representing the deformability, the original image including an object to be deformed according to the deformability;

[0007] Obtain the image description text of the original image;

[0008] Based on the image description text and the tag, generate target description text;

[0009] A target image is generated based on the original image and the target description text, the target image comprising the object after deformation according to the deformation capability.

[0010] Secondly, this disclosure provides a model training method, the method comprising:

[0011] Multiple sets of deformed image samples are obtained. Each set of deformed image samples includes an original sample image, at least one label, and a deformed effect image of the original sample image corresponding to the at least one label.

[0012] Using the multiple sets of deformed image samples as training data, the first preset initial ControlNet (an image deformation algorithm based on control points) network is trained to obtain the target deformation processing model.

[0013] Thirdly, this disclosure provides a model training method, the method comprising:

[0014] Acquire multiple sets of first image sample data, each set of first image sample data including the original sample image and the descriptive text annotation data corresponding to the original sample image;

[0015] Using the multiple sets of first image sample data as training data, a preset initial image description algorithm is trained to obtain an image description generation module;

[0016] Acquire multiple sets of second image sample data. Each set of second image sample data includes an original sample image, labeled text data, and a deformation effect image corresponding to the original sample image. The labeled text data includes descriptive text labeled data corresponding to the original sample image and at least one label.

[0017] Using the multiple sets of second image sample data as training data, the second preset initial ControlNet network is trained to obtain the deformation combination module;

[0018] The image description generation module and the deformation combination module are combined into a target deformation processing model.

[0019] Fourthly, this disclosure provides an image processing apparatus, the apparatus comprising:

[0020] The first acquisition module is configured to acquire an original image and a label for representing deformation capability, the original image including an object to be deformed according to the deformation capability;

[0021] The determination module is configured to acquire image description text of the original image; generate target description text based on the image description text and the tag; and generate a target image based on the original image and the target description text, wherein the target image includes the object after being deformed according to the deformation capability.

[0022] Fifthly, this disclosure provides a model training apparatus, the apparatus comprising:

[0023] The second acquisition module is configured to acquire multiple sets of deformation image samples, each set of deformation image samples including an original sample image, at least one label, and a deformation effect image of the original sample image corresponding to the at least one label.

[0024] The first training module is configured to train a first preset initial ControlNet network using the multiple sets of deformed image samples as training data to obtain the target deformation processing model.

[0025] Sixthly, this disclosure provides a model training apparatus, wherein the apparatus includes:

[0026] The third acquisition module is configured to acquire multiple sets of first image sample data, each set of first image sample data including an original sample image and descriptive text annotation data corresponding to the original sample image;

[0027] The second training module is configured to use the multiple sets of first image sample data as training data to train a preset initial image description algorithm to obtain the image description generation module.

[0028] The fourth acquisition module is configured to acquire multiple sets of second image sample data. Each set of second image sample data includes an original sample image, labeled text data, and a deformation effect image corresponding to the original sample image. The labeled text data includes descriptive text labeled data corresponding to the original sample image and at least one label.

[0029] The third training module is configured to train the second preset initial ControlNet network using the multiple sets of second image sample data as training data to obtain the deformation combination module;

[0030] The coupling module is configured to combine the image description generation module and the deformation combination module into the target deformation processing model.

[0031] In a seventh aspect, this disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the methods described in the first, second, or third aspects above.

[0032] Eighthly, this disclosure provides an electronic device, comprising:

[0033] A storage device on which computer programs are stored;

[0034] A processing device for executing the computer program in the storage device to implement the steps of the method described in the first, second, or third aspect.

[0035] Ninthly, this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in the first, second, or third aspect. Attached Figure Description

[0036] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale. In the drawings:

[0037] Figure 1 is a flowchart illustrating an image processing method according to an exemplary embodiment of the present disclosure;

[0038] Figure 2 is a flowchart illustrating an image processing method according to the embodiment shown in Figure 1;

[0039] Figure 3 is a flowchart illustrating a model training method in an exemplary embodiment of this disclosure;

[0040] Figure 4 is a flowchart illustrating a model training method in another exemplary embodiment of this disclosure;

[0041] Figure 5 is a block diagram illustrating an image processing apparatus according to an exemplary embodiment of the present disclosure;

[0042] Figure 6 is a block diagram illustrating a model training apparatus according to an exemplary embodiment of the present disclosure;

[0043] Figure 7 is a block diagram illustrating a model training apparatus according to another exemplary embodiment of the present disclosure; and

[0044] Figure 8 is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0045] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0046] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0047] The term "comprising" and its variations as used in this disclosure are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions for other terms will be given in the description below.

[0048] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0049] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0050] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0051] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0052] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0053] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0054] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0055] At the same time, it is understood that the data involved in this disclosed technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0056] Figure 1 is a flowchart illustrating an exemplary embodiment of the present disclosure of an image processing method; as shown in Figure 1, the image processing method may include:

[0057] Step 101: Obtain the original image and the label used to represent the deformation capability.

[0058] The image may contain one or more tags, each of which may be an identifier of a deformation capability. The original image may include a specified deformation object, which can be understood as an object that will be deformed according to the deformation capability.

[0059] It should be noted that the specified deformation object can be a human image, a human face, an animal body, or other specified objects. When the specified deformation object is a human body, the deformation ability can include lengthening the neck, slimming the waist, lengthening or thinning the legs, lengthening or thinning the arms, and adjusting the proportions of the upper and lower body. Each deformation ability can be identified by any specified symbol, which can be considered a token. Different tokens can authorize different deformation ability operation permissions. For example, the identifier for thinning the arms could be SBBX, &slim-arm&, 001, or simply "thin arms," ​​while the identifier for lengthening the neck could be "swan neck," TEJ, or 002, or other specified letters, numbers, words, and symbols. When the specified deformation object is a human face, the deformation ability can include whitening, double eyelid surgery, eyebrow shaping, and lip shaping, and the label can be the corresponding token.

[0060] Step 102: Obtain the image description text of the original image.

[0061] The image description text is used to describe the content of the original image. For example, the image description text of an original image could be "A girl in a red dress is dancing".

[0062] Step 103: Generate target description text based on the image description text and the tag.

[0063] The image description text can be concatenated with one or more tags used to indicate deformation capability to obtain the target description text.

[0064] It should be noted that during the splicing process, you can splice according to a pre-set splicing format. For example, you can add a specified separator symbol between different tags, such as a space, comma, semicolon, vertical bar, etc.

[0065] For example, if the image description text is "A girl in a red dress is dancing", and the user inputs the labels "swan neck" and "&slim-arm&", then the target description text could be "A girl in a red dress is dancing swan neck&slim-arm&".

[0066] Step 104: Generate a target image based on the original image and the target description text, wherein the target image includes the object after deformation according to the deformation capability.

[0067] The target image includes an object that has been deformed according to one or more labels representing deformability.

[0068] In one embodiment, the original image and the target description text can be input into a preset deformation combination network to obtain the target image output by the deformation combination module. This deformation combination network can present deformation effects by freely combining the deformation capabilities corresponding to one or more tags representing deformation capabilities.

[0069] In another embodiment, steps 102 to 104 above can be implemented by inputting the original image and one or more labels for representing deformation capability into a preset target deformation processing model, thereby obtaining the target image output by the target deformation processing model.

[0070] The target deformation processing model can be a neural network structure model based on the ControlNet algorithm.

[0071] For example, if the user inputs labels such as "elongated neck", "lengthened legs", and "thinned arms", the human body in the target image can be an object that has been processed by elongating the neck, lengthening the legs, and thinning the arms.

[0072] The training of the target deformation processing model can be implemented in two ways. In the first implementation, the training process of the target deformation processing model may include: acquiring multiple sets of deformation image samples, each set of deformation image samples including an original sample image, at least one label, and a deformation effect image of the original sample image corresponding to the at least one label; using the multiple sets of deformation image samples as training data, training a first preset initial ControlNet network to obtain the target deformation processing model.

[0073] It should be noted that, due to the zero-shot learning capability of the ControlNet network, it can fully utilize existing training data to identify and process unseen categories or tasks. Therefore, during the training of this target deformation processing model, the model does not need to be trained for all label combinations and can directly use existing deformation capability combinations to predict other combinations. Typically, when the multiple sets of deformed image samples include n labels representing deformation capabilities, the trained target deformation processing model can present 2^n combinations. Thus, when the user inputs one or more labels representing deformation capabilities, the target deformation processing model can present the deformation effect after freely combining the deformation capabilities corresponding to the one or more labels, thereby effectively improving the model's efficiency in combining various deformation capabilities and facilitating the rapid acquisition of image processing results satisfactory to the user.

[0074] In a second embodiment, the training process of the target deformation processing model may include: acquiring multiple sets of first image sample data, each set of first image sample data including an original sample image and descriptive text annotation data corresponding to the original sample image; training a preset initial image description algorithm using the multiple sets of first image sample data as training data to obtain the image description generation module; acquiring multiple sets of second image sample data, each set of second image sample data including an original sample image, annotation text data, and a deformation effect image corresponding to the original sample image, wherein the annotation text data includes descriptive text annotation data corresponding to the original sample image and at least one label; training a second preset initial ControlNet network using the multiple sets of second image sample data as training data to obtain the deformation combination module; and combining the image description generation module and the deformation combination module to form the target deformation processing model.

[0075] The descriptive text annotation data is a descriptive text for the image content in a specified language. For example, the descriptive text annotation data for a certain image could be "a girl is dancing ballet". The first preset initial ControlNet network and the second preset initial ControlNet network can be the same or different. For example, the first preset initial ControlNet network and the second preset initial ControlNet network can be ControlNet algorithms with the same hyperparameters, or they can be two ControlNet algorithms with some or all different hyperparameters.

[0076] Additionally, it should be noted that if the multiple sets of second image sample data include n labels of deformation capabilities, the trained target deformation processing model can have the ability to present 2^n combinations.

[0077] The above technical solutions can obtain a deformation effect formed by the combination of deformation capabilities corresponding to the one or more labels used to represent deformation capabilities by inputting an original image and one or more labels. This can provide reliable technical support for human body beautification, which is conducive to further improving the human body beautification effect and can also effectively improve the convenience of image processing.

[0078] Figure 2 is a flowchart illustrating an image processing method according to the embodiment shown in Figure 1. Optionally, the implementation of steps 102 to 104 in Figure 1 may involve inputting the original image and one or more labels representing deformation capability into a target deformation processing model. The target deformation processing model may include an image description generation module and a deformation combination module coupled to the image description generation module. Based on the target deformation processing model, image description text of the original image is first obtained, and then target description text is generated based on the image description text and the labels. Finally, a target image is generated based on the original image and the target description text. Specific implementation methods are shown in Figure 2 and may include:

[0079] Step 1021: Input the original image into the image description generation module to obtain the image description text corresponding to the original image output by the image description generation module.

[0080] This step generates image description text corresponding to the original image through the image description generation module. This image description generation module can be a network based on the NIC (Neural Image Caption) algorithm to convert image content into natural language descriptions, or an image description network based on Visual Attention, or an image description network based on SCA-CNN (Spatial and Channel-wise Attention in Convolutional Networks), or an image description generation algorithm based on Transformer, or other neural network algorithms.

[0081] Step 1022: Concatenate the image description text and one or more tags used to represent deformation capability to obtain the target description text.

[0082] During the splicing process, the splicing can be performed according to a pre-set splicing format. For example, a specified separator symbol can be added between different tags, such as a space, comma, semicolon, vertical bar, etc.

[0083] For example, if the image description text is "A girl in a red dress is dancing", and the user inputs the labels "swan neck" and "&slim-arm&", then the target description text could be "A girl in a red dress is dancing swan neck&slim-arm&".

[0084] Step 1023: Input the target description text and the original image into the deformation combination module to obtain the target image output by the deformation combination module.

[0085] The deformation combination module can output the deformation effect after combining the deformation capabilities of multiple labels when the user inputs multiple labels.

[0086] It should be noted that when the user inputs one or more labels representing deformation capabilities, the target deformation processing model can present the deformation effect after freely combining the deformation capabilities corresponding to the one or more labels representing deformation capabilities. This can effectively improve the model's efficiency in combining various deformation capabilities, thereby facilitating the rapid acquisition of image processing results that satisfy the user.

[0087] The above technical solution can concatenate the image description text and one or more labels representing deformation capabilities to obtain the target description text. By inputting the target description text and the original image into the deformation combination module, the target image output by the deformation combination module can be obtained. The input of the image description text can improve the model's accurate recognition of image content, thereby providing further data support for the model's deformation processing and deformation capability combination, which is conducive to further improving the accuracy and reliability of the model's output results.

[0088] Figure 3 is a flowchart illustrating a model training method according to an exemplary embodiment of this disclosure. As shown in Figure 3, the method may include:

[0089] S11, acquire multiple sets of deformed image samples, each set of deformed image samples includes an original sample image, at least one label, and a deformed effect image of the original sample image corresponding to the at least one label.

[0090] Each set of deformed image samples may include one label or multiple labels. In the case of multiple labels, the deformation effect of the original sample image is the effect after processing by multiple deformation capabilities corresponding to the multiple labels.

[0091] It should be noted that during training, the original sample images are used as control conditions for the ControlNet network, and the at least one label is used as the text input to the network. The deformation effect images of the original sample images corresponding to the at least one label are used to supervise the ControlNet network. Due to the zero-shot learning capability of the ControlNet network itself, it can make full use of the existing training data to identify and process unseen categories or tasks. Therefore, even if there are combinations of labels that do not appear in the training data, the target deformation processing model can directly use the existing deformation capability combinations to predict other combinations. Typically, when the multiple sets of deformation image samples include n labels of deformation capabilities, the trained target deformation processing model can have the ability to present 2^n combinations.

[0092] S12, using the multiple sets of deformed image samples as training data, train the first preset initial ControlNet network to obtain the target deformation processing model.

[0093] The target deformation processing model can present the deformation effect after freely combining the deformation capabilities corresponding to the one or more labels used to represent deformation capabilities when the user inputs one or more labels used to represent deformation capabilities. This can effectively improve the model's efficiency in combining various deformation capabilities, thereby facilitating the rapid acquisition of image processing effects that satisfy the user.

[0094] For example, when multiple sets of deformed image samples include labels such as "elongated neck," "lengthened legs," and "thinned arms," ​​the trained target deformation processing model can arbitrarily combine the three deformation capabilities of elongating the neck, lengthening the legs, and thinning the arms. That is, it can provide images processed with elongated neck, lengthened legs, and thinned arms according to the user's needs; it can also provide images processed with a combination of elongated neck and lengthened legs, or a combination of elongated neck and thinned arms; it can also provide images processed with lengthened legs and thinned arms; or it can provide images processed with only one of the three deformation capabilities. The user's needs can be determined by the input labels. For example, if the user inputs the labels "elongated neck and lengthened legs," the user's need is determined to perform elongated neck and lengthened legs processing; if the user inputs the label "lengthened legs," the user's need is determined to perform leg lengthening processing only.

[0095] It should be noted that the specific implementation method for training the first preset initial ControlNet network using the multiple sets of deformed image samples as training data can refer to the training process of the ControlNet network, and this disclosure does not limit it.

[0096] The above technical solutions can train a model with more combinations of deformation capabilities by using a limited number of label combinations in a limited number of deformable image samples. Furthermore, the target deformation processing model can enable users to obtain images of combinations of deformation capabilities corresponding to multiple labels by inputting multiple labels, which can effectively improve the convenience of image processing and further enhance the user experience of the model.

[0097] Figure 4 is a flowchart illustrating a model training method in another exemplary embodiment of this disclosure. As shown in Figure 4, the method may include:

[0098] S21, acquire multiple sets of first image sample data, each set of first image sample data including the original sample image and the descriptive text annotation data corresponding to the original sample image.

[0099] S22, using the multiple sets of first image sample data as training data, a preset initial image description algorithm is trained to obtain an image description generation module.

[0100] The initial image description algorithm can be the initial NIC algorithm, the initial Visual Attention image description network, the initial SCA-CNN algorithm, or the initial Transformer algorithm, or other initial neural network algorithms.

[0101] It should be noted that the first image sample data is training data. The specific implementation method for training the preset initial image description algorithm can refer to the training process of the corresponding algorithm, and this disclosure does not limit it.

[0102] S23, acquire multiple sets of second image sample data. Each set of second image sample data includes an original sample image, labeled text data, and a deformation effect image corresponding to the original sample image. The labeled text data includes descriptive text labeled data corresponding to the original sample image and at least one label.

[0103] In this process, at least one label in the labeled text data can be pre-labeled by the user, and the labeled text data can be obtained by concatenating the pre-labeled labels by the user with the description text generated by the image description generation module through step S22.

[0104] For example, if the image description generation module generates the description text "A girl is dancing ballet", and the user inputs the labels "swan neck", "slim-arm", and "long legs", then the annotation text data can be "A girl is dancing ballet, swan neck, slim-arm, long legs".

[0105] S24, using the multiple sets of second image sample data as training data, train the second preset initial ControlNet network to obtain the deformation combination module.

[0106] The first preset initial ControlNet network and the second preset initial ControlNet network can be the same or different. For example, the first preset initial ControlNet network and the second preset initial ControlNet network can be ControlNet algorithms with the same hyperparameters, or they can be two ControlNet algorithms with some or all different hyperparameters.

[0107] S25, the image description generation module and the deformation combination module are combined into a target deformation processing model.

[0108] In this step, the output end of the image description generation module can be coupled to the first input end of the deformation combination module. The input end of the image description generation module and the second input end of the deformation combination module serve as the input ends of the target deformation processing model, and the output end of the deformation combination module serves as the output end of the target deformation processing model.

[0109] It should be noted that if the multiple sets of second image sample data include n labels representing deformation capabilities, the trained target deformation processing model can present 2^n combinations. This means that even if a combination of labels is not present in the training data, the target deformation processing model can directly predict other combinations using existing deformation capability combinations. Thus, when a user inputs one or more labels representing deformation capabilities, the target deformation processing model can present the deformation effect after freely combining the deformation capabilities corresponding to those labels. This effectively improves the model's efficiency in combining various deformation capabilities and provides reliable data for quickly obtaining images that satisfy the user.

[0110] It should be noted that the multiple sets of second image sample data are training data. The specific implementation method for training the second preset initial ControlNet network can refer to the training process of the ControlNet network, and this disclosure does not limit it.

[0111] The above technical solution integrates the image description generation module into the target deformation processing model, enabling the target deformation processing model to further improve the accuracy of image content recognition through the input of the image description text. This provides further data support for the combination of deformation processing and deformation capabilities of the model, which is conducive to further improving the accuracy and reliability of the model's output results.

[0112] Figure 5 is a block diagram illustrating an image processing apparatus according to an exemplary embodiment of the present disclosure; as shown in Figure 5, the image processing apparatus may include:

[0113] The first acquisition module 501 is configured to acquire an original image and a label for representing deformability, the original image including an object to be deformed according to the deformability;

[0114] The determining module 502 is configured to acquire the image description text of the original image; generate target description text based on the image description text and the tag; and generate a target image based on the original image and the target description text, wherein the target image includes the object after being deformed according to the deformation capability.

[0115] The above technical solutions can obtain a deformation effect formed by the combination of deformation capabilities corresponding to the one or more labels used to represent deformation capabilities by inputting an original image and one or more labels, thereby providing reliable technical support for human body beautification and helping to further improve the human body beautification effect.

[0116] Optionally, the training process of the target deformation processing model may include:

[0117] Multiple sets of deformed image samples are obtained. Each set of deformed image samples includes an original sample image, at least one label, and a deformed effect image of the original sample image corresponding to the at least one label.

[0118] Using the multiple sets of deformed image samples as training data, the first preset initial ControlNet network is trained to obtain the target deformation processing model.

[0119] Optionally, the target deformation processing model includes an image description generation module and a deformation combination module coupled to the image description generation module;

[0120] The determining module 502 is configured as follows:

[0121] The original image and one or more labels representing deformation capability are input into the target deformation processing model, which includes an image description generation module and a deformation combination module coupled to the image description generation module.

[0122] The image description generation module generates image description text corresponding to the original image.

[0123] The image description text and one or more tags used to represent deformation capability are concatenated to obtain the target description text;

[0124] The target description text and the original image are input into the deformation combination module to obtain the target image output by the deformation combination module.

[0125] Optionally, the target deformation processing model is trained in the following manner:

[0126] Acquire multiple sets of first image sample data, each set of first image sample data including the original sample image and the descriptive text annotation data corresponding to the original sample image;

[0127] Using the multiple sets of first image sample data as training data, a preset initial image description algorithm is trained to obtain the image description generation module;

[0128] Acquire multiple sets of second image sample data. Each set of second image sample data includes an original sample image, labeled text data, and a deformation effect image corresponding to the original sample image. The labeled text data includes descriptive text labeled data corresponding to the original sample image and at least one label.

[0129] Using the multiple sets of second image sample data as training data, the second preset initial ControlNet network is trained to obtain the deformation combination module;

[0130] The image description generation module and the deformation combination module are combined to form the target deformation processing model.

[0131] Optionally, combining the image description generation module and the deformation combination module into the target deformation processing model includes:

[0132] The output of the image description generation module is coupled to the first input of the deformation combination module. The input of the image description generation module and the second input of the deformation combination module serve as the input of the target deformation processing model, and the output of the deformation combination module serves as the output of the target deformation processing model.

[0133] The above technical solution integrates the image description generation module into the target deformation processing model, enabling the target deformation processing model to further improve the accuracy of image content recognition through the input of the image description text. This provides further data support for the combination of deformation processing and deformation capabilities of the model, which is conducive to further improving the accuracy and reliability of the model's output results.

[0134] Figure 6 is a block diagram illustrating a model training apparatus according to an exemplary embodiment of the present disclosure. The model training apparatus may include:

[0135] The second acquisition module 601 is configured to acquire multiple sets of deformation image samples, each set of deformation image samples including an original sample image, at least one label, and a deformation effect image of the original sample image corresponding to the at least one label.

[0136] The first training module 602 is configured to train the first preset initial ControlNet network using the multiple sets of deformed image samples as training data to obtain the target deformation processing model.

[0137] The above technical solutions can train a model with more combinations of deformation capabilities by using a limited number of label combinations in a limited number of deformable image samples. Furthermore, the target deformation processing model can enable users to obtain images of combinations of deformation capabilities corresponding to multiple labels by inputting multiple labels, which can effectively improve the convenience of image processing and further enhance the user experience of the model.

[0138] Figure 7 is a block diagram illustrating a model training apparatus according to another exemplary embodiment of the present disclosure, the model training apparatus may include:

[0139] The third acquisition module 701 is configured to acquire multiple sets of first image sample data, each set of first image sample data including an original sample image and descriptive text annotation data corresponding to the original sample image;

[0140] The second training module 702 is configured to use the multiple sets of first image sample data as training data to train a preset initial image description algorithm to obtain the image description generation module.

[0141] The fourth acquisition module 703 is configured to acquire multiple sets of second image sample data. Each set of second image sample data includes an original sample image, labeled text data, and a deformation effect image corresponding to the original sample image. The labeled text data includes descriptive text labeled data corresponding to the original sample image and at least one label.

[0142] The third training module 704 is configured to train the second preset initial ControlNet network using the multiple sets of second image sample data as training data to obtain the deformation combination module;

[0143] The coupling module 705 is configured to combine the image description generation module and the deformation combination module into a target deformation processing model.

[0144] Optionally, the coupling module 705 is configured as follows:

[0145] The output of the image description generation module is coupled to the first input of the deformation combination module. The input of the image description generation module and the second input of the deformation combination module serve as the input of the target deformation processing model, and the output of the deformation combination module serves as the output of the target deformation processing model.

[0146] The above technical solution integrates the image description generation module into the target deformation processing model, enabling the target deformation processing model to further improve the accuracy of image content recognition through the input of the image description text. This provides further data support for the combination of deformation processing and deformation capabilities of the model, which is conducive to further improving the accuracy and reliability of the model's output results.

[0147] Referring now to FIG8, a schematic diagram of the structure of an electronic device 800 suitable for implementing embodiments of the present disclosure is shown. The terminal devices in embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in FIG8 is merely an example and should not impose any limitation on the functionality and scope of use of embodiments of the present disclosure.

[0148] As shown in Figure 8, the electronic device 800 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the electronic device 800. The processing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0149] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 808 including, for example, magnetic tapes, hard disks, etc.; and communication devices 809. Communication device 809 allows electronic device 800 to communicate wirelessly or wiredly with other devices to exchange data. Although FIG8 shows an electronic device 800 with various devices, it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0150] According to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by a processing device 801, it performs the functions defined in the methods of embodiments of this disclosure.

[0151] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0152] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol), and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0153] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0154] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire an original image and a label representing a deformability, the original image including an object to be deformed according to the deformability; acquire image description text of the original image; generate target description text based on the image description text and the label; and generate a target image based on the original image and the target description text, the target image including the object deformed according to the deformability.

[0155] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0156] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0157] The modules described in the embodiments of this disclosure can be implemented in software or in hardware. The names of the modules do not necessarily limit the module itself; for example, the first acquisition module can also be described as "acquiring the original image and a label for representing deformation capability".

[0158] The functions described above in this disclosure can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0159] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0160] According to one or more embodiments of this disclosure, Example 1 provides an image processing method, the method comprising: acquiring an original image and a label for representing deformability, the original image including an object to be deformed according to the deformability; acquiring image description text of the original image; generating target description text based on the image description text and the label; and generating a target image based on the original image and the target description text, the target image including the object deformed according to the deformability.

[0161] According to one or more embodiments of this disclosure, Example 2 provides the image processing method described in Example 1, wherein the steps of obtaining image description text of the original image; generating target description text based on the image description text and the tag, and generating a target image based on the original image and the target description text, are implemented through a target deformation processing model, the training process of the target deformation processing model may include:

[0162] Multiple sets of deformed image samples are obtained. Each set of deformed image samples includes an original sample image, at least one label, and a deformed effect image of the original sample image corresponding to the at least one label.

[0163] Using the multiple sets of deformed image samples as training data, the first preset initial ControlNet network is trained to obtain the target deformation processing model.

[0164] According to one or more embodiments of this disclosure, Example 3 provides the image processing method described in Example 1, wherein the step of obtaining image description text of the original image; generating target description text based on the image description text and the tag; and generating a target image based on the original image and the target description text includes:

[0165] The original image and one or more labels representing deformation capability are input into the target deformation processing model, which includes an image description generation module and a deformation combination module coupled to the image description generation module.

[0166] The image description generation module generates image description text corresponding to the original image.

[0167] The image description text and one or more tags used to represent deformation capability are concatenated to obtain the target description text;

[0168] The target description text and the original image are input into the deformation combination module to obtain the target image output by the deformation combination module.

[0169] According to one or more embodiments of this disclosure, Example 4 provides the image processing method described in Example 3, wherein the target deformation processing model is trained in the following manner:

[0170] Acquire multiple sets of first image sample data, each set of first image sample data including the original sample image and the descriptive text annotation data corresponding to the original sample image;

[0171] Using the multiple sets of first image sample data as training data, a preset initial image description algorithm is trained to obtain the image description generation module;

[0172] Acquire multiple sets of second image sample data. Each set of second image sample data includes an original sample image, labeled text data, and a deformation effect image corresponding to the original sample image. The labeled text data includes descriptive text labeled data corresponding to the original sample image and at least one label.

[0173] Using the multiple sets of second image sample data as training data, the second preset initial ControlNet network is trained to obtain the deformation combination module;

[0174] The image description generation module and the deformation combination module are combined to form the target deformation processing model.

[0175] According to one or more embodiments of this disclosure, Example 5 provides the image processing method described in Example 4, wherein combining the image description generation module and the deformation combination module into the target deformation processing model includes:

[0176] The output of the image description generation module is coupled to the first input of the deformation combination module. The input of the image description generation module and the second input of the deformation combination module serve as the input of the target deformation processing model, and the output of the deformation combination module serves as the output of the target deformation processing model.

[0177] According to one or more embodiments of this disclosure, Example 6 provides a model training method, the method comprising:

[0178] Multiple sets of deformed image samples are obtained. Each set of deformed image samples includes an original sample image, at least one label, and a deformed effect image of the original sample image corresponding to the at least one label.

[0179] Using the multiple sets of deformed image samples as training data, the first preset initial ControlNet network is trained to obtain the target deformation processing model.

[0180] According to one or more embodiments of this disclosure, Example 7 provides a model training method, the method comprising:

[0181] Acquire multiple sets of first image sample data, each set of first image sample data including the original sample image and the descriptive text annotation data corresponding to the original sample image;

[0182] Using the multiple sets of first image sample data as training data, a preset initial image description algorithm is trained to obtain an image description generation module;

[0183] Acquire multiple sets of second image sample data. Each set of second image sample data includes an original sample image, labeled text data, and a deformation effect image corresponding to the original sample image. The labeled text data includes descriptive text labeled data corresponding to the original sample image and at least one label.

[0184] Using the multiple sets of second image sample data as training data, the second preset initial ControlNet network is trained to obtain the deformation combination module;

[0185] The image description generation module and the deformation combination module are combined into a target deformation processing model.

[0186] According to one or more embodiments of this disclosure, Example 8 provides the method of Example 7, wherein combining the image description generation module and the deformation combination module into the target deformation processing model includes:

[0187] The output of the image description generation module is coupled to the first input of the deformation combination module. The input of the image description generation module and the second input of the deformation combination module serve as the input of the target deformation processing model, and the output of the deformation combination module serves as the output of the target deformation processing model.

[0188] According to one or more embodiments of this disclosure, Example 9 provides an image processing apparatus, the apparatus comprising:

[0189] The first acquisition module is configured to acquire an original image and a label for representing deformation capability, the original image including an object to be deformed according to the deformation capability;

[0190] The determination module is configured to acquire image description text of the original image; generate target description text based on the image description text and the tag; and generate a target image based on the original image and the target description text, wherein the target image includes the object after being deformed according to the deformation capability.

[0191] According to one or more embodiments of this disclosure, Example 10 provides a model training apparatus, the apparatus comprising:

[0192] The second acquisition module is configured to acquire multiple sets of deformation image samples, each set of deformation image samples including an original sample image, at least one label, and a deformation effect image of the original sample image corresponding to the at least one label.

[0193] The first training module is configured to train a first preset initial ControlNet network using the multiple sets of deformed image samples as training data to obtain a target deformation processing model.

[0194] According to one or more embodiments of this disclosure, Example 11 provides a model training apparatus, the apparatus comprising:

[0195] The third acquisition module is configured to acquire multiple sets of first image sample data, each set of first image sample data including an original sample image and descriptive text annotation data corresponding to the original sample image;

[0196] The second training module is configured to use the multiple sets of first image sample data as training data to train a preset initial image description algorithm to obtain an image description generation module.

[0197] The fourth acquisition module is configured to acquire multiple sets of second image sample data. Each set of second image sample data includes an original sample image, labeled text data, and a deformation effect image corresponding to the original sample image. The labeled text data includes descriptive text labeled data corresponding to the original sample image and at least one label.

[0198] The third training module is configured to train the second preset initial ControlNet network using the multiple sets of second image sample data as training data to obtain the deformation combination module;

[0199] The coupling module is configured to combine the image description generation module and the deformation combination module into a target deformation processing model.

[0200] According to one or more embodiments of the present disclosure, Example 12 provides a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processing device, implements the steps of the method described in any one of Examples 1-8.

[0201] According to one or more embodiments of this disclosure, Example 13 provides an electronic device comprising:

[0202] A storage device on which computer programs are stored;

[0203] A processing device for executing the computer program in the storage device to implement the steps of any one of the methods in Examples 1-8.

[0204] According to one or more embodiments of the present disclosure, Example 14 provides a computer program product including a computer program that, when executed by a processor, implements the steps of the method described in any one of Examples 1-8.

[0205] The above description is merely an embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0206] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0207] Although this disclosure has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative forms of implementing the claims. Regarding the apparatus in the above embodiments, the specific manner in which the various modules perform their operations has been described in detail in the embodiments relating to the method, and will not be elaborated upon here.

Claims

1. An image processing method, comprising: Acquire an original image and a label for representing the deformability, the original image including objects that will be deformed according to the deformability; Obtain the image description text of the original image; Based on the image description text and the tag, generate target description text; A target image is generated based on the original image and the target description text, the target image comprising the object after deformation according to the deformation capability.

2. The image processing method according to claim 1, wherein, The steps of obtaining image description text of the original image; generating target description text based on the image description text and the label; and generating a target image based on the original image and the target description text are implemented through a target deformation processing model. The training process of the target deformation processing model includes: Multiple sets of deformed image samples are obtained. Each set of deformed image samples includes an original sample image, at least one label, and a deformed effect image of the original sample image corresponding to the at least one label. Using the multiple sets of deformed image samples as training data, the first preset initial ControlNet network is trained to obtain the target deformation processing model.

3. The image processing method according to claim 1, wherein, The process involves obtaining the image description text of the original image; and generating target description text based on the image description text and the tag. A target image is generated based on the original image and the target description text. The target image includes the object after deformation according to the deformation capability, including: The original image and one or more labels representing deformation capability are input into the target deformation processing model, which includes an image description generation module and a deformation combination module coupled to the image description generation module. The image description generation module generates image description text corresponding to the original image. The image description text and one or more tags used to represent deformation capability are concatenated to obtain the target description text; The target description text and the original image are input into the deformation combination module to obtain the target image output by the deformation combination module.

4. The image processing method according to claim 3, wherein, The target deformation processing model is trained in the following way: Acquire multiple sets of first image sample data, each set of first image sample data including the original sample image and the descriptive text annotation data corresponding to the original sample image; Using the multiple sets of first image sample data as training data, a preset initial image description algorithm is trained to obtain the image description generation module; Acquire multiple sets of second image sample data. Each set of second image sample data includes an original sample image, labeled text data, and a deformation effect image corresponding to the original sample image. The labeled text data includes descriptive text labeled data corresponding to the original sample image and at least one label. Using the multiple sets of second image sample data as training data, the second preset initial ControlNet network is trained to obtain the deformation combination module; The image description generation module and the deformation combination module are combined to form the target deformation processing model.

5. The image processing method according to claim 4, wherein, The step of combining the image description generation module and the deformation combination module into the target deformation processing model includes: The output of the image description generation module is coupled to the first input of the deformation combination module. The input of the image description generation module and the second input of the deformation combination module serve as the input of the target deformation processing model, and the output of the deformation combination module serves as the output of the target deformation processing model.

6. A model training method, comprising: Multiple sets of deformed image samples are obtained. Each set of deformed image samples includes an original sample image, at least one label, and a deformed effect image of the original sample image corresponding to the at least one label. Using the multiple sets of deformed image samples as training data, the first preset initial ControlNet network is trained to obtain the target deformation processing model.

7. A model training method, comprising: Acquire multiple sets of first image sample data, each set of first image sample data including the original sample image and the descriptive text annotation data corresponding to the original sample image; Using the multiple sets of first image sample data as training data, a preset initial image description algorithm is trained to obtain an image description generation module; Acquire multiple sets of second image sample data. Each set of second image sample data includes an original sample image, labeled text data, and a deformation effect image corresponding to the original sample image. The labeled text data includes descriptive text labeled data corresponding to the original sample image and at least one label. Using the multiple sets of second image sample data as training data, the second preset initial ControlNet network is trained to obtain the deformation combination module; The image description generation module and the deformation combination module are combined into a target deformation processing model.

8. The method according to claim 7, wherein, The step of combining the image description generation module and the deformation combination module into the target deformation processing model includes: The output of the image description generation module is coupled to the first input of the deformation combination module. The input of the image description generation module and the second input of the deformation combination module serve as the input of the target deformation processing model, and the output of the deformation combination module serves as the output of the target deformation processing model.

9. An image processing apparatus, comprising: The first acquisition module is configured to acquire an original image and a label for representing deformability, the original image including an object to be deformed according to the deformability; The determination module is configured to acquire the image description text of the original image; Based on the image description text and the label, a target description text is generated; based on the original image and the target description text, a target image is generated, the target image including the object after deformation according to the deformation capability.

10. A model training device, comprising: The second acquisition module is configured to acquire multiple sets of deformation image samples, each set of deformation image samples including an original sample image, at least one label, and a deformation effect image of the original sample image corresponding to the at least one label. The first training module is configured to train a first preset initial ControlNet network using the multiple sets of deformed image samples as training data to obtain a target deformation processing model.

11. A model training device, comprising: The third acquisition module is configured to acquire multiple sets of first image sample data, each set of first image sample data including an original sample image and descriptive text annotation data corresponding to the original sample image; The second training module is configured to use the multiple sets of first image sample data as training data to train a preset initial image description algorithm to obtain an image description generation module. The fourth acquisition module is configured to acquire multiple sets of second image sample data. Each set of second image sample data includes an original sample image, labeled text data, and a deformation effect image corresponding to the original sample image. The labeled text data includes descriptive text labeled data corresponding to the original sample image and at least one label. The third training module is configured to train the second preset initial ControlNet network using the multiple sets of second image sample data as training data to obtain the deformation combination module; The coupling module is configured to combine the image description generation module and the deformation combination module into a target deformation processing model.

12. A computer-readable medium storing a computer program, wherein, When the program is executed by the processing device, it implements the image processing method or model training method according to any one of claims 1-8.

13. An electronic device, wherein, include: Storage device containing computer programs; The processing apparatus is configured to execute the computer program in the storage device to implement the image processing method or model training method according to any one of claims 1-8.

14. A computer program product comprising a computer program, wherein, When the computer program is executed by the processor, it implements the image processing method or model training method according to any one of claims 1-8.

Images