Image editing method and related equipment

Abstract

An embodiment of the present application provides an image editing method and related equipment, which relates to the field of AI; in the method, a first image and a first text are obtained, and the first text represents the editing requirements of the first image. The editing requirements are decoupled into at least two first basic editing tasks according to the first text, and each first basic editing task is used to implement an image processing stage in the process of editing the first image to meet the editing requirements. A first edited image is generated based on the first image, the first text and at least two first basic editing tasks. This solution decouples the editing requirements to obtain multiple basic editing tasks, and the multiple basic editing tasks can be executed by one image editing model, so that multiple image editing requirements can be met by using one image editing model, and there is no need to deploy different subtask models for different editing tasks, which is conducive to improving image editing efficiency.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of images, and in particular to an image editing method and related equipment. Background Art

[0002] An image is a vivid, lifelike description or portrait of an objective object. It is the most commonly used information carrier in human social activities. Alternatively, an image is a representation of an objective object, containing relevant information about the depicted object. It is people's primary source of information.

[0003] With the development of artificial intelligence (AI) technology, AI models can be used to edit images according to different editing requirements, generating images that meet the editing requirements. An example of an AI model is the ControlNet model. However, as the number of image editing tasks increases, the ControlNet model needs to define and deploy more and more subtask models. For example, the task of changing clothes requires the deployment of a subtask model, while the task of moving an object requires the deployment of another subtask model. Therefore, when a new editing task is required, a new subtask model needs to be deployed to perform the newly emerged editing task. Model deployment is time-consuming and cannot meet the efficiency requirements of image editing. Summary of the Invention

[0004] The present application provides an image editing method and related equipment, which can improve image editing efficiency.

[0005] In a first aspect, an image editing method is provided. The method can be executed by an image editing device or a chip in the image editing device.

[0006] The image editing method includes the following steps: obtaining a first image and a first text, wherein the first text represents an editing requirement for the first image; decomposing the editing requirement into at least two first basic editing tasks based on the first text, each first basic editing task being used to implement an image processing stage in the process of editing the first image to meet the editing requirement; and generating a first edited image based on the first image, the first text, and the at least two first basic editing tasks.

[0007] The first image may be a single image, and accordingly, the first edited image is a single edited image. The first image may also be multiple images, and accordingly, the first edited image is multiple edited images. For example, when the first image is multiple images, it can be understood that this solution can be applied to video editing, where a video is composed of multiple temporally consecutive images.

[0008] The above-mentioned image processing stage can be a processing stage for a complete function, such as any one of the functions of generation function, fusion function, picking function, elimination function, portrait beauty function, portrait shaping function, etc., or any combination of the above functions. The above-mentioned image processing stage can also be a processing stage for realizing partial functions, such as generation function 1, generation function 2, fusion function 1, fusion function 2, fusion function 3, etc., wherein the combination of the two image processing stages corresponding to generation function 1 and generation function 2 can realize the complete generation function. Similarly, the combination of the three image processing stages corresponding to fusion function 1, fusion function 2, and fusion function 3 can realize the complete fusion function.

[0009] When editing a first image, this solution decouples the editing requirements of the first image into at least two first basic editing tasks based on the first text. The first edited image is then generated based on these at least two first basic editing tasks, the first image, and the first text. Therefore, by decoupling the editing requirements into multiple basic editing tasks, this solution can be executed by a single image editing model. This allows a single image editing model to meet multiple image editing requirements, eliminating the need to deploy separate subtask models for different editing tasks and improving image editing efficiency.

[0010] In a possible implementation of the first aspect, generating the first edited image based on the first image, the first text, and the at least one first basic editing task specifically includes the following steps: generating first noise based on the first text; generating at least one second noise based on the first image and the at least two first basic editing tasks; inputting the first noise and the at least one second noise into a noise decision network for processing to obtain third noise; and generating the first edited image based on the third noise.

[0011] It can be seen that this scheme uses the noise decision network to obtain a third noise that matches the first text from the first noise and at least one second noise, and then generates a first edited image based on the third noise, which can improve the matching degree between the first edited image and the first text, that is, the editing accuracy.

[0012] In a possible implementation of the first aspect, the image editing method further includes the following steps: determining a first score of the first edited image based on the first edited image, the first image, and the first text, wherein the first score represents the editing quality of the first edited image.

[0013] It can be seen that in this solution, the first image, the first text and the first edited image are used to determine the first score reflecting the quality of image editing, so that the user can understand the editing effect of the first edited image according to the first score.

[0014] In one possible implementation of the first aspect, the image editing method further includes the following steps: decoupling the editing requirement into at least two second basic editing tasks based on the first text and the first score. Each of the second basic editing tasks is used to implement an image processing stage in the process of editing the first image to meet the editing requirement. A second edited image is generated based on the first image, the first text, and the at least two second basic editing tasks, where the editing quality of the second edited image is higher than that of the first edited image.

[0015] Similar to the first basic editing task mentioned above, the image processing stage implemented by the second basic editing task may be a processing stage with complete functions, or the image processing stage implemented by the second basic editing task may be a processing stage with partial functions.

[0016] The first image may be one image, and correspondingly, the second edited image may be one edited image. The first image may also be multiple images, and correspondingly, the second edited image may be multiple edited images.

[0017] In this solution, after obtaining a first score for a first edited image, the editing requirements for the first image can be decoupled based on the first score and the first text to obtain at least two second basic editing tasks. A second edited image can then be generated based on the first image, the first text, and the at least two second basic editing tasks. The first score is used as a factor influencing the generation of the edited image to produce a second edited image with higher editing quality.

[0018] In a possible implementation of the first aspect, the first basic editing task or the second basic editing task includes at least one of the following: generation, fusion, picking, elimination, portrait beautification, and portrait shaping.

[0019] In a second aspect, the present application also provides an image editing device, comprising an acquisition module, a decoupling module and a generation module.

[0020] The acquisition module is configured to acquire a first image and a first text, wherein the first text represents an editing requirement of the first image.

[0021] The decoupling module is configured to decouple the editing requirement into at least two first basic editing tasks according to the first text. Each first basic editing task is configured to implement an image processing stage in the process of editing the first image to meet the editing requirement.

[0022] A generating module is configured to generate a first edited image based on a first image, a first text, and at least two first basic editing tasks.

[0023] When editing a first image, the image editing device of this solution decouples the editing requirements of the first image into at least two first basic editing tasks based on the first text, and then generates a first edited image based on the at least two first basic editing tasks, the first image, and the first text. Therefore, the image editing device of this solution decomposes the editing requirements into multiple basic editing tasks, and these multiple basic editing tasks can be executed by a single image editing model. This allows a single image editing model to meet multiple image editing requirements, eliminating the need to deploy different subtask models for different editing tasks, thereby improving image editing efficiency.

[0024] In a possible implementation of the second aspect, the generation module is specifically configured to: generate first noise based on the first text; generate at least one second noise based on the first image and at least two first basic editing tasks; input the first noise and the at least one second noise into a noise decision network for processing to obtain third noise; and generate the first edited image based on the third noise.

[0025] In a possible implementation of the second aspect, the image editing device further includes a determination module.

[0026] The determining module is configured to determine a first score of the first edited image based on the first edited image, the first image, and the first text, wherein the first score represents an editing quality of the first edited image.

[0027] In a possible implementation of the second aspect, the decoupling module is further configured to decouple the editing requirement into at least two second basic editing tasks based on the first text and the first score. Each second basic editing task is configured to implement an image processing stage in a process of editing the first image to meet the editing requirement.

[0028] The above-mentioned generation module is further used to generate a second edited image based on the first image, the first text and at least two second basic editing tasks, and the editing quality of the second edited image is higher than the editing quality of the first edited image.

[0029] In a possible implementation of the second aspect, the first basic editing task or the second basic editing task includes at least one of the following: generation, fusion, picking, elimination, portrait beautification, and portrait shaping.

[0030] In a third aspect, the present application also provides an image editing device, comprising a processor and a memory, wherein the processor and the memory are connected, wherein the memory is used to store program code, and the processor is used to call the program code to execute the image editing method as described in the first aspect.

[0031] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the image editing method as described in the first aspect.

[0032] In a fifth aspect, the present application further provides a computer program product comprising instructions, which, when run on a computer, enables the computer to execute the image editing method described in the first aspect.

[0033] In a sixth aspect, the present application further provides a chip comprising a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface and executes the image editing method described in the first aspect.

[0034] Optionally, as an implementation method, the chip may further include a memory, in which instructions are stored, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute the image editing method described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The following is an introduction to the drawings used in the embodiments of this application.

[0036] Figure 1A is a schematic diagram of an image editing method provided in an embodiment of the present application;

[0037] Figure 1B is a schematic diagram of another image editing method provided in an embodiment of the present application;

[0038] Figure 1C is a schematic diagram of another image editing method provided in an embodiment of the present application;

[0039] Figure 1D is a schematic diagram of another image editing method provided in an embodiment of the present application;

[0040] Figure 2 It is a schematic diagram of the system architecture provided by the embodiment of the present application;

[0041] Figure 3 This is a flowchart of an image editing method provided by an embodiment of the present application;

[0042] Figure 4 This is a structural diagram of an image editing model provided in an embodiment of the present application;

[0043] Figure 5 This is a structural diagram of an editing model provided in an embodiment of the present application;

[0044] Figure 6is a structural diagram of an image editing device provided in an embodiment of the present application;

[0045] Figure 7 It is a structural diagram of an image editing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The technical solution in this application will be described below with reference to the accompanying drawings.

[0047] Due to the applications involved in the embodiments of the present application, in order to facilitate understanding, the relevant terms and other related concepts involved in the embodiments of the present application are first introduced below.

[0048] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0049] The "at least one" mentioned in the embodiments of this application refers to one or more, and "plurality" refers to two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, (a and b), (a and c), (b and c), or (a and b and c), where a, b, c can be single or multiple. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can be represented by: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the related objects before and after are in an "or" relationship. The serial numbers of the steps in the embodiments of this application (such as step S1, step S21, etc.) are only for distinguishing different steps and do not limit the order of execution between the steps.

[0050] Furthermore, unless otherwise specified, ordinal numbers such as "first" and "second" in the embodiments of this application are used to distinguish multiple objects and are not used to limit the order, timing, priority, or importance of multiple objects. For example, the first device and the second device are only for ease of description and do not indicate differences in structure, importance, etc. between the first and second devices. In some embodiments, the first device and the second device can also be the same device.

[0051] In the above embodiments, the term "when" can be interpreted to mean "if...", "after...", "in response to determining...", or "in response to detecting...", depending on the context. The above are merely optional embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the concepts and principles of the present application shall be included in the scope of protection of the present application.

[0052] Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that seeks to understand the essence of intelligence and develop new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making. Research in the field of AI includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and fundamental AI theory.

[0053] AI models can be used to edit images based on different editing requirements, generating images that meet these requirements. An example of an AI model is the ControlNet model. However, as image editing tasks increase, the ControlNet model requires the definition and deployment of multiple subtask models. For example, a task like changing clothes requires one subtask model, while another requires the deployment of another subtask model for moving an object. Consequently, whenever a new editing task arises, a new subtask model must be deployed to perform it. This time-consuming model deployment fails to meet the efficiency requirements of image editing.

[0054] To this end, an embodiment of the present application provides an image editing method, which decouples the editing requirements of an image to obtain multiple basic editing tasks. Multiple basic editing tasks can be performed by one image editing model, so that a variety of image editing requirements can be met by using one image editing model. There is no need to deploy different subtask models for different editing tasks, which is beneficial to improving image editing efficiency. For example: when the editing requirement is to change clothes, it can be achieved through the two basic editing tasks of picking up and generating. When the editing requirement is to move the target object, it can be achieved through the three basic editing tasks of picking up, eliminating, and merging. And when the editing requirement is to change the background, it can be achieved through the three basic editing tasks of picking up, generating, and merging.

[0055] refer to Figure 1A When the editing requirement is to combine the objects in the photo, the image editing method of the embodiment of the present application can be used to combine the objects in the photo. Figure 1AThe kitten and puppy in the two images on the left are combined to get Figure 1A Image on the right.

[0056] When the editing requirement is to fuse multiple photos from different angles into a 360-degree panoramic image, the fused image obtained by the image editing method of the embodiment of the present application is as follows: Figure 1B shown.

[0057] When the editing requirement is to make a character in the image jump higher, the image editing method of the embodiment of the present application can be used to Figure 1C In the left image, the position of the girl on the left is raised, and we get Figure 1C Image on the right.

[0058] When the editing requirement is to process the image into an image with a fragmentation effect, the image editing method of the embodiment of the present application can be used to obtain an image with a fragmentation effect, such as Figure 1D shown.

[0059] The image editing method of the embodiment of the present application can be executed by an image editing device, or by a chip in the image editing device.

[0060] The following describes the system architecture provided by the embodiments of the present application.

[0061] See attached Figure 2 , an embodiment of the present application provides a system architecture 200. As shown in the system architecture 200, the data acquisition device 260 is used to acquire processing data of the image editing model. In this embodiment, the processing data is the image to be edited. The data acquisition device 260 stores the processing data in the database 230. The training device 220 can train the initial image editing model based on the processing data maintained in the database 230 to obtain the image editing model 201. Exemplarily, in the embodiment of the present application, after completing the model training, the image editing model 201 can obtain the processing capabilities of various basic editing tasks (such as the first basic editing task and the second basic editing task below) and the ability to generate edited images. The training device 220 can be a server or a cloud service device, etc., and can also be a mobile phone terminal, a tablet computer, a laptop computer, AR / VR, a vehicle-mounted terminal, a monitoring device, a vehicle-mounted automatic driving system, an unmanned vending machine, and other devices. It should be noted that in actual applications, the processing data maintained in the database 230 does not necessarily come from the collection of the data acquisition device 260, but may also be received from other devices, such as a camera. It should also be noted that the training device 220 does not necessarily perform model training entirely based on the processed data maintained by the database 230, but may also obtain processed data from the cloud or other places for model training. The above description should not be regarded as a limitation on the embodiments of the present application.

[0062] The image editing model 201 obtained after the training device 220 is processed can be applied to a system or device with the same hardware configuration as the training device 220, such as Figure 2 The terminal device 210 shown can edit an image based on the image editing model 201 to obtain an edited image. The terminal device 210 can be a terminal such as a mobile phone, tablet computer, laptop computer, AR / VR, vehicle-mounted terminal, monitoring equipment, vehicle-mounted autonomous driving system, unmanned vending machine, etc., or a server or cloud. The training device 220 and the terminal device 210 can be the same device.

[0063] In the attached Figure 2 In the embodiment, the terminal device 210 is configured with an I / O interface 212 for data interaction with external devices. The user can input data to the I / O interface 212 through the client device 240. In this embodiment, the input data is an image to be edited, which can be input by the user or from the database 230; wherein the client device 240 can be an image acquisition device, such as a camera.

[0064] The preprocessing module 213 is configured to preprocess the input data received by the I / O interface 212. In the embodiment of the present application, the preprocessing module 213 is configured to preprocess the input data received by the I / O interface 212, and the preprocessed data enters the computing module 211. In the embodiment of the present application, the preprocessing module 213 can be configured to perform at least one of filtering, enhancement, and denoising on the image to be edited to obtain an image to be edited that meets the requirements.

[0065] When the terminal device 210 preprocesses the input data, or when the computing module 211 of the terminal device 210 performs calculations and other related processing, the terminal device 210 can call the data, code, etc. in the data storage system 250 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing in the data storage system 250.

[0066] Finally, the I / O interface 212 returns the model processing result of the input data, that is, the edited image, to the client device 240 to provide it to the user. In this case, the client device 240 can be a display.

[0067] In the attached Figure 2In the case shown in , the user can manually input data, which can be operated through the interface provided by the I / O interface 212. In another case, the client device 240 can automatically send input data to the I / O interface 212. If the client device 240 is required to automatically send input data and requires user authorization, the user can set the corresponding permissions in the client device 240. The user can view the results output by the terminal device 210 on the client device 240, and the specific presentation form can be display, sound, action, etc. The client device 240 can also be used as a data acquisition terminal to collect data such as Figure 2 The input data of the input I / O interface 212 and the output result of the output I / O interface 212 are taken as new sample data and stored in the database 230. Of course, it is also possible to collect the data without going through the client device 240, but directly by the I / O interface 212. Figure 2 The input data of the input I / O interface 212 and the output result of the output I / O interface 212 are stored in the database 230 as new sample data.

[0068] It is worth noting that the Figure 2 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in the attached Figure 2 In the embodiment, the data storage system 250 is an external memory relative to the terminal device 210. In other cases, the data storage system 250 can also be placed in the terminal device 210.

[0069] The image editing method of an embodiment of the present application is described in detail below, with an image editing device being taken as an example of the execution subject.

[0070] refer to Figure 3 , Figure 3 300 is a flowchart of an image editing method provided in an embodiment of the present application. The above image editing method 300 includes the following steps:

[0071] 301. An image editing apparatus obtains a first image and a first text.

[0072] The first image can be a single image, and accordingly, the first edited image is a single edited image. The first image can also be multiple images, and accordingly, the first edited image is multiple edited images. For example, when the first image is multiple images, it can be understood that this solution can be applied to video editing, where a video is composed of multiple temporally continuous images. The first text represents the editing requirements for the first image. Generally, the first text is composed of text expressed in a natural language.

[0073] For example, the first image may be obtained from data stored locally on the image editing device, or captured in real time by the image editing device from a camera. The first text may be input into the image editing device by a user via an input device (keyboard, voice input, etc.). For example, if the first text is "Change clothes for the person in the image," the corresponding editing requirement is "Change clothes."

[0074] 302. The image editing device decouples the editing requirement into at least two first basic editing tasks according to the first text.

[0075] Among them, each first basic editing task is used to implement an image processing stage in the process of editing the first image to meet the editing requirements. The above-mentioned image processing stage can be a processing stage with complete functions, such as generation function, fusion function, picking function, elimination function, portrait beauty function, and portrait shaping function. The elimination function refers to removing the selected image area from the image. The above-mentioned image processing stage can also be a processing stage that realizes partial functions, such as generation function 1, generation function 2, fusion function 1, fusion function 2, fusion function 3, etc., wherein the two image processing stages corresponding to generation function 1 and generation function 2 are combined to realize the complete generation function. Similarly, the three image processing stages corresponding to fusion function 1, fusion function 2, and fusion function 3 are combined to realize the complete fusion function.

[0076] For example, when the editing requirement is to change clothes, it can be achieved through the two first basic editing tasks of picking and generating.

[0077] 303. The image editing apparatus generates a first edited image based on the first image, the first text, and at least two first basic editing tasks.

[0078] In the embodiments of the present application, multiple basic editing tasks are generated from editing requirements. These basic editing tasks can be executed by a single image editing model, enabling the use of a single image editing model to meet multiple image editing requirements. This eliminates the need to deploy different subtask models for different editing tasks, thereby improving image editing efficiency. Thus, when faced with a new editing requirement, a new editing result can be obtained simply by changing the first text and first image input.

[0079] When the image editing device decouples the editing requirement into a first basic editing task based on the first text, specifying the image editing requirement to the first basic editing task can help quickly complete the editing process of the image.

[0080] The first basic editing task includes at least one of the following: generate, merge, pick, remove, portrait beautify, and portrait reshape. The first basic editing task can be any one of the above or a combination thereof. The first basic editing task can also be other possible tasks, which are not limited to this.

[0081] In one possible implementation, step 303 specifically includes the following steps:

[0082] 331. The image editing apparatus generates first noise based on the first text.

[0083] In the embodiments of the present application, noise can be understood as a hidden variable. A first noise is generated based on the first text, and the first noise matches the first text. The specific method for generating the first noise based on the text can adopt existing technologies and is not limited to this. For example, a stable diffusion model or the encoding part of a diffusion model is used to implement it.

[0084] Here, a corresponding text vector (or semantic vector) is first generated based on the first text, and then a first noise is generated based on the text vector. In addition, an image can be restored (or generated) based on the first noise.

[0085] 332. The image editing apparatus generates at least one second noise based on the first image and at least two first basic editing tasks.

[0086] Specifically, since the at least two first basic editing tasks are decoupled from the first text, after processing the first image according to the at least two first basic editing tasks to obtain processing results of the respective first basic editing tasks, the processing results are then fused to obtain at least one second noise. Furthermore, at least one image can be restored (or generated) based on the at least one second noise.

[0087] 333. The image editing device inputs the first noise and at least one second noise into a noise decision network for processing to obtain a third noise.

[0088] A noise decision network is used to obtain a third noise matching the first text from the first noise and at least one second noise.

[0089] Optionally, feature extraction may be performed on the noise (the first noise and at least one second noise) to obtain a feature vector, and then the feature vector corresponding to the noise is input into a noise decision network for processing to obtain the third noise.

[0090] 334. The image editing device generates a first edited image based on the third noise.

[0091] Generating the first edited image based on the third noise matching the first text can improve the matching degree between the first edited image and the first text, that is, the editing accuracy.

[0092] For example, the noise decision network may first select a second noise from at least one second noise, and then add the selected second noise to the first noise to obtain the third noise. The second noise may be selected randomly or according to a specific rule, which is not limited to this.

[0093] As another example, after the noise decision network has learned the network parameters through training, it can generate a new noise, namely, a third noise, by fusing the first noise and at least one second noise.

[0094] In a possible implementation, the image editing method 300 further includes the following steps:

[0095] The image editing apparatus determines a first score of the first edited image based on the first edited image, the first image, and the first text, the first score representing an editing quality of the first edited image.

[0096] Specifically, the editing quality includes at least one of the following: accuracy of natural language understanding, fidelity, degree of harmonization, rationality of composition, degree of distortion, etc. The editing quality may also be other indicators for evaluating image quality, which is not limited thereto.

[0097] The accuracy of natural language understanding refers to whether the first text is correctly understood, that is, whether the edited image meets the editing requirements corresponding to the first text.

[0098] Fidelity refers to whether the edited image is distorted compared to the original image. For example, whether the same objects in the edited image and the original image remain consistent and without any differences.

[0099] The degree of harmonization refers to the degree to which the edited image looks awkward.

[0100] Distortion refers to the degree to which objects and / or backgrounds in the edited image are drastically changed.

[0101] In an embodiment of the present application, a first image, a first text, and a first edited image are used to determine a first score reflecting the quality of image editing, so that a user can understand the editing effect of the first edited image based on the first score.

[0102] In a possible implementation, the image editing method 300 further includes the following steps:

[0103] 304. The image editing apparatus decouples the editing requirement into at least two second basic editing tasks according to the first text and the first score.

[0104] For example, the user decides whether to execute step 304 based on the specific situation of the first score. Alternatively, when the first score is less than the score threshold, step 304 is executed. The specific value of the score threshold can be set according to actual situation.

[0105] Each of the above-mentioned second basic editing tasks is used to implement an image processing stage in the process of editing the first image to meet the editing requirements. Similar to the first basic editing tasks, the image processing stage implemented by the above-mentioned second basic editing tasks can be a processing stage for a complete function, or the image processing stage implemented by the second basic editing tasks can also be a processing stage for a partial function.

[0106] 305. The image editing apparatus generates a second edited image based on the first image, the first text, and at least two second basic editing tasks, wherein the editing quality of the second edited image is higher than the editing quality of the first edited image.

[0107] The first image may be one image, and correspondingly, the second edited image may be one edited image. The first image may also be multiple images, and correspondingly, the second edited image may be multiple edited images.

[0108] For example, the editing quality of the second edited image is higher than that of the first edited image, which may be achieved by having a score for the second edited image higher than the first score. Alternatively, the performance of the second edited image in a certain quality dimension may be better than that of the first edited image in that quality dimension, for example, the fidelity of the second edited image may be higher than that of the first edited image, or the distortion of the second edited image may be lower than that of the first edited image.

[0109] In an embodiment of the present application, after obtaining a first score for a first edited image, the editing requirements for the first image can be decoupled based on the first score and the first text to obtain at least two second basic editing tasks. A second edited image can then be generated based on the first image, the first text, and the at least two second basic editing tasks. The first score can be used as a factor influencing the generation of the edited image to generate a second edited image with higher editing quality.

[0110] For example, the user may change the first score and then input the score into the image editing model to generate an edited image with higher editing quality.

[0111] When the image editing device decouples the editing requirement into a second basic editing task based on the first score and the first text, specifying the image editing requirement to the second basic editing task can help quickly complete the image editing process.

[0112] The second basic editing task includes at least one of the following: generate, merge, pick, remove, portrait beautify, and portrait reshape. The second basic editing task can be any one of the above or a combination thereof. The second basic editing task can also be other possible tasks, which are not limited to this.

[0113] The following describes the structure of a possible image editing model:

[0114] refer to Figure 4 , Figure 4 : is a structural diagram of an image editing model provided in an embodiment of the present application. In this embodiment, the content input to the image editing model includes a first text and a dynamic condition, and the above-mentioned dynamic condition includes a first image. Further, optionally, the dynamic condition also includes a score for the edited image (such as the first score above). The output content of the image editing model includes the edited image. Optionally, the output content of the image editing model also includes a score corresponding to the edited image.

[0115] The image editing model includes a large language model, an editing model, and an evaluation model. The large language model (LLM) is used to generate a corresponding text vector based on the first text. Specifically, the large language model is a model based on machine learning and natural language processing technologies. It learns to serve human language understanding and generation capabilities by training on large amounts of text data. The large language model can handle a variety of natural language tasks, such as text classification, question-answering, and conversation, and is an important path to artificial intelligence.

[0116] The editing model is used to generate an edited image based on the above text vector and dynamic conditions. The edited image obtained by the editing model serves as one of the inputs of the evaluation model. In this way, the evaluation model can obtain an evaluation score corresponding to the edited image based on the text vector, dynamic conditions, and edited image. The above evaluation score can be output to the user to let the user know the editing quality of the edited image. Optionally, the evaluation score can also be input into the image editing model as one of the dynamic conditions, so that the image editing model supports multiple complex condition inputs and improves the controllability of the dynamic conditions.

[0117] refer to Figure 5 , Figure 5It is a structural diagram of an editing model provided by an embodiment of the present application; the above-mentioned editing model includes a first noise generation network, a second noise generation network, a noise decision network and an image generation network. Among them, the first noise generation network is used to generate a first noise ε based on the text vector of the first text. The second noise network is decoupled into at least two basic editing tasks based on the text vector, and generates processing results corresponding to the basic editing tasks based on the at least two basic editing tasks and the text vector, and then fuses the processing results of the at least two basic editing tasks to obtain at least one second noise. In this embodiment, the second noise generation network generates N second noises, which are ε 0 , ε 1 、……、ε N The specific value of N can be set according to the actual situation and is not limited. For example, N is the number of dynamic conditions. When the dynamic condition is a first image, N is 1. When the dynamic condition is two first images, N is 2. When the dynamic condition is two first images and a score, N is 3.

[0118] The noise decision network is used to obtain an optimal third noise ε according to the first noise and at least one second noise generated by the super control network ’ The image generation network is used to generate the image according to the third noise ε ’ to generate a valid edited image.

[0119] Exemplarily, the first noise generation network and the image generation network may be two parts of an image generation model, that is, the first noise generation network is the encoding part of the image generation model, and the image generation network is the decoding part of the image generation model. For example, the image generation model may be a stable diffusion model or a diffusion model.

[0120] In another exemplary embodiment, the second noise generation network supports variable input. The second noise generation network can be implemented using a super control network SuperNet, which can include multiple sub-networks and a fusion network. Exemplarily, each sub-network is used to implement a basic editing task of an image (such as a first basic editing task and a second basic editing task), and the fusion network is used to fuse the processing results of the sub-networks to obtain the second noise. Alternatively, the second noise generation network controls network learning under various conditions based on a learnable prompt paradigm, so that the second noise generation network has the processing capabilities of multiple basic editing tasks and the ability to generate the second noise.

[0121] The numbering of the steps above is only for distinguishing different steps and does not limit the order in which the steps are executed.

[0122] The above describes in detail the method of the embodiment of the present application. The following describes the device provided by the embodiment of the present application.

[0123] Figure 6 A schematic diagram of the structure of a possible image editing device provided in an embodiment of the present application. This image editing device can be used to implement the functions of the above-mentioned method embodiment, thereby also achieving the beneficial effects of the above-mentioned method embodiment. In the embodiment of the present application, the image editing device can be an electronic device or a module (such as a chip) used in an electronic device.

[0124] like Figure 6 As shown, the image editing device 600 includes an acquisition module 601, a decoupling module 602 and a generation module 603. The image editing device 600 is used to implement the above Figure 3 Alternatively, the image editing apparatus 600 may include a device for implementing the above-mentioned Figure 3 Any function or operation module in the method embodiment shown in the figure may be implemented in whole or in part by software, hardware, firmware or any combination thereof.

[0125] When the image editing device 600 is used to implement Figure 3 The functions in the method embodiment shown are as follows: an acquisition module 601 is used to acquire a first image and a first text. The first text represents the editing requirements of the first image. A decoupling module 602 is used to decouple the editing requirements into at least two first basic editing tasks based on the first text. Each first basic editing task is used to implement an image processing stage in the process of editing the first image to meet the editing requirements. A generation module 603 is used to generate a first edited image based on the first image, the first text, and the at least two first basic editing tasks.

[0126] When editing a first image, the image editing device 600 of the embodiment of the present application decouples the editing requirements of the first image into at least two first basic editing tasks based on the first text, and then generates a first edited image based on the at least two first basic editing tasks, the first image, and the first text. Therefore, the image editing device of this solution generates multiple basic editing tasks by processing the editing requirements. These multiple basic editing tasks can be executed by a single image editing model, achieving the goal of using a single image editing model to meet multiple image editing requirements. There is no need to deploy different subtask models for different editing tasks, which is beneficial to improving image editing efficiency.

[0127] In one possible implementation, the generation module 603 is specifically configured to: generate a first noise based on the first text; generate at least one second noise based on the first image and at least two first basic editing tasks; input the first noise and the at least one second noise into a noise decision network for processing to obtain a third noise; and generate the first edited image based on the third noise.

[0128] In a possible implementation, the image editing device 600 further includes a determination module 604 .

[0129] The determining module 604 is configured to determine a first score of the first edited image based on the first edited image, the first image, and the first text. The first score represents the editing quality of the first edited image.

[0130] In one possible implementation, the decoupling module 602 is further configured to decouple the editing requirement into at least two second basic editing tasks based on the first text and the first score. Each second basic editing task is configured to implement an image processing stage in the process of editing the first image to meet the editing requirement.

[0131] The generating module 603 is further configured to generate a second edited image based on the first image, the first text and at least two second basic editing tasks, wherein the editing quality of the second edited image is higher than that of the first edited image.

[0132] In a possible implementation, the first basic editing task or the second basic editing task includes at least one of the following: generation, fusion, picking, elimination, portrait beautification, and portrait shaping.

[0133] This application also provides an image editing device, referring to Figure 7 , Figure 7 7 is a schematic diagram of the structure of an image editing device provided in an embodiment of the present application. The image editing device 700 includes a memory 701, a processor 702, a communication interface 704, and a bus 703. The memory 701, the processor 702, and the communication interface 704 are connected to each other via the bus 703.

[0134] Memory 701 can be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). Memory 701 can store programs. When the program stored in memory 701 is executed by processor 702, processor 702 and communication interface 704 are used to perform the various steps of the image editing method of the embodiment of the present application.

[0135] The processor 702 can be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the functions required to be performed by the units in the image editing device of the embodiment of the present application, or to execute the image editing method of the embodiment of the present application.

[0136] The processor 702 may also be an integrated circuit chip with signal processing capabilities. During implementation, the various steps of the image editing method of the embodiment of the present application may be completed by hardware integrated logic circuits or software instructions in the processor 702. The aforementioned processor 702 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the image editing method in conjunction with the embodiment of the present application may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor. The software module may be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or the like. The storage medium is located in the memory 701, and the processor 702 reads the information in the memory 701 and combines its hardware to complete the functions required to be performed by the units included in the image editing device of the embodiment of the present application, or executes the image editing method of the embodiment of the present application.

[0137] The communication interface 704 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the image editing device 700 and other devices or a communication network. For example, the first text and the first image can be obtained through the communication interface 704.

[0138] The bus 703 may include a path for transmitting information between various components of the image editing device 700 (eg, the memory 701 , the processor 702 , and the communication interface 704 ).

[0139] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0140] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0141] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0142] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk).

[0143] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. An image editing method, characterized in that: Acquire a first image and a first text, where the first text represents an editing requirement of the first image; Decoupling the editing requirement into at least two first basic editing tasks according to the first text, each of the first basic editing tasks being used to implement an image processing stage in a process of editing the first image to meet the editing requirement; generating a first edited image based on the first image, the first text, and the at least two first basic edit tasks; The generating a first edited image based on the first image, the first text and the at least two first basic editing tasks comprises: generating a first noise based on the first text; generating at least one second noise based on the first image and the at least two first basic editing tasks; the at least one second noise is obtained by processing the first image according to the at least two first basic editing tasks to obtain processing results of the first basic editing tasks, and then fusing the processing results; Inputting the first noise and the at least one second noise into a noise decision network for processing to obtain a third noise; The first edited image is generated based on the third noise.

2. The method according to claim 1, characterized in that The method further comprises: A first score of the first edited image is determined based on the first edited image, the first image, and the first text, the first score representing an edit quality of the first edited image.

3. The method according to claim 2, characterized in that The method further comprises: decoupling the editing requirement into at least two second basic editing tasks according to the first text and the first score, each of the second basic editing tasks being used to implement an image processing stage in a process of editing the first image to meet the editing requirement; A second edited image is generated based on the first image, the first text, and the at least two second basic editing tasks, wherein the editing quality of the second edited image is higher than the editing quality of the first edited image.

4. The method according to claim 3, characterized in that The first basic editing task or the second basic editing task includes at least one of the following: generation, fusion, picking, elimination, portrait beautification, and portrait shaping.

5. An image editing device, characterized in that: include: An acquisition module, used for acquiring a first image and a first text, wherein the first text represents an editing requirement of the first image; a decoupling module, configured to decouple the editing requirement into at least two first basic editing tasks according to the first text, each of the first basic editing tasks being configured to implement an image processing stage in a process of editing the first image to meet the editing requirement; A generating module, configured to generate a first editing image based on the first image, the first text and the at least two first basic editing tasks; The generation module is specifically used for: generating a first noise based on the first text; generating at least one second noise based on the first image and the at least two first basic editing tasks; the at least one second noise is obtained by processing the first image according to the at least two first basic editing tasks to obtain processing results of the first basic editing tasks, and then fusing the processing results; Inputting the first noise and the at least one second noise into a noise decision network for processing to obtain a third noise; The first edited image is generated based on the third noise.

6. The device according to claim 5, characterized in that The device also includes: A determination module is used to determine a first score of the first edited image based on the first edited image, the first image and the first text, wherein the first score represents an editing quality of the first edited image.

7. The device according to claim 6, characterized in that The decoupling module is further used to decouple the editing requirement into at least two second basic editing tasks according to the first text and the first score, each of the second basic editing tasks being used to implement an image processing stage in a process of editing the first image to meet the editing requirement; The generating module is further used to generate a second edited image based on the first image, the first text and the at least two second basic editing tasks, wherein the editing quality of the second edited image is higher than the editing quality of the first edited image.

8. The device according to claim 7, characterized in that The first basic editing task or the second basic editing task includes at least one of the following: generation, fusion, picking, elimination, portrait beautification, and portrait shaping.

9. An image editing device, characterized in that: It comprises a processor and a memory, wherein the processor and the memory are connected, wherein the memory is used to store program codes, and the processor is used to call the program codes to execute the image editing method as described in any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the image editing method according to any one of claims 1 to 4.

Images