Interaction method for image processing and image processing method

Abstract

The invention provides an interaction method for image processing and a corresponding image processing method. The method comprises the following steps: firstly, receiving a stroke which is added on a displayed first image by a user through an operable control and is used for representing a local area needing to be changed, and then displaying a prompt word which aims at the stroke and is used for representing a modification target of the local area. A second image generated according to the stroke, the prompt word and the first image is displayed, other parts, except for the stroke coverage range, of the second image are similar to the first image, and the content represented by the modification target is correspondingly generated at the position where the stroke is located in the second image. A user can draw the content which wants to be creatively modified on the first image by adding strokes on the image, and the device can recognize the strokes of the user and generate the content needed by the user. In this way, even if a user is not familiar with image processing software, efficient creative processing can be performed on the image through simple operation.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of image processing technology, and more particularly to an interactive method for image processing and an image processing method. Background Art

[0002] With the development of computer technology, users have more and more requirements for image processing. In the prior art, images are generally processed by image processing software.

[0003] Image processing software generally has functions such as adjusting the overall tone of an image (for example, changing the overall tone of an image by adding filters) and fine-tuning parts of an image (for example, modifying the face shape). If you want to use image processing software in the prior art to creatively process an image, it will require complex operations to achieve, which is inconvenient. For example, adding personalized creative content to the original image, removing unwanted parts of the original image, and changing the color of some objects in the original image will require complex operations to achieve. Summary of the invention

[0004] In view of this, one or more embodiments of the present specification provide an interactive method for image processing and an image processing method.

[0005] According to a first aspect of one or more embodiments of this specification, an interactive method for image processing is proposed, including:

[0006] Receiving a brush stroke added by a user on a displayed first image through an operable control; the brush stroke is used to represent a local area in the first image that needs to be modified;

[0007] Displaying a prompt word corresponding to the brush stroke, which indicates a modification target of the local area;

[0008] A second image is generated and displayed according to the prompt word, the brush stroke and the first image, wherein the second image presents the modification target in the local area and is the same as or similar to the first image in other areas.

[0009] According to a second aspect of one or more embodiments of this specification, an image processing method is proposed, including:

[0010] Acquire editing data input by a user for a first image; the editing data is used to represent a local area in the first image that needs to be modified;

[0011] Determining a prompt word corresponding to the editing data, which indicates a modification target of the local area;

[0012] A second image is generated according to the prompt word, the editing data and the first image, wherein the second image presents the modification target in the local area and is the same as or similar to the first image in the remaining areas.

[0013] According to a third aspect of an embodiment of this specification, there is provided an interactive device for image processing, including:

[0014] A stroke receiving module, used for receiving a stroke added by a user on the displayed first image through an operable control; the stroke is used for representing a local area in the first image that needs to be modified;

[0015] A prompt word display module, used to display the prompt word corresponding to the brush stroke, which indicates the modification target of the local area;

[0016] The image display module is used to generate and display a second image according to the prompt word, the brush stroke and the first image, wherein the second image presents the modification target in the local area and is the same as or similar to the first image in other areas.

[0017] According to a fourth aspect of the embodiments of this specification, there is provided an image processing device, including:

[0018] An editing data acquisition module, used to acquire editing data input by a user for a first image; the editing data is used to characterize a local area in the first image that needs to be modified;

[0019] A prompt word determination module, used to determine a prompt word corresponding to the editing data, which indicates a modification target of the local area;

[0020] An image generation module is used to generate a second image according to the prompt word, the editing data and the first image, wherein the second image presents the modification target in the local area and is the same as or similar to the first image in other areas.

[0021] According to a fifth aspect of the embodiments of this specification, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the above-mentioned interactive method for image processing or image processing method is implemented.

[0022] According to a sixth aspect of an embodiment of this specification, a computer device is provided, the computer device comprising:

[0023] processor;

[0024] a memory for storing processor-executable instructions;

[0025] The processor implements the above-mentioned interactive method for image processing or image processing method by running the executable instructions.

[0026] According to a seventh aspect of the embodiments of this specification, a computer program product is provided, which, when executed by a processor, implements the above-mentioned interactive method for image processing or image processing method.

[0027] This specification proposes an interactive method for image processing, and a corresponding image processing method. First, a brush stroke added by a user through an operable control on a displayed first image to represent a local area that needs to be changed is received, and then a prompt word for the brush stroke that represents a modification target for the local area is displayed. Then, a second image generated based on the brush stroke, the prompt word and the first image is displayed. The second image is similar to the first image except for the other parts covered by the brush stroke, and the position of the brush stroke in the second image corresponds to the content represented by the modification target. The user can draw the content that he wants to creatively modify on the first image by adding brush strokes on the image, and the device can recognize the user's brush strokes and generate the content required by the user. In this way, even if the user is not familiar with image processing software, he can perform efficient and creative processing on the image through simple operations.

[0028] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and, together with the description, serve to explain the principles of the specification.

[0030] Figure 1 The flowchart of an interactive method for image processing is shown in this specification according to an exemplary embodiment.

[0031] Figure 2 It is a schematic diagram of a first image with editing data and a corresponding second image according to an embodiment of the present specification.

[0032] Figure 3 It is a schematic diagram of another first image with editing data and a corresponding second image shown in this specification according to an embodiment.

[0033] Figure 4 It is a schematic diagram of a first interface shown in this specification.

[0034] Figure 5 is a flowchart of an image processing method according to an exemplary embodiment of the present specification.

[0035] Figure 6 It is a schematic diagram of an edge graph shown in this specification.

[0036] Figure 7 It is a schematic diagram of a mask shown in this specification.

[0037] Figure 8 It is a schematic diagram of an image processing flow shown in this specification.

[0038] Fig. 9 It is a block diagram of an interactive device for image processing according to an exemplary embodiment of the present specification.

[0039] Fig.10 is a block diagram of an image processing device according to an exemplary embodiment of the present specification.

[0040] Fig.11 It is a hardware structure diagram of a computer device shown in this specification according to an exemplary embodiment. DETAILED DESCRIPTION

[0041] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0042] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0043] In order to solve the problem that image processing in the prior art is inconvenient for users, this specification proposes an interactive method for image processing applied to the front end, and a corresponding back-end image processing method. First, a brush stroke added by the user through an operable control on the displayed first image to represent a local area that needs to be changed is received, and then a prompt word for the brush stroke to indicate the modification target of the local area is displayed. Then, a second image generated according to the brush stroke, the prompt word and the first image is displayed. The second image is similar to the first image except for the other parts covered by the brush stroke, and the content represented by the modification target is generated corresponding to the position of the brush stroke in the second image.

[0044] The user can add brush strokes to the image to draw the content he wants to modify creatively on the first image, and the device can recognize the user's brush strokes and generate the content the user needs. In this way, even if the user is not familiar with image processing software, he can perform efficient creative processing on the image through simple operations, allowing the user to create more personalized and creative works at a low cost.

[0045] Next, the method shown in this specification will be described through a specific embodiment. This specification shows an interactive method for image processing applied to the front end, and an image processing method applied to the back end. Next, an interactive method for image processing will be described through the interaction between the front end and the user. The two methods in this specification are mainly for creative processing of images. Creative processing may include changing local areas in the image, such as adding objects to the image; creative processing may also include deleting a local area in the image, such as deleting a person in the background of the image; in addition, creative processing may include changing the color of a part of the image, such as changing the color of an object in the image.

[0046] For the interactive method for image processing applied to the front end, it can be executed by a terminal that can interact with the user, such as a mobile phone, a computer or a tablet computer. For the image processing method applied to the back end, it can be jointly executed by the terminal and the server, or by the server alone, or by the terminal alone, and this specification does not limit this.

[0047] like Figure 1 As shown, this specification shows an interactive method for image processing, including:

[0048] Step 101: receiving a brush stroke added by a user on a displayed first image through an operable control.

[0049] The brush stroke is used to represent a local area in the first image that needs to be modified.

[0050] Specifically, the user can simply draw on the first image through the operable control to trace the local area to be changed. Accordingly, the above-mentioned brushstrokes can be any traces drawn by the user on the first image using the operable control, and the expression forms can include lines, area coating, etc.

[0051] The operable control may be a control that can add a brushstroke. The first image is an image to be processed, which may be an image uploaded by a user received before executing step 101, or an example image pre-stored in a database and selected by a user.

[0052] The first image can be displayed in a first display interface so that the user can process the first image. The first display interface can be an interface of image processing software, or an image processing page in the form of a web page. The first display interface can also be an interface in other forms. For example, the method can be implemented through a browser plug-in. For example, in any web page displayed by the browser, the image selected by the user can be used as the first image, the strokes added by the user in the first image can be received, and the corresponding second image can be generated, so that the user can achieve the image processing goal more flexibly.

[0053] The brush stroke may include at least one of a first brush stroke, a second brush stroke and a third brush stroke. The first brush stroke is used to outline the area; the second brush stroke is used to paint the area whose content needs to be changed; and the third brush stroke is used to paint the area whose color needs to be changed and present the color to be modified. The user can choose to use one or a combination of the above brush strokes to express his or her intention to modify the first image as needed.

[0054] In one embodiment, a combination of the first brush stroke and the second brush stroke may be used to indicate content modification of a local area. Specifically, when a user wants to modify the content of a local area in an image, the first brush stroke may be used to outline the local area to be modified or the outline of an object to be added, and the second brush stroke may be used to paint the area to be modified.

[0055] For example, Figure 2 As shown, Figure 2 The left figure in the middle shows the brush strokes added by the user in an image of a cake. The user wants to modify the whole cake shown in the first image into a cake with a corner cut off by adding the brush strokes. Figure 2 The white stroke in the figure is the first stroke, which outlines the content that the user wants to change, that is, the outline of the cut surface of the cake. Figure 2 The black stroke in the figure is the second stroke, which covers the area where the content needs to be changed. Figure 2 The strokes in the left picture can be obtained Figure 2 The second image is shown in the figure on the right.

[0056] For example, Figure 3 The figure on the left shows the user's added brush strokes in the image of a woman, and the user wants to modify the image of the woman into an image of a woman holding a cat. Figure 3 The white stroke is the first stroke, outlining the shape of the cat, and the black stroke is the second stroke, covering the area that needs to be changed into a cat. Figure 3 The left graph can be generated Figure 3 The picture on the right.

[0057] In another embodiment, only the second stroke may be used to indicate the deletion of a local area. Specifically, when the user wants to delete a local area in an image, unlike when the user wants to modify a local area in an image, there is no need to outline the area to be modified with the first stroke, and only the second stroke may be used to cover the area to be deleted, which makes it easier for the device to distinguish the user's intention. Figure 2 The black sector area without outline is the area covered only by the second stroke and is the area that the user wants to delete. Figure 2 The vase in the background of the left picture can be covered with the second stroke.

[0058] In another embodiment, a third stroke can be used to indicate a change in the color of a local area. If the user wants to change the color of a local area in the image, the third stroke can be used to cover the local area whose color is to be changed. The third stroke is different from the first or second stroke in that the first and second strokes cannot select a color, while the third stroke can select the color to be changed. For example, if the user wants to change Figure 2 The vase in the background of the left picture is changed to pink. You can first select the third stroke as pink, and then use the pink third stroke to cover the area where the vase is located. The third stroke can be used alone, that is, only use the third stroke of the target color to paint an area to indicate that the area is changed to the target color. Or, in some cases where detailed outlining is required, the third stroke can also be used in combination with the first stroke, for example, first use the first stroke to outline the outline, and then use the third stroke of the target color to paint the area within the outline.

[0059] The above-mentioned brush strokes can be used in more combinations as needed. For example, the user can use the first brush stroke to outline multiple sub-areas, paint some of the sub-areas with the second brush stroke, paint some of the sub-areas with the third brush stroke, and also use the second brush stroke to paint the sub-areas without the outline (formed by the first brush stroke).

[0060] In addition, when the brush stroke includes at least one of the first brush stroke, the second brush stroke and the third brush stroke, in addition to the first image, the first display interface can also display a control icon of an operable control. Specifically, the first display interface also displays the control icon of the operable control, including a first control icon corresponding to the first brush stroke, a second control icon corresponding to the second brush stroke, and a third control icon corresponding to the third brush stroke.

[0061] For example, Figure 4 As shown, Figure 4 is a possible schematic diagram of the first interface. Figure 4 410 is the first control icon, 420 is the second control icon, and 430 is the third control icon. Figure 4In addition to the control icons shown in the figure, the first interface may also include control icons for adjusting the thickness of the brush strokes, erasing brush strokes, etc. Figure 4 The examples in the description are not meant to limit the present description.

[0062] The above three brush strokes are just a possible specific implementation method and do not limit the present application.

[0063] Step 103: display the prompt word corresponding to the brush stroke.

[0064] The prompt word indicates the modification target of the local area.

[0065] The prompt word can be used to indicate the specific content that the user wants to modify in the local area (that is, the modification target). Figure 2 In the illustrated scenario, the prompt word may be a piece of cake. Figure 3 In the scene shown, the prompt word can be a cat. In the scene of changing the color of the local area, the prompt word can be the text or logo (such as a color number) of the color to be changed.

[0066] As for the specific method of obtaining the prompt word, it can be to obtain the prompt word input by the user.

[0067] In addition, in order to improve editing efficiency and save user time, so that the processed image can be obtained with fewer operations after the user enters the strokes, the prompt words can be a device that interacts with the user, or the backend server can automatically generate according to the strokes entered by the user and the first image. In this way, the device can recognize the user's strokes and guess what the user wants to draw, without the user having to manually fill in the prompt words, thereby improving editing efficiency. The specific method for generating the prompt words will be described in detail in the image processing method corresponding to the backend, and will not be repeated here.

[0068] In other words, in step 103, the predicted first prompt word is displayed. Correspondingly, the second image in the subsequent step can be generated based on the first prompt word. The first prompt word predicted by the device is displayed to the user, so that the user can confirm whether the generated prompt word is accurate. In addition, the user can also confirm whether the first prompt word is accurate. If the user confirms that the first prompt word is accurate, step 105 is executed to generate the second image based on the first prompt word.

[0069] In addition, in some cases, due to the limited drawing skills of the user, the device may not be able to generate accurate prompt words based on the user's brushstrokes. In order to ensure the accuracy of the generated result, a control for modifying the displayed first prompt word can be provided to the user, so that when the predicted first prompt word is inaccurate, it will not affect the generation effect of the second image.

[0070] In other words, step 103 may include: displaying the predicted first prompt word; and in response to the user's modification operation on the first prompt word, displaying the modified second prompt word. Correspondingly, the second image in the subsequent step may be generated according to the second prompt word.

[0071] In addition, the first interface mentioned above may also include a text box, which may be used to display a prompt word. In the case where the user can modify the displayed first prompt word, the text box may also be used to receive a second prompt word input by the user. Figure 4 As shown, Figure 4 The 440 in the text box is the text box.

[0072] Step 105 : generating and displaying a second image according to the prompt word, the brush stroke and the first image.

[0073] The second image presents the modification target in the local area, and is the same as or similar to the first image in other areas.

[0074] The second image is the image that the user wants to obtain by creatively processing the first image. During the processing, only the area covered by the brushstrokes is processed corresponding to the modification target, such as deleting an object, making creative modifications, changing the color of some areas, etc. The remaining areas are the same or similar to the first image. The same or similar here means that from the perspective of ordinary users' naked eye observation, the second image and the first image are basically the same in areas other than the local area. From the perspective of image processing, it can be considered that in the remaining areas, the similarity of the two images exceeds the predetermined threshold.

[0075] The second image can be displayed in the first interface as shown in FIG. Figure 4 As shown, it can be displayed on one side of the first image with the brushstrokes. The user can intuitively see the generation result of the second image. In addition, the user can also be provided with an operable control for downloading the second image.

[0076] The specific implementation of step 105 will be described in detail later and will not be repeated here.

[0077] After a brief description of the method applied to the front end, we will then Figure 5 The flowchart shown in the figure is used to illustrate the image processing method applied to the back end. Figure 5 As shown, the image processing method comprises the following steps:

[0078] Step 501: Obtain editing data input by a user for a first image.

[0079] The editing data is used to represent a local area in the first image that needs to be modified.

[0080] Step 501 corresponds to step 101, and the front end obtains the brush strokes input by the user, and can transmit them to the back end so that the back end obtains the editing data for the first image. The editing data refers to the brush strokes obtained in step 101. Through step 501, the editing data input by the user can be obtained, so that the processing of the first image can be further completed.

[0081] Step 503: Determine the prompt word corresponding to the edited data.

[0082] The prompt word indicates the modification target of the local area.

[0083] Step 503 corresponds to step 103 , where the backend determines the prompt word corresponding to the edited data and sends it to the frontend, and the frontend displays the determined prompt word.

[0084] The specific implementation of step 503 is similar to that of step 103, and will not be described here. Here, how to automatically generate prompt words according to the first image and the editing data will be described. Next, an embodiment will be used for description, and the following embodiment does not limit this specification.

[0085] In an optional implementation, step 503 may be implemented by inputting the first image and the editing data into a multimodal large model to obtain a predicted prompt word for the user to confirm or modify. The multimodal large model may be pre-trained to generate a corresponding multimodal large model of prompt words for representing the modification target based on the input image and editing data.

[0086] Step 505: Generate a second image based on the prompt word, the editing data and the first image, wherein the second image presents the modification target in the local area and is the same as or similar to the first image in other areas.

[0087] Step 505 corresponds to step 101. After the backend generates a second image that meets the requirements according to the acquired prompt words, editing data and the first image, the second image can be transmitted to the frontend, and the frontend displays the second image.

[0088] Next, a specific embodiment will be used to describe how the backend specifically performs step 505. The following embodiment does not limit this specification, and step 505 may also be directly implemented by an end-to-end model, inputting the first image, prompt words and editing data, and outputting the second image.

[0089] In this embodiment, step 505 specifically includes:

[0090] Step 5051: Generate an edge map according to the first image and the edited data.

[0091] The edge map can be used to obtain the shape of the local range that needs to be changed, as well as the relationship between the local range and other surrounding lines in the first image, so that the shape of the local range that needs to be changed can be identified later, and a more realistic image can be generated based on the relationship between the local range and other surrounding lines.

[0092] The edge map refers to the edge information extracted from the image, which is the area where the grayscale changes sharply in the image. The outline of the object can be identified through the edge map. Figure 3 In the scene shown, the corresponding edge map is as follows Figure 6 shown.

[0093] The specific method for extracting the edge map may be: inputting the first image and the edited data into a pre-trained edge extraction model to obtain the edge map, wherein the edge extraction model uses a convolutional neural network (CNN) as a model backbone, which can achieve a better edge map extraction effect.

[0094] In addition, since the strokes in the edited data are generally simple, the contour curve can be directly extracted, for example, the first stroke mentioned above can be used as the contour line in the edge map. In order to facilitate the extraction, the edge map of the first image can be extracted first, and then the edited data can be overlaid on the corresponding position of the edge map corresponding to the first image to obtain the edge map.

[0095] When overwriting with editing data, if there is at least one of the first stroke, the second stroke, and the third stroke mentioned above, for the first stroke, since it outlines the contour of the object to be changed, it can be used as the contour in the edge map. For the second stroke, the content covered by the second stroke is the content that needs to be changed or deleted, and this part of the content needs to be regenerated based on the following steps, and the range covered by the second stroke can be reset to black in the edge map. For the third stroke, the content covered by the third stroke is the local range that needs to change the color, and the edge of the range covered by the third stroke can be added as a contour to the edge map corresponding to the first image.

[0096] The contour lines corresponding to the brush strokes in the edge map and the contour lines corresponding to other parts can be represented by different lines, such as lines of different thicknesses, so that the image generation model in the following text can easily identify which part needs to be changed.

[0097] In addition, in other embodiments, the contour lines corresponding to the brush strokes in the edge map and the contour lines corresponding to other parts can be represented by the same lines. The form of the contour lines will be specifically described below in conjunction with the specific implementation method of step 5052.

[0098] Step 5052: Encode the first image to obtain encoding information of the first image.

[0099] By encoding the first image in step 5052, the model can obtain information of the original image, ensuring that the second image is similar to the first image except for the area covered by the edited data.

[0100] In addition, by obtaining the encoding information in step 5052, the image generation model can also obtain the style of the original image, ensuring that the content generated at the corresponding position of the edited data in the second image is consistent with the overall style. Figure 2 The cake in the image shown is in a realistic style, and the corresponding cake in the second image is also in a realistic style.

[0101] In an optional implementation, since the image at the position corresponding to the edited data needs to be regenerated, it is not necessary to ensure that the portion is similar to the corresponding portion in the first image, and therefore the portion may not be encoded. In other words, step 5052 includes: extracting the remaining area except the local area from the first image; encoding the remaining area using an encoder to obtain encoding information of the first image. This allows for more efficient encoding.

[0102] Among them, the above encoder can be a variational auto-encoder (VAE).

[0103] In a specific implementation, a mask may be generated based on the edited data, and the mask distinguishes the local range covered by the edited data from other ranges. For example, in the mask, the local range covered by the edited data may be white, and other ranges may be black. Figure 3 The scene shown can generate Figure 7 The mask shown. The mask and the first image can then be input into the encoder, and the encoder only encodes the black part of the mask to obtain the encoded information.

[0104] In the above case, in the edge map obtained in step 5051, the contour line corresponding to the edited data and the contour line corresponding to the first image may be the same. Since the coding information only includes other ranges except the edited data, it is possible to determine which part of the edge map needs to be newly generated and which part needs to be maintained through the coding information and the edge map.

[0105] Step 5053: input the edge map, the encoding information and the prompt word into an image generation model, and obtain a second image according to the output image.

[0106] After obtaining the edge map, the coding information and the prompt word, the above content can be input into the image generation model so that the image generation model can generate a second image.

[0107] In some cases, some image generation models cannot receive data in multiple forms, such as stable diffusion, which can only receive prompt words in text form. In order to enable the image generation model to process data in multiple forms normally, step 5053 can perform the following operations: input the edge map into the first model to obtain first data for image encoding of the edge map; input the first data and the encoding information into the second model to obtain second data converted into a text embedding space; input the second data and the prompt words into the image generation model.

[0108] Specifically, through the first model, the edge map can be encoded to obtain the first data. Through the second model, the information of the edge map corresponding to the first data and the encoding information of the first image can be summarized, and the summarized information can be converted into a text embedding space to obtain the second data. As a result, some image generation models that can only process prompt words can generate the required model according to the second data. Among them, the second model can summarize the edge map and the encoding information. Correspondingly, through the second data, the image generation model can obtain: the shape of the local range that needs to be changed, the relationship between the local range and other surrounding objects, the style of the local range that needs to be changed, and the information of the original first image.

[0109] In the above method, the entire image is regenerated instead of just redrawing the local area covered by the edited data, so that the content generated in the local area covered by the edited data and the surrounding content are more logical. Figure 3 In the scene shown, a kitten sitting on the woman's arm can be generated based on the relationship between the local range and the surrounding contour lines in the edge map. If only the local range is redrawn, a kitten may be generated hanging on the woman's arm, resulting in the generated cat's feet not touching the woman, making the generated image illogical and unrealistic.

[0110] In addition, after the output is obtained according to the image generation model, the image generated by the image generation model may have a lower definition than the first image. In order to ensure that the part that the user does not want to modify is closer to the original image and improve the user experience, the second image obtained according to the output image may include: extracting a first sub-image in the local area from the output image; fusing the first sub-image with a second sub-image in the first image excluding the local area to obtain the second image.

[0111] In specific implementation, the mask shown above can be used for fusion, and the mask can be used to distinguish the local range corresponding to the edited data from other ranges. In this way, the image generated by the image generation model is used as the standard in the local range corresponding to the edited data, and the first image is used as the standard in other ranges. In this way, the clarity of the fused second image is almost the same as that of the first image, which improves the user experience.

[0112] by Figure 3 The scenario shown is an example. The specific implementation process of step 505 shown above can be as follows: Figure 8 shown. Figure 3 The solid-line boxes represent the models used in each processing step, and the dotted-line boxes represent the data generated in the processing steps.

[0113] First, the first image with the edited data is input into the edge extraction model to obtain an edge map, and the edge map is input into the first model. Through the edge map, the image generation model can be controlled to generate the content that the user wants to generate.

[0114] Secondly, the first image with the editing data and the mask automatically generated around the user's brushstrokes are input into the encoder to obtain the encoding information (vector) of the first image. The encoding information is input into the second model, and the output data of the first model is also input into the second model. Since some image generation models use noise to generate images, in order to make the output data of the second model match the input of the image generation model, noise can also be input into the second model here. The encoding information can ensure that the content generated corresponding to the editing data is the same as the style of the first image, and that the position other than the editing data is the same as the first image.

[0115] Next, the first image with the edited data is input into the multimodal large model to predict the content that the user wants to generate, and the predicted prompt words and noise are input into the image generation model to obtain the second image. If the image generation model does not need to use noise for image generation, the noise can also be omitted here.

[0116] Finally, the image generated by the image generation model is fused with the first image according to the mask. During the fusion, the local range covered by the edited data is based on the generated image, and the remaining range is based on the first image. In this way, when the clarity of the generated image is lower than that of the first image, the clarity of the second image can still be guaranteed.

[0117] Corresponding to the embodiments of the aforementioned method, this specification also provides embodiments of a device and a terminal to which it is applied.

[0118] like Fig. 9 As shown, Fig. 9is a block diagram of an interactive device for image processing according to an exemplary embodiment of the present specification, the device comprising:

[0119] The stroke receiving module 910 is used to receive the strokes added by the user on the displayed first image through the operable control; the strokes are used to represent the local area in the first image that needs to be modified.

[0120] The prompt word display module 920 is used to display the prompt word corresponding to the stroke, which indicates the modification target of the local area.

[0121] The image display module 930 is used to generate and display a second image according to the prompt word, the brush stroke and the first image, wherein the second image presents the modification target in the local area and is the same as or similar to the first image in other areas.

[0122] In an optional implementation manner, the prompt word display module 920 is specifically configured to: display a predicted first prompt word; and generate the second image according to the first prompt word.

[0123] In an optional implementation, the prompt word display module 920 is specifically used to: display a predicted first prompt word; in response to a user's modification operation on the first prompt word, display a modified second prompt word; and the second image is generated according to the second prompt word.

[0124] In an optional embodiment, the brush stroke includes at least one of a first brush stroke, a second brush stroke and a third brush stroke, wherein the first brush stroke is used to outline the area; the second brush stroke is used to paint the area whose content needs to be changed; and the third brush stroke is used to paint the area whose color needs to be changed and present the color that needs to be modified.

[0125] In an optional embodiment, the first image is displayed in a first display interface, and the first display interface also displays control icons of the operable controls, including a first control icon corresponding to the first stroke, a second control icon corresponding to the second stroke, and a third control icon corresponding to the third stroke.

[0126] In an optional implementation, the first display interface further includes a text box, and the text box is used to display the prompt word.

[0127] like Fig.10 As shown, Fig.10 is a block diagram of an image processing device according to an exemplary embodiment of the present specification, including:

[0128] The editing data acquisition module 110 is used to acquire editing data input by a user for a first image; the editing data is used to represent a local area in the first image that needs to be modified.

[0129] The prompt word determination module 120 is used to determine the prompt word corresponding to the editing data, which represents the modification target of the local area.

[0130] The image generation module 130 is used to generate a second image according to the prompt word, the editing data and the first image, wherein the second image presents the modification target in the local area and is the same as or similar to the first image in other areas.

[0131] In an optional implementation, the prompt word determination module 120 is specifically used to: input the first image and the editing data into a multimodal large model to obtain a predicted prompt word for the user to confirm or modify.

[0132] In an optional implementation, the image generation module 130 includes:

[0133] The edge map generating submodule 1301 (not shown in the figure) is used to generate an edge map according to the first image and the editing data.

[0134] A coding information acquisition submodule 1302 (not shown in the figure), configured to encode the first image and acquire coding information of the first image;

[0135] The second image generation submodule 1303 (not shown in the figure) is used to input the edge map, the encoding information and the prompt word into an image generation model, and obtain a second image according to the output image.

[0136] In an optional implementation, the edge map generation submodule 1301 is specifically used to: input the first image and the edited data into a pre-trained edge extraction model to obtain an edge map, wherein the edge extraction model uses a convolutional neural network CNN as a model backbone.

[0137] In an optional implementation manner, the coding information acquisition submodule 1302 is specifically used to: extract a remaining area except the local area from the first image; and encode the remaining area using an encoder to obtain coding information of the first image.

[0138] In an optional implementation, the second image generation submodule 1303 is specifically used to: input the edge map into the first model to obtain first data for image encoding the edge map; input the first data and the encoding information into the second model to obtain second data converted to the text embedding space; and input the second data and the prompt word into the image generation model.

[0139] In an optional implementation, the second image generation submodule 1303 is specifically used to: extract a first sub-image in the local area from the output image; and fuse the first sub-image with a second sub-image in the first image excluding the local area to obtain a second image.

[0140] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0141] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0142] like Fig.11 As shown, Fig.11 A hardware structure diagram of a computer device is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0143] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification. The processor implements the above method by running executable instructions.

[0144] The memory 1020 for storing processor executable instructions can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020.

[0145] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0146] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0147] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0148] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0149] The embodiments of the present specification also provide a computer program product, which, when executed by a processor, implements the above-mentioned interactive method for image processing or image processing method.

[0150] The embodiments of the present specification also provide a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned interactive method for image processing or image processing method is implemented.

[0151] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0152] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0153] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0154] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

Claims

1. An interactive method for image processing, comprising: receiving a brush stroke added by a user on the displayed first image through an operable control; The brush stroke is used to represent a local area in the first image that needs to be modified; Displaying a prompt word corresponding to the brush stroke, which indicates a modification target of the local area; A second image is generated and displayed according to the prompt word, the brush stroke and the first image, wherein the second image presents the modification target in the local area and is the same as or similar to the first image in other areas.

2. The method according to claim 1, wherein: Displaying the prompt words corresponding to the brush strokes, including: Display the predicted first cue word; The second image is generated according to the first prompt word.

3. The method according to claim 1, wherein: Displaying the prompt words corresponding to the brush strokes, including: Display the predicted first cue word; In response to the user's modification operation on the first prompt word, displaying the modified second prompt word; The second image is generated according to the second prompt word.

4. The method according to claim 1, wherein: The brush stroke includes at least one of a first brush stroke, a second brush stroke and a third brush stroke, wherein the first brush stroke is used to outline the area; the second brush stroke is used to paint the area whose content needs to be changed; and the third brush stroke is used to paint the area whose color needs to be changed and present the color that needs to be modified.

5. The method according to claim 4, wherein: The first image is displayed in a first display interface, which also displays control icons of the operable controls, including a first control icon corresponding to the first stroke, a second control icon corresponding to the second stroke, and a third control icon corresponding to the third stroke. 6 . According to the method of claim 5 , the first display interface further comprises a text box, and the text box is used to display the prompt word.

7. An image processing method, comprising: Acquire editing data input by a user for the first image; The editing data is used to represent the local area in the first image that needs to be modified; Determining a prompt word corresponding to the editing data, which indicates a modification target of the local area; A second image is generated according to the prompt word, the editing data and the first image, wherein the second image presents the modification target in the local area and is the same as or similar to the first image in other areas.

8. The method according to claim 7, wherein: Determining the prompt word corresponding to the edited data includes: The first image and the editing data are input into the multimodal macro model to obtain predicted prompt words for the user to confirm or modify.

9. The method according to claim 7, wherein: Generating a second image according to the prompt word, the editing data and the first image, comprising: generating an edge map based on the first image and the edited data; Encode the first image to obtain encoding information of the first image; The edge map, the encoding information and the prompt word are input into an image generation model, and a second image is obtained according to the output image.

10. The method according to claim 9, wherein: The step of generating an edge map according to the first image and the edited data comprises: The first image and the edited data are input into a pre-trained edge extraction model to obtain an edge map, wherein the edge extraction model uses a convolutional neural network (CNN) as a model backbone.

11. The method according to claim 9, wherein: The step of encoding the first image to obtain encoding information of the first image includes: extracting a remaining area except the local area from the first image; The remaining area is encoded using an encoder to obtain encoding information of the first image.

12. The method according to claim 9, wherein: The step of inputting the edge map, the encoding information and the prompt word into an image generation model comprises: Inputting the edge map into a first model to obtain first data for performing image encoding on the edge map; Inputting the first data and the encoding information into a second model to obtain second data converted into a text embedding space; The second data and the prompt word are input into an image generation model.

13. The method according to claim 9, wherein: Obtaining a second image according to the output image includes: extracting a first sub-image in the local area from the output image; The first sub-image is fused with a second sub-image in the first image excluding the local area to obtain a second image.

14. An interactive device for image processing, comprising: A stroke receiving module, used for receiving a stroke added by a user on the displayed first image through an operable control; The brush stroke is used to represent a local area in the first image that needs to be modified; A prompt word display module, used to display a prompt word corresponding to the brush stroke, which indicates a modification target of the local area; The image display module is used to generate and display a second image according to the prompt word, the brush stroke and the first image, wherein the second image presents the modification target in the local area and is the same as or similar to the first image in other areas.

15. An image processing device, comprising: An editing data acquisition module, used to acquire editing data input by a user for the first image; The editing data is used to represent the local area in the first image that needs to be modified; A prompt word determination module, used to determine a prompt word corresponding to the editing data, which indicates a modification target of the local area; An image generation module is used to generate a second image according to the prompt word, the editing data and the first image, wherein the second image presents the modification target in the local area and is the same as or similar to the first image in other areas.

16. A computer program product, wherein when the computer program product is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

17. A computer device comprising: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 13 by running the executable instructions.

Images