Image processing method and electronic equipment

Through the combination of graphic and text correlation processing and image generation model, the problem of accurate processing of large language models in image processing is solved, the accuracy and user experience of image processing are improved, and the precise and controllable processing of images is achieved.

CN119027949APending Publication Date: 2024-11-26LENOVO (BEIJING) LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410937034.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-12
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

Large language models are difficult to achieve accurate processing of images in multiple rounds of dialogues in image processing, resulting in low accuracy of output results and affecting user experience.

Method used

By obtaining the input text, determining its corresponding pending image, performing graphic and text correlation processing, generating a target image, and using a multimodal large language model and image generation model to process the target entity as a target result, and generating a target image.

Benefits of technology

Accurate and controllable processing of images is achieved, the accuracy and user experience of image processing are improved, and the accuracy of target images is ensured.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027949A_ABST
    Figure CN119027949A_ABST
Patent Text Reader

Abstract

The invention provides an image processing method and electronic equipment, and relates to the technical field of computers. The image processing method comprises the steps that an input text is acquired, a to-be-processed image corresponding to the input text is determined, and the input text is used for indicating that a target entity in the to-be-processed image is processed into a target result; performing image-text association processing on the input text and the to-be-processed image to obtain an association relationship between the input text and the to-be-processed image; and generating a target image based on the association relationship, wherein the target image is an image after the target entity in the to-be-processed image is processed into a target result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more particularly to an image processing method and an electronic device. Background Art

[0002] Large Language Models (LLMs) can generate human-like dialogues and answer complex questions. However, most LLMs struggle to accurately process images based on user input in multi-turn dialogues involving image processing, resulting in low-precision image processing outputs that negatively impact user experience. Therefore, ensuring that LLMs achieve accurate and controllable image processing in multi-turn dialogues involving image processing is a pressing issue that needs to be addressed. Summary of the Invention

[0003] This disclosure provides an image processing method and an electronic device to at least solve the above-mentioned technical problems existing in the prior art.

[0004] According to a first aspect of this disclosure, an image processing method is provided, comprising: acquiring input text; determining an image to be processed corresponding to the input text, wherein the input text is used to instruct a target entity in the image to be processed into a target result; performing image-text association processing on the input text and the image to be processed to obtain an association relationship between the input text and the image to be processed; and generating a target image based on the association relationship, wherein the target image is an image after the target entity in the image to be processed into the target result.

[0005] In one possible implementation, the step of performing image-text association processing on the input text and the image to be processed to obtain the association relationship between the input text and the image to be processed includes: inputting the input text and the image to be processed into a multimodal large language model for image-text association processing, and the multimodal large language model outputting the association relationship between the input text and the image to be processed.

[0006] In one embodiment, the step of performing image-text association processing on the input text and the image to be processed to obtain the association relationship between the input text and the image to be processed further includes: performing text recognition on the input text to determine the text corresponding to the target entity and the text corresponding to the target result; performing image recognition on the image to be processed to obtain multiple entities in the image to be processed and the position information of the entities in the image to be processed; and associating the text corresponding to the target entity, the position information of the target entity in the image to be processed, and the text corresponding to the target result to obtain the association relationship.

[0007] In one possible implementation, generating the target image based on the association relationship includes: inputting the association relationship and the image to be processed into an image generation model, wherein the image generation model processes the target entity in the image to be processed into the target result, thereby obtaining the target image.

[0008] In one embodiment, generating the target image based on the association relationship further includes: determining a first sub-image corresponding to the target entity in the image to be processed and mask information of the first sub-image based on the position information of the target entity in the image to be processed in the association relationship; generating a second sub-image that processes the target entity into the target result based on the mask information; and fusing the second sub-image with the image to be processed to obtain the target image.

[0009] In one possible implementation, generating the target image based on the association relationship further includes: determining the location information of the target entity in the image to be processed and the text corresponding to the target result based on the association relationship; generating the target image based on the location information, the text corresponding to the target result, and the image to be processed, wherein the image features in the target image other than the location information are the same as those in the image to be processed, and the image features at the location information are the image features corresponding to the target result.

[0010] In one possible implementation, generating the target image based on the association relationship further includes: semantically expanding the text corresponding to the target result in the association relationship to obtain multiple sub-results, wherein the similarity between the text corresponding to the sub-results and the text corresponding to the target result satisfies a target threshold; generating multiple sub-images based on the association relationship, wherein the sub-images are images after processing the target entities in the image to be processed into the sub-results; and, in response to a selection operation for the multiple sub-images, determining the sub-image corresponding to the selection operation as the target image.

[0011] In one embodiment, an image processing method further includes: receiving feedback information for the target image; updating the input text based on the feedback information; and adjusting the target image based on the updated input text.

[0012] In one possible implementation, determining the image to be processed corresponding to the input text includes: determining the image to be processed corresponding to the input text among all images in the current dialogue based on the text in the input text used to indicate the image; or, determining the image to be processed corresponding to the input text based on at least one of a click operation, a selection operation, and a drag operation performed on the image.

[0013] According to a second aspect of this disclosure, an electronic device is provided, comprising:

[0014] At least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the methods described in this disclosure.

[0017] According to a third aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing the computer to perform the methods described in this disclosure.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0019] The above and other objects, features, and advantages of this disclosure will become readily apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. Several embodiments of this disclosure are illustrated in the drawings by way of example and not limitation, in which:

[0020] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

[0021] Figure 1 A flowchart illustrating an image processing method according to an embodiment of this disclosure is shown. Figure 1 ;

[0022] Figure 2 This illustration shows a scenario illustrating an image processing method according to an embodiment of the present disclosure. Figure 1 ;

[0023] Figure 3 A flowchart illustrating an image processing method according to an embodiment of this disclosure is shown. Figure 2 ;

[0024] Figure 4 A flowchart illustrating an image processing method according to an embodiment of this disclosure is shown. Figure 3 ;

[0025] Figure 5 A flowchart illustrating an image processing method according to an embodiment of this disclosure is shown. Figure 4 ;

[0026] Figure 6 A flowchart illustrating an image processing method according to an embodiment of this disclosure is shown. Figure 5 ;

[0027] Figure 7 This illustration shows a scenario illustrating an image processing method according to an embodiment of the present disclosure. Figure 2 ;

[0028] Figure 8 This illustration shows a scenario illustrating an image processing method according to an embodiment of the present disclosure. Figure 3 ;

[0029] Figure 9 A flowchart illustrating an image processing method according to an embodiment of this disclosure is shown. Figure 6 ;

[0030] Figure 10 This illustration shows a scenario illustrating an image processing method according to an embodiment of the present disclosure. Figure 4 ;

[0031] Figure 11 This illustration shows a scenario illustrating an image processing method according to an embodiment of the present disclosure. Figure 5 ;

[0032] Figure 12 A schematic diagram of the composition structure of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0033] To make the objectives, features, and advantages of this disclosure more apparent and understandable, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0034] Figure 1 A flowchart illustrating an image processing method according to an embodiment of this disclosure is shown. Figure 1 ,like Figure 1 As shown, an image processing method includes:

[0035] Step S101: Obtain the input text and determine the image to be processed corresponding to the input text.

[0036] In this embodiment, the input text is used to instruct the target entity in the image to be processed into the target result. After obtaining the input text, the image to be processed indicated by the input text can be determined. In one example, the input text could be "Change the sofa on the left in this image to red," where "this image" indicates the image to be processed, "the sofa on the left" indicates the target entity, and "red" indicates the target result. Based on "this image" in the input text, it can be determined that the image to be processed is the most recently generated image or the most recently input image in the current dialogue.

[0037] Step S102: Perform image-text association processing on the input text and the image to be processed to obtain the association relationship between the input text and the image to be processed.

[0038] In this embodiment, after determining the input text and the corresponding image to be processed, it is necessary to perform image-text association processing on the input text and the image to be processed to obtain the association relationship between the input text and the image to be processed. The association relationship represents the correspondence between the word segmentation or phrase in the input text and the position information of the entity object in the image to be processed. The association relationship can be represented in the form of a triple such as [target entity, position information of the target entity in the image to be processed, target result]. For example, if the input text is "Change the sofa on the left in this image to red", the "sofa on the left", the position information of the "sofa on the left" in the image to be processed, and the "red" in the input text can be associated to obtain an association relationship such as [sofa on the left, position information, red].

[0039] Step S103: Generate the target image based on the association relationship.

[0040] In this embodiment, the association relationship reflects the location information of the target entity in the image to be processed and the target result of processing the target entity. The target image is generated by processing the image features at the location information of the target entity in the image to be processed into the image features corresponding to the target result. The target image is the image after processing the target entity in the image to be processed into the target result. For example, if the input text is "Change the sofa on the left in this image to red", the RGB (Red, Green, Blue) values ​​of the pixels at the location information of the "left sofa" in the image to be processed can be changed to red, that is, the R value is set to the maximum, and the G and B values ​​are set to 0.

[0041] In this disclosure, image-text association processing is performed on the input text and the image to be processed to obtain the association relationship between the input text and the image to be processed. Finally, based on the association relationship, a target image is generated after processing the target entities in the image to be processed into the target result. Since the association relationship represents the correspondence between the word segments or phrases in the input text and the positional information of the entity objects in the image to be processed, the positional information of the target object indicated in the input text in the image to be processed, as well as the processing result for the target object, can be accurately obtained based on the association relationship. This allows for precise and controllable processing of the target objects in the image to be processed, improving the accuracy of the target image and the user experience.

[0042] In another embodiment, the "determine the image to be processed corresponding to the input text" in step S101 includes: determining the image to be processed corresponding to the input text among all images in the current dialogue based on the text in the input text used to indicate the image; or, determining the image to be processed corresponding to the input text based on at least one of the click operation, box selection operation, and drag operation performed on the image.

[0043] In this embodiment, based on the text in the input text used to indicate the image, the image to be processed corresponding to the input text can be determined from all images in the current dialogue. The text used to indicate the image can be something like "this image" or "previous image." In one example, the LLM determines the image to be processed corresponding to the input text. To enable the LLM to determine the image to be processed based on the text in the input text used to indicate the image, a training dataset for the LLM is first obtained. The training dataset includes sample images to be processed, sample input text, and sample target images. Then, an image proxy token is added to each sample image to be processed. The image proxy uniquely identifies the sample image to be processed, and the same image proxy is added to the corresponding sample input text. When training the LLM based on the training dataset, the image proxy can be used to establish a reference between the sample input text and the sample image to be processed, enabling the LLM to learn the ability to determine the image to be processed based on the text in the input text used to indicate the image, thereby improving the accuracy of determining the image to be processed.

[0044] In this embodiment, the image to be processed corresponding to the input text can also be determined based on at least one of the click operation, selection operation, and drag operation performed on the image. In one example, if the user inputs text into the LLM and then clicks on image A on the currently displayed interface, then image A can be determined as the image to be processed. Figure 2 This illustration shows a scenario illustrating an image processing method according to an embodiment of the present disclosure. Figure 1 ,like Figure 2As shown, the left side of the LLM application interface is the image display area, used to display all images involved in the current dialogue. These can be images generated in the current dialogue or images entered by the user, including images A, B, C, and D. Users can click or select the images they want to process in the image display area, or directly drag the images they want to process from the image display area to the dialogue area. The left side of the LLM application interface is the dialogue area, used for human-computer dialogue and displaying dialogue information. In one example, the user can first click image A and then enter "Change the sofa on the left in this image to red," or first enter "Change the sofa on the left in this image to red" and then click image A, thus confirming that "this image" refers to image A. After confirming that the image to be processed is image A, it can be processed as follows: Figure 2 The dialog box will automatically display image A, or you can choose not to display image A in the dialog box.

[0045] In another embodiment, step S102, "performing image-text association processing on the input text and the image to be processed to obtain the association relationship between the input text and the image to be processed," includes: inputting the input text and the image to be processed into a multimodal large language model for image-text association processing, and the multimodal large language model outputting the association relationship between the input text and the image to be processed.

[0046] In this embodiment, the Multimodal Large Language Model (MLLM) can understand and generate data of different modalities, such as text, images, and audio. After inputting the input text and the image to be processed into the MLLM, the MLLM can directly capture the contextual relationship between the text and the image, thereby establishing the association between the input text and the image to be processed. There is no need to understand the input text and the image to be processed separately and then integrate them. This not only results in a more accurate association, but also improves the efficiency of text-image association processing.

[0047] In another embodiment, step S103, "generating a target image based on the association relationship", includes: inputting the association relationship and the image to be processed into an image generation model, and the image generation model processing the target entity in the image to be processed into a target result to obtain the target image.

[0048] In this embodiment, the image generation model can be a Transformer architecture model or a variant of a Generative Adversarial Network (GAN). The image generation model can be combined with an LLM and generate a matching target image by parsing the output of the LLM. Therefore, the association relationship and the image to be processed can be input into the image generation model. The image generation model directly processes the target entity in the image to be processed into the target result according to the association relationship, and obtains the target image.

[0049] Figure 3 A flowchart illustrating an image processing method according to an embodiment of this disclosure is shown. Figure 2 ,like Figure 3 As shown, an image processing method includes:

[0050] Step S201: Obtain the input text and determine the image to be processed corresponding to the input text.

[0051] The specific implementation details of step S201 are similar to those of step S101, and will not be repeated here.

[0052] Step S202: Perform text recognition on the input text to determine the text corresponding to the target entity and the text corresponding to the target result.

[0053] In this embodiment, the input text can be parsed based on a Natural Language Processing (NLP) model to extract its semantic information and identify the text indicating the target entity and the text indicating the target result. For example, if the input text is "Change the sofa on the left in this image to red", it is necessary to identify the text "the sofa on the left" indicating the target entity and the text "red" indicating the target result.

[0054] Step S203: Perform image recognition on the image to be processed to obtain multiple entities in the image to be processed and their location information in the image to be processed.

[0055] In this embodiment, object detection models such as YOLO (You Only Look Once) and Faster R-CNN (FastRegion-based Convolutional Neural Network) can be used to identify all entities in the image to be processed and their location information within the image, and to add a category label to each entity. For example, if the image to be processed includes two sofas, one on the left and one on the right, and a table located between the two sofas, it is necessary to identify the location information of the two sofas and the table in the image to be processed, and to add the category label "sofa" to the sofas and the category label "table" to the table.

[0056] Step S204: Associate the text corresponding to the target entity, the location information of the target entity in the image to be processed, and the text corresponding to the target result to obtain the association relationship.

[0057] In this embodiment, it is necessary to integrate the text recognition results and the image recognition results. Based on the text indicating the target entity in the text recognition results and the category label in the image recognition results, the location information of the target entity in the image to be processed is determined. Then, the text indicating the target entity, the location information of the target entity in the image to be processed, and the text indicating the target result are associated to obtain the association relationship. For example, based on the text indicating the target entity "the sofa on the left", the entity with the category label "sofa" can be first determined. Then, based on the location information of the entity with the category label "sofa" in the image to be processed, the location information of "the sofa on the left" in the image to be processed is determined. Finally, the text indicating the target entity "the sofa on the left", the location information of "the sofa on the left" in the image to be processed, and the text indicating the target result "red" are associated.

[0058] Step S205: Generate the target image based on the association relationship.

[0059] The specific implementation details of step S205 are similar to those of step S103, and will not be repeated here.

[0060] In this disclosure, text recognition can be performed on the input text separately to obtain the text recognition result, and image recognition can be performed on the image to be processed separately to obtain the image recognition result. Then, the text recognition result and the image recognition result are integrated to obtain the association relationship. This single-modal image-text association method can be optimized for specific modal data, thereby improving the accuracy of the association relationship to a certain extent. Moreover, the processing of specific modal data can be adjusted without affecting the processing of other modal data, which is highly flexible.

[0061] Figure 4 A flowchart illustrating an image processing method according to an embodiment of this disclosure is shown. Figure 3 ,like Figure 4 As shown, an image processing method includes:

[0062] Step S301: Obtain the input text and determine the image to be processed corresponding to the input text.

[0063] Step S302: Perform image-text association processing on the input text and the image to be processed to obtain the association relationship between the input text and the image to be processed.

[0064] The specific implementation details of steps S301-S302 are similar to those of steps S101-S102, and will not be repeated here.

[0065] Step S303: Based on the position information of the target entity in the image to be processed in the association relationship, determine the first sub-image and the mask information of the first sub-image corresponding to the target entity in the image to be processed.

[0066] In this embodiment, based on the position information of the target entity in the image to be processed in the association relationship, an image segmentation network can be used to segment the first sub-image corresponding to the target entity from the image to be processed, and determine the mask information of the first sub-image. In one example, the position information of the target entity "left sofa" in the image to be processed is the bounding box of "left sofa". The first sub-image corresponding to "left sofa" is segmented based on the bounding box. The mask information of the first sub-image can indicate the pixels in the first sub-image that belong to "left sofa" and the pixels that do not belong to "left sofa".

[0067] Step S304: Based on the mask information, generate a second sub-image that processes the target entity into the target result.

[0068] In this embodiment, the mask information can indicate the pixels belonging to the target entity in the first sub-image. Combined with the text corresponding to the target result in the association relationship, the image features of the pixels belonging to the target entity in the first sub-image can be processed into the image features corresponding to the target result, thereby obtaining the second sub-image. In one example, the text indicating the target result is "red", then the RGB value corresponding to the target result is (255,0,0). The R value of the pixels belonging to the target entity in the first sub-image can be set to 255, and the G and B values ​​can be set to 0, thereby processing the sofa in the first sub-image to be red.

[0069] Step S305: The second sub-image is fused with the image to be processed to obtain the target image.

[0070] In this embodiment, a second sub-image can be overlaid on the location information of the target entity in the image to be processed, and edge blending technology is used to ensure a smooth transition between the second sub-image and the surrounding environment, thereby obtaining the target image.

[0071] In this disclosure, based on the mask information of the first sub-image corresponding to the target entity in the image to be processed, a second sub-image is generated to process the target entity into the target result. Then, the second sub-image is fused with the image to be processed to obtain the target image. This allows the first sub-image corresponding to the target entity to be processed separately, which not only allows for better control over the processing effect of the first sub-image, but also improves the generation efficiency of the target image.

[0072] Figure 5 A flowchart illustrating an image processing method according to an embodiment of this disclosure is shown. Figure 4 ,like Figure 5As shown, an image processing method includes:

[0073] Step S401: Obtain the input text and determine the image to be processed corresponding to the input text.

[0074] Step S402: Perform image-text association processing on the input text and the image to be processed to obtain the association relationship between the input text and the image to be processed.

[0075] The specific implementation details of steps S401-S402 are similar to those of steps S101-S102, and will not be repeated here.

[0076] Step S403: Determine the location information of the target entity in the image to be processed and the text corresponding to the target result based on the association relationship.

[0077] Step S404: Generate the target image based on the location information, the text corresponding to the target result, and the image to be processed.

[0078] In this embodiment, the location information of the target entity in the image to be processed and the text corresponding to the target result can be determined first from the association relationship. Then, the target image is directly generated based on the location information, the text corresponding to the target result, and the image to be processed. When generating the target image, the image features other than the location information are kept the same as those in the image to be processed, and the image features at the location information are generated as the image features corresponding to the target result. That is, the image features other than the location information in the target image are the same as those in the image to be processed, and the image features at the location information are the image features corresponding to the target result. This makes the target image more natural and ensures the overall consistency of the target image.

[0079] Figure 6 A flowchart illustrating an image processing method according to an embodiment of this disclosure is shown. Figure 5 ,like Figure 6 As shown, an image processing method includes:

[0080] Step S501: Obtain the input text and determine the image to be processed corresponding to the input text.

[0081] Step S502: Perform image-text association processing on the input text and the image to be processed to obtain the association relationship between the input text and the image to be processed.

[0082] The specific implementation details of steps S501-S502 are similar to those of steps S101-S102, and will not be repeated here.

[0083] Step S503: Semantically expand the text corresponding to the target result in the association relationship to obtain multiple sub-results.

[0084] In this embodiment, the similarity between the text corresponding to the sub-result and the text corresponding to the target result meets the target threshold. For example, if the text corresponding to the target result is "red", then the sub-results can be "dark red", "light red", "orange-red", and "pink", etc. In one example, word vector models such as Word2Vec or GloVe can be used to obtain the vector representation of the text corresponding to the target result, and the cosine similarity between the vector representation of the target result and the vector representations of all words in the vocabulary can be calculated. Words in the vocabulary with a cosine similarity greater than the target threshold are identified as sub-results. Alternatively, the BERT model can be used to obtain the embedding representation of the text corresponding to the target result in different contexts, and cluster analysis can be performed on the embeddings corresponding to the target result. Or, similarity search can be used to find sub-results that are semantically similar to the text corresponding to the target result. The BERT model is a pre-trained language representation model based on the Transformer architecture.

[0085] Step S504: Generate multiple sub-images based on the association relationship.

[0086] In this embodiment, a sub-image is an image after the target entity in the image to be processed has been processed into a sub-result. For example, the color of the target entity "the sofa on the left" in the image to be processed can be changed to "dark red", "light red", "orange red" and "pink", etc., to obtain multiple sub-images. The process of generating sub-images based on the association relationship is similar to step S103 "generating target images based on association relationship", and will not be described in detail here.

[0087] Step S505: In response to the selection operation for multiple sub-images, the sub-image corresponding to the selection operation is determined as the target image.

[0088] In this embodiment, multiple sub-images can be displayed, allowing the user to select a satisfactory sub-image from among them. The system responds to selection operations on multiple sub-images, designating the selected sub-image as the target image, thereby obtaining the target image that meets the user's needs and improving the user experience. In one example, the selection operation can be a click operation, a selection box operation, or a drag operation, etc.

[0089] Figure 7 This illustration shows a scenario illustrating an image processing method according to an embodiment of the present disclosure. Figure 2 ,like Figure 7As shown, the user inputs "Change the sofa on the left in this image to red." Based on the user's click on image A, "this image" is determined to be image A. The target result "red" in the input information can be expanded to "dark red," "light red," and "orange-red." Then, the color of the "sofa on the left" in image A is changed to "dark red," "light red," and "orange-red" respectively, resulting in images A1, A2, and A3. Based on the user's click on image A1, image A1 is determined to be the target image requested by the user. After determining the target image as image A1, it can be processed as follows... Figure 7 The dialog box will automatically display image A1, or you can choose not to display image A1 in the dialog box.

[0090] Figure 8 This illustration shows a scenario illustrating an image processing method according to an embodiment of the present disclosure. Figure 3 ,like Figure 8 As shown, after determining the target image as image A1, image A1 can be displayed in the image display area on the left side of the application interface. If the user needs to process image A1 later, they can click or select image A1 in the image display area, or drag image A1 directly from the image display area to the dialog area on the right. Alternatively, images A1, A2, and A3 can all be displayed in the image display area.

[0091] Figure 9 A flowchart illustrating an image processing method according to an embodiment of this disclosure is shown. Figure 6 ,like Figure 9 As shown, an image processing method includes:

[0092] Step S601: Obtain the input text and determine the image to be processed corresponding to the input text.

[0093] Step S602: Perform image-text association processing on the input text and the image to be processed to obtain the association relationship between the input text and the image to be processed.

[0094] Step S603: Generate the target image based on the association relationship.

[0095] The specific implementation details of steps S601-S603 are similar to those of steps S101-S103, and will not be repeated here.

[0096] Step S604: Receive feedback information for the target image.

[0097] In this embodiment, if the user is not satisfied with the target image, they can input feedback information about the target image into the LLM through a dialog box in the LLM application interface. For example, the feedback information could be "The color of the sofa is too bright; I want a darker, richer red."

[0098] Step S605: Update the input text based on the feedback information.

[0099] In this embodiment, after receiving feedback information regarding the target image, the input text needs to be updated based on the feedback information to obtain more detailed and specific input text. For example, the updated input text could be "Change the sofa in the image to a darker, richer red." In one example, text recognition can be performed on the feedback information and the input text to determine the intent of the feedback information and the intent of the input text, respectively. The intents of the feedback information and the input text are then merged, and the updated input text is generated based on the merged intent.

[0100] Step S606: Adjust the target image based on the updated input text.

[0101] In this embodiment, the process of adjusting the target image based on the updated input text is similar to the process of processing the image to be processed based on the input text, and will not be described again here.

[0102] Figure 10 This illustration shows a scenario illustrating an image processing method according to an embodiment of the present disclosure. Figure 4 ,like Figure 10 As shown, the user inputs "Change the sofa on the left in this image to red." Based on the user's click on image A, "this image" is determined to be image A. The model changes the color of the "sofa on the left" in image A to red, resulting in image A1. If the user is not satisfied with image A1, they input feedback information: "The sofa color is too bright; I want a darker, more saturated red." The model combines the user's initial input information and feedback information to change the sofa on the left in image A1 to a darker, more saturated red, resulting in image A2. If the user is satisfied with image A2, the image generation is complete. If the user is not satisfied with image A2, they can input feedback information for image A2, and the model continues to adjust image A2 until a satisfactory image is generated.

[0103] In this disclosure, the target image can be continuously adjusted based on feedback information to obtain a target image that satisfies the user. During the adjustment process, it can also continuously learn and adapt to the user's preferences, thereby improving the accuracy of the target image and user satisfaction.

[0104] For ease of understanding of this disclosure, the following is combined with Figure 11 An image processing method in this disclosure will be explained:

[0105] Figure 11 A scene diagram illustrating an image processing method according to an embodiment of this disclosure is shown, such as... Figure 11As shown, the multimodal large language model 10, the first mapping layer 20, the second mapping layer 30, the image segmentation network 40, and the image generation model 50 together constitute a controllable network for image processing. The user inputs the text "Please change the cherry on the cake to a hazelnut" into the multimodal large language model 10. The multimodal large language model 10 determines the image to be processed corresponding to the input text. In the image to be processed, a cake has cherries on it. The multimodal large language model 10 performs feature extraction processing on the input text and the image to be processed, obtaining the feature information of the input text and the image to be processed, such as... Figure 11 As shown, feature information is represented by different tokens (feature units). The tokens in white and black areas represent the text features of the input text. The tokens in black areas include the text features corresponding to the target entity "cherry on the cake" and the target result "hazelnut". The tokens in the diamond-shaped shaded areas represent the image features of the image to be processed. The first mapping layer 20 is used to determine the text features corresponding to the target entity "cherry on the cake" from the feature information and maps the text features to a dimension that the image segmentation network 40 can understand. The image segmentation network 40 identifies the location information of "cherry on the cake" in the image to be processed based on the text features input from the first mapping layer, and maps the text features corresponding to "cherry on the cake" and the image features of "cherry on the cake" in the image to be processed. The location information in the processed image is sent to the image generation model 50; the second mapping layer 30 is used to determine the text features corresponding to the target result "hazelnut" from the feature information, and map the text features to a dimension that the image generation model 50 can understand. Then, the text features corresponding to the target result "hazelnut" are sent to the image generation model 50. The text features corresponding to "cherry on cake", the location information of "cherry on cake" in the image to be processed and the text features corresponding to "hazelnut" are associated as an association relationship; at the same time, the image features of the image to be processed are also sent to the image generation model 50 as a condition. Based on the association relationship and the image features of the image to be processed, the image generation model 50 can directly generate the target image, in which the cherry is replaced with hazelnut.

[0106] According to embodiments of this disclosure, this disclosure also provides an electronic device and a readable storage medium.

[0107] Figure 12A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0108] like Figure 12 As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0109] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0110] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as an image processing method. For example, in some embodiments, an image processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of an image processing method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform an image processing method by any other suitable means (e.g., by means of firmware).

[0111] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0112] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0113] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0114] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0115] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0116] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0117] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0118] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means two or more, unless otherwise explicitly specified.

[0119] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. An image processing method, include: Acquire input text, and determine an image to be processed corresponding to the input text, wherein the input text is used to indicate that a target entity in the image to be processed is processed as a target result; Performing image-text association processing on the input text and the image to be processed to obtain an association relationship between the input text and the image to be processed; A target image is generated based on the association relationship, where the target image is an image obtained by processing the target entity in the image to be processed into the target result.

2. The method according to claim 1, wherein the input text and the image to be processed are subjected to image-text association processing to obtain an association relationship between the input text and the image to be processed, include: The input text and the image to be processed are input into a multimodal large language model for image-text association processing, and the multimodal large language model outputs an association relationship between the input text and the image to be processed.

3. The method according to claim 1, wherein the input text and the image to be processed are subjected to image-text association processing to obtain an association relationship between the input text and the image to be processed; include: Performing text recognition on the input text to determine the text corresponding to the target entity and the text corresponding to the target result; Performing image recognition on the image to be processed to obtain multiple entities in the image to be processed and location information of the entities in the image to be processed; The text corresponding to the target entity, the position information of the target entity in the image to be processed, and the text corresponding to the target result are associated to obtain the association relationship.

4. The method according to claim 1, wherein the target image is generated based on the association relationship. include: The association relationship and the image to be processed are input into an image generation model, and the image generation model processes the target entity in the image to be processed into the target result to obtain the target image.

5. The method according to claim 1, wherein the target image is generated based on the association relationship, include: Based on the position information of the target entity in the association relationship in the image to be processed, determining a first sub-image corresponding to the target entity in the image to be processed and mask information of the first sub-image; Based on the mask information, generating a second sub-image in which the target entity is processed into the target result; The second sub-image is fused with the image to be processed to obtain the target image.

6. The method according to claim 1, wherein the target image is generated based on the association relationship, include: Determine the position information of the target entity in the image to be processed and the text corresponding to the target result based on the association relationship; Based on the position information, the text corresponding to the target result and the image to be processed, a target image is generated, the image features of the target image outside the position information are the same as those of the image to be processed, and the image features at the position information are the image features corresponding to the target result.

7. The method according to claim 1, wherein the target image is generated based on the association relationship, include: Performing semantic expansion on the text corresponding to the target result in the association relationship to obtain multiple sub-results, wherein the similarity between the text corresponding to the sub-results and the text corresponding to the target result satisfies a target threshold; Generate a plurality of sub-images based on the association relationship, wherein the sub-images are images obtained by processing the target entity in the image to be processed into the sub-results; In response to a selection operation on the plurality of sub-images, the sub-image corresponding to the selection operation is determined as the target image.

8. The method according to any one of claims 1 to 7, further comprising: include: receiving feedback information for the target image; updating the input text based on the feedback information; The target image is adjusted based on the updated input text.

9. The method according to any one of claims 1 to 7, wherein the determining the image to be processed corresponding to the input text, include: Based on the text in the input text used to indicate the image, determining the image to be processed corresponding to the input text among all the images in the current conversation; or, Based on at least one of a click operation, a frame selection operation, and a drag operation performed on the image, an image to be processed corresponding to the input text is determined.

10. An electronic device, include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.