Image processing method and apparatus, device, and computer readable storage medium

Abstract

The present application discloses an image processing method and apparatus, a device, and a computer readable storage medium. Embodiments of the present invention can be applied to scenarios such as cloud technology, artificial intelligence, intelligent transportation, and assisted driving. The method comprises: acquiring a reference image, and acquiring a similar image similar to the reference image; determining difference information between the reference image and the similar image; determining a target mask map in the reference image for the difference information, wherein the target mask map is used for highlighting a difference object, determined on the basis of the difference information, in the reference image relative to the similar image,; on the basis of the difference object and the reference image, expanding a text to obtain a difference description text; and on the basis of the target mask map, the difference description text, and the reference image, performing local adjustment on the similar image to obtain a target image. The present application can improve the accuracy of image adjustment, so that the adjusted image effect conforms to actual requirements, facilitating the development of other subsequent services.

Description

Image processing method, apparatus, device, and computer-readable storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on August 9, 2023, with application number 202310999358.9 and application name “Image processing method, device, apparatus and computer-readable storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computer technology, and in particular to an image processing method, apparatus, device, and computer-readable storage medium. Background Art

[0003] Artificial Intelligence (AI) has been applied to a wide range of fields, encompassing technologies such as computer vision, speech processing, and natural language processing. Computer vision technology has profound implications for image processing. For example, computer vision can be used to perform various image processing tasks.

[0004] In order to complete the image adjustment task, related technologies generally adjust the target image based on the existing diffusion model and the description information of the image adjustment to obtain the adjusted image.

[0005] During the research and practice of related technologies, the inventors of this application discovered that when performing image adjustment, the related technology adaptively adjusts the description information while keeping the network parameters of the existing diffusion model unchanged, so as to adjust the target image based on the adjusted description information. This easily ignores more image details, reduces the accuracy of image adjustment, and makes the adjusted image effect inconsistent with actual needs, which is not conducive to the development of subsequent business.

[0006] Summary of the Invention

[0007] The embodiments of the present application provide an image processing method, apparatus, device, and computer-readable storage medium, which can solve the problem that the adjusted image effect does not meet actual needs and improve the accuracy of image adjustment.

[0008] The present invention provides an image processing method, including:

[0009] Obtain a reference image, obtain the similarity between each image in a preset database and the reference image, and determine the image with the greatest similarity as a similar image similar to the reference image;

[0010] determining difference information between a reference image and a similar image;

[0011] Determining a target mask map for the difference information in the reference image; the target mask map is used to highlight the difference object in the reference image determined according to the difference information relative to the similar image;

[0012] Based on the difference object and the reference image, a textual expansion expression is performed to obtain a difference description text; the difference description text is used to describe the content difference between the reference image and the similar image;

[0013] Based on the target mask image, the difference description text and the reference image, local adjustments are made to the similar images to obtain the target image; the target image is an image that meets the content requirements of the reference image.

[0014] Accordingly, an embodiment of the present application provides an image processing device, including:

[0015] an acquiring unit, configured to acquire a reference image, obtain a similarity between each image in a preset database and the reference image, and determine an image with the greatest similarity as a similar image to the reference image;

[0016] a first determining unit, configured to determine difference information between a reference image and a similar image;

[0017] A second determining unit is configured to determine a target mask map for the difference information in the reference image; the target mask map is configured to highlight the difference object in the reference image determined based on the difference information relative to the similar image;

[0018] An expansion unit is used to expand the text based on the difference object and the reference image to obtain a difference description text; the difference description text is used to describe the content difference between the reference image and the similar image;

[0019] The adjustment unit is used to perform local adjustments on the similar images according to the target mask image, the difference description text and the reference image to obtain the target image; the target image is an image that meets the content requirements of the reference image.

[0020] In some embodiments, the adjustment unit is further configured to:

[0021] Performing noise processing on the similar image, and obtaining a first similar noise map and a second similar noise map at adjacent time steps in the noise processing;

[0022] De-noising the first similar noise image according to the target mask image and the difference description text to obtain a first image;

[0023] De-noising the second similar noise image according to the target mask image and the reference image to obtain a second image;

[0024] The first image is fused with the second image to obtain a target image.

[0025] In some embodiments, the adjustment unit is further configured to:

[0026] Performing mask processing on the first similar noise image according to the target mask image to obtain a first mask noise image;

[0027] Obtaining a difference text vector corresponding to the difference description text, and performing noise reduction processing on the first similar noise map according to the difference text vector to obtain a first feature map;

[0028] The first feature map is decoded to obtain a first image.

[0029] In some embodiments, the adjustment unit is further configured to:

[0030] Invert the target mask image to obtain the target inverse mask image corresponding to the target mask image;

[0031] performing masking processing on the second similar noise image according to the target inverse mask image to obtain a second masked noise image;

[0032] performing noise reduction processing on the second mask noise map according to the feature map corresponding to the reference image to obtain a second feature map;

[0033] The second feature map is decoded to obtain a second image.

[0034] In some embodiments, the adjustment unit is further configured to:

[0035] According to the target mask image and the difference description text, the similar image is locally fine-tuned to obtain the first image;

[0036] According to the target mask image and the reference image, the similar image is locally fine-tuned to obtain a second image;

[0037] The first image is fused with the second image to obtain a target image.

[0038] In some embodiments, the adjustment unit is further configured to:

[0039] Performing mask processing on the similar image according to the target mask image to obtain a first similar mask image;

[0040] Performing noise processing on the first similar mask image to obtain a first similar noise image;

[0041] Get the difference text vector corresponding to the difference description text;

[0042] Performing noise reduction processing on the first similar noise image according to the difference text vector to obtain a first feature image;

[0043] The first feature map is decoded to obtain a first image.

[0044] In some embodiments, the noise reduction process is a continuous multiple noise reduction process; the adjustment unit is further configured to:

[0045] The difference text vector is incorporated into each denoising process of the first similar noise map through the attention mechanism to obtain a first feature map; the first feature map contains semantic information associated with the difference text vector.

[0046] In some embodiments, the adjustment unit is further configured to:

[0047] Performing encoding processing on the first similarity mask image to obtain an encoded feature map;

[0048] Noise processing is performed on the encoded feature map to obtain a first similar noise map.

[0049] In some embodiments, the adjustment unit is further configured to:

[0050] Using a first neural network model, locally fine-tuning the similar image based on the target mask image and the difference description text to generate a first image;

[0051] The image processing apparatus further includes a training unit, configured to:

[0052] Acquire a sample reference image, a sample similar image, and a first sample target image;

[0053] Based on the difference information between the sample reference image and the sample similar image, a sample target mask image and a sample difference description text of the sample reference image are generated;

[0054] Based on the sample target mask image and the sample difference description text, a preset model is used to perform local fine-tuning on the sample similar image to generate a first predicted image;

[0055] determining a prediction loss based on the first sample target image and the first predicted image;

[0056] The preset model is iteratively trained based on the prediction loss until the preset model convergence condition is reached to obtain the first neural network model.

[0057] In some embodiments, the adjustment unit is further configured to:

[0058] Invert the target mask image to obtain the target inverse mask image corresponding to the target mask image;

[0059] Performing mask processing on the similar image according to the target inverse mask image to obtain a second similar mask image;

[0060] Noising the second similar mask image to obtain a second similar noise map, wherein a time step of the second similar noise map is adjacent to a time step of the first similar noise map;

[0061] Performing noise reduction processing on the second similar noise map according to the feature map corresponding to the reference image to obtain a second feature map;

[0062] The second feature map is decoded to obtain a second image.

[0063] In some embodiments, the noise reduction process is a continuous multiple noise reduction process; the adjustment unit is further configured to:

[0064] The feature map corresponding to the reference image is incorporated into the denoising process of the second similar noise map through the attention mechanism to obtain a second feature map; the second feature map contains semantic information associated with the feature map corresponding to the reference image.

[0065] In some embodiments, the expansion unit is further configured to:

[0066] determining a difference object of the reference image relative to the similar image based on the difference information;

[0067] determining object relationship information between the difference object and each object in the reference image;

[0068] Obtaining a global description text of the reference image and a target object description text of the difference object;

[0069] The text is expanded based on the global description text, the target object description text and the object relationship information to obtain the difference description text.

[0070] In some embodiments, the acquisition unit is further configured to:

[0071] determining the reference cluster center to which the reference image belongs;

[0072] Determine the feature category distance between each image cluster center pre-constructed in a preset database and a reference cluster center;

[0073] Based on the feature category distance, the similarity between the image corresponding to each image cluster center and the reference image is determined, and the image with the maximum similarity is determined as the similar image similar to the reference image.

[0074] In some embodiments, the first determining unit is further configured to:

[0075] Obtaining a first description text corresponding to the reference image;

[0076] Obtaining a second description text corresponding to the similar image;

[0077] Based on the difference between the first description text and the second description text, difference information is generated.

[0078] In some embodiments, the first determining unit is further configured to:

[0079] Performing a global description of the reference image using a first preset model to generate a global description text of the reference image;

[0080] Processing the pixel area where each object in the reference image is located using the second preset model to obtain object description text corresponding to each object in the reference image;

[0081] A first description text corresponding to the reference image is determined according to the global description text of the reference image and the object description text corresponding to each object.

[0082] In some embodiments, the second determining unit is further configured to:

[0083] determining, based on the difference information, a difference object in the reference image relative to the similar image;

[0084] Based on the difference objects in the reference image, a target mask map is generated.

[0085] In addition, an embodiment of the present application also provides a computer device, including a processor and a memory, the memory storing a computer program, and the processor is used to run the computer program in the memory to implement the steps in any image processing method provided in the embodiment of the present application.

[0086] In addition, an embodiment of the present application also provides a computer-readable storage medium, which stores multiple instructions, and the instructions are suitable for a processor to load to execute the steps in any image processing method provided in the embodiment of the present application.

[0087] In addition, an embodiment of the present application also provides a computer program product, including computer instructions, which, when executed, implement the steps of any image processing method provided in the embodiment of the present application.

[0088] The embodiment of the present application can first obtain a similar image similar to the reference image from existing data, and then generate a target mask map based on the difference information between the reference image and the similar image. The target mask map is used to highlight the difference object in the reference image determined according to the difference information relative to the similar image, and then perform textual expansion based on the difference object and the reference image to enrich the difference description text representing the difference information. The difference description text is a text used to describe the content difference between the reference image and the similar image. Finally, the existing similar image is locally fine-tuned by combining the target mask, the difference description text and the reference image to obtain a target image that can meet the content requirements of the reference image; in this way, the difference between the images can be expanded and described, and the similar image can be locally adjusted based on the expanded difference description text and the reference image as constraints, thereby improving the accuracy of the image adjustment and making the adjusted image effect consistent with the actual needs, so as to facilitate the subsequent development of other businesses. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0090] FIG1 is a schematic diagram of a scene of an image processing system provided in an embodiment of the present application;

[0091] FIG2 is a schematic diagram of the steps of the image processing method provided in an embodiment of the present application;

[0092] FIG3 is a schematic diagram of a generation scenario of a global description text provided in an embodiment of the present application;

[0093] FIG4 is a schematic diagram of a scene for generating description text of an object in an image provided by an embodiment of the present application;

[0094] FIG5 is a schematic diagram of a mask segmentation scenario of a mask segmentation model provided in an embodiment of the present application;

[0095] FIG6 is a schematic diagram of the structure of the latent space diffusion model provided in an embodiment of the present application;

[0096] FIG7 is a schematic diagram of the structure of a noise reduction network layer in reverse diffusion provided by an embodiment of the present application;

[0097] FIG8 is a schematic diagram of another step flow of the image processing method provided in an embodiment of the present application;

[0098] FIG9 is a schematic diagram of the framework structure of the image processing system provided in an embodiment of the application;

[0099] FIG10 is a schematic diagram of the structure of the residual network layer provided in an embodiment of the present application;

[0100] FIG11 is a schematic diagram of a scenario in which difference information is aggregated to generate a difference description text according to an embodiment of the present application;

[0101] FIG12 is a schematic diagram of a scene of an image fine-tuning process provided in an embodiment of the present application;

[0102] FIG13 is a schematic structural diagram of an image processing device provided in an embodiment of the present application;

[0103] FIG14 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0104] The embodiments of the present application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application and are not to be construed as limiting the present application.

[0105] Some processes described in the specification, claims, and accompanying figures include multiple steps that appear in a specific order. However, it should be understood that these steps may be performed in a different order or in parallel. Step numbers are used solely to distinguish between different steps and do not inherently indicate an order of execution. Furthermore, terms such as "first" and "second" are used to distinguish similar items and do not necessarily describe a specific order or precedence.

[0106] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0107] The embodiments of the present application provide an image processing method, apparatus, device and computer-readable storage medium. Specifically, the embodiments of the present application will be described from the dimension of an image processing apparatus, which can be specifically integrated into a computer device, which can be a server or a user terminal or other device. Among them, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Among them, the user terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, smart home appliance, car terminal, intelligent voice interaction device, aircraft, etc., but is not limited to this.

[0108] It is understandable that in the specific implementation of this application, data related to user information, user usage records, user status, etc. is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0109] It should be noted that the image processing method provided in the embodiments of the present application can be applied to any image adjustment scenario, which is not limited to being implemented through cloud services, big data, artificial intelligence, or a combination thereof, and is specifically illustrated by the following embodiments:

[0110] The image processing method provided in the embodiments of the present application involves artificial intelligence (AI) technology. Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0111] Computer vision (CV) is the science of making machines "see." Specifically, it refers to the use of cameras and computers to replace the human eye in identifying, tracking, and measuring targets, and further processing them to produce images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems that can extract information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0112] However, the embodiments of the present application can implement image processing through computer vision technologies such as image processing, image recognition, and image semantic understanding to complete image processing tasks. This is specifically described in the following embodiments:

[0113] It should be noted that in this image processing scenario, it is primarily implemented using a neural network (ANN) model, hereinafter referred to as the "model." The image processing process can include a model training phase (A) and an application phase (B). The training and application phases can be implemented using a combination of one or more devices in the image processing system.

[0114] For example, see Figure 1, which is a scenario diagram of the image processing system provided in an embodiment of the present application. The scenario system may include a server and / or a terminal; when the system only includes a server or a terminal, the server or the terminal includes a target database, a model training device, and a model application device; when the system is a combination of a terminal and a server, the server may include a target database, a model training device, and a model application device.

[0115] The target database may store a large amount of data, which is not limited to image data, to serve as sample similar images in the model training phase.

[0116] (A) Model training phase:

[0117] In the model training phase, the model training device may, after acquiring training data as a sample, train the preset model based on the acquired training data. Specifically, the model training phase may include preparing training data and model training.

[0118] The training data preparation process involves setting a sample reference image and a sample target image to be adjusted. Next, sample similar images similar to the sample reference image are obtained from the target database. Furthermore, based on the difference information between the sample reference image and the sample similar images, a sample target mask image and sample difference description text are generated for the sample reference image. This completes the training data.

[0119] The training of the model can be understood as comparative learning training between the image output by the model and the sample target image. Since the image adjustment process of the embodiment of the present application may include: fine-tuning similar images using the sample difference description text as a guiding condition, and fine-tuning similar images using the reference image as a constraint condition, this can be achieved through two models for image fine-tuning respectively. Therefore, the model training process can include training different models. In this case, the sample target image includes a first sample target image and a second sample target image.

[0120] As shown in Figure 1, the training process of the model with the sample difference description text as the guiding condition is taken as an example. Specifically, the process of the model training is: based on the sample target mask map and the sample difference description text, the sample similar image is locally fine-tuned by the preset model to obtain a first predicted image, and then the first sample target image is compared with the first predicted image, and when there is a difference between the two, the prediction loss is constructed according to the difference between the first sample target image and the first predicted image, so as to train the preset model based on the prediction loss; the model is iteratively trained in the above manner until the preset model convergence condition is reached, such as the first predicted image output by the preset model is the same as the first sample target image, or the number of iterative training reaches a certain number, or the first predicted image output by the preset model no longer changes, etc., to obtain the trained first neural network model.

[0121] Similarly, in combination with the training process of the model with the sample reference image as the constraint condition shown in Figure 1, an example is taken. Specifically, in the process of preparing the training data, in addition to obtaining the sample reference image, the sample target mask map and the sample similarity image, it is also necessary to obtain the sample target inverse mask map associated with the sample target mask map (that is, the mask map after the sample target mask map is inverted). Then, the sample similarity image is locally fine-tuned based on the sample target inverse mask map and the sample reference image through the preset model to obtain a second predicted image, and then, the second sample target image is compared with the second predicted image, and when there is a difference between the two, a prediction loss is constructed according to the difference between the second sample target image and the second predicted image, so as to iteratively train the preset model based on the prediction loss until the preset model convergence condition is reached, thereby obtaining a trained second neural network model.

[0122] At this point, the training process based on the model training device is completed, and the first neural network model and the second neural network model are obtained respectively. The first neural network model and the second neural network model obtained by training can be used to participate in the image processing process of this application. It should be noted that in the model training stage, a conditional latent space diffusion (Stable Diffusion, SD) model can be used for training, and the trained first neural network model and the second neural network model are SD models.

[0123] In addition, the sample similar images are fine-tuned using the sample difference description text as a guiding condition, and the sample reference image as a constraint condition. A single model can also be used to fine-tune the image and complete image processing. During the model training phase, the sample similar images can be locally fine-tuned based on the sample target mask map, the sample difference description text, and the sample reference image using a preset model to obtain a predicted image. The sample target image is then compared with the predicted image. When there is a difference between the two, a prediction loss is constructed based on the difference between the sample target image and the predicted image. The preset model is then iteratively trained based on the prediction loss until the preset model converges, resulting in a trained target model.

[0124] (B) Model application stage:

[0125] In the application phase of the model, the trained first neural network model and the second neural network model can be uploaded or installed into a model application device, so that the model application device runs the first neural network model and the second neural network model during the image processing process to cooperate with the completion of the relevant process of image processing. Specifically, the image processing process includes: obtaining a reference image and obtaining a similar image similar to the reference image; determining difference information between the reference image and the similar image; determining a target mask map in the reference image for the difference information, the target mask map being used to highlight the difference objects in the reference image relative to the similar image determined according to the difference information; performing textual expansion based on the difference objects and the reference image to obtain a difference description text; the difference description text is used to describe the content difference between the reference image and the similar image; performing local fine-tuning on the similar image based on the target mask map and the difference description text by the first neural network model to generate a first image; and performing local fine-tuning on the similar image based on the target inverse mask map (i.e., a mask map obtained by inverting the target mask map) and the reference image by the second neural network model to generate a second image; and fusing the first image with the second image to obtain a target image, which is an image that meets the content requirements of the reference image.

[0126] In addition, if a model is used to fine-tune an image, the trained target model can be uploaded or installed into a model application device so that the model application device runs the target model during the image processing process to cooperate with the completion of relevant image processing processes. The image processing process includes: obtaining a reference image and obtaining a similar image that is similar to the reference image; determining the difference information between the reference image and the similar image; determining a target mask map for the difference information in the reference image, the target mask map is used to highlight the difference object in the reference image determined according to the difference information relative to the similar image; performing textual expansion based on the difference object and the reference image to obtain a difference description text; the difference description text is used to describe the content difference between the reference image and the similar image; performing local adjustments to the similar image based on the target mask map, the difference description text and the reference image to obtain the above-mentioned target image.

[0127] Through the scenarios of the training phase and application phase of the above model, the image processing method of the present application can be implemented.

[0128] For example, assuming that a server or terminal includes a target database, a model training device, and a model application device, the server or terminal can prepare training data based on sample image data in the target database, and train a preset model according to the training data through the model training device, and transmit the trained first neural network model and second neural network model to the model application device for operation. In this case, the terminal or server can implement the following: obtain a reference image and obtain a similar image similar to the reference image; determine difference information between the reference image and the similar image; determine a target mask map for the difference information in the reference image, the target mask map is used to highlight the difference object in the reference image determined according to the difference information relative to the similar image; perform textual expansion based on the difference object and the reference image to obtain a difference description text; the difference description text is used to describe the content difference between the reference image and the similar image; perform local adjustments on the similar image according to the target mask map, the difference description text, and the reference image to obtain a target image.

[0129] For another example, consider a system consisting of a terminal and a server, where a communication connection is established between the terminal and the server. The server can be a distributed service system composed of multiple physical servers, comprising at least a target database, a model training device, and a model application device. After completing model training on the server, the trained first and second neural network models, or the trained target model, can be run on the server to implement the image processing process. Specifically, during the application phase, a reference image can be sent to the server via a client on the terminal. After obtaining the reference image, the server can retrieve similar images from the target database that are similar to the reference image; determine difference information between the reference image and the similar image; determine a target mask in the reference image based on the difference information, the target mask being used to highlight the difference objects in the reference image relative to the similar image, as determined based on the difference information; perform textual expansion based on the difference objects and the reference image to obtain a difference description text; the difference description text is used to describe the content difference between the reference image and the similar image; and perform local adjustments to the similar image based on the target mask, the difference description text, and the reference image to obtain the target image. The server can then return the target image to the terminal.

[0130] For example, as shown in FIG1 , assuming that an image processing application (client) is installed on a terminal, a user can select an image search task on the image processing application to execute an image processing process. Specifically, the image processing process is as follows: first, the user can select an image search task on the client page on the terminal and set a reference image to be searched for the image search task; then, the client transmits the reference image to the server. After obtaining the reference image, the server can obtain similar images similar to the reference image from a target database; determine difference information between the reference image and the similar image; determine a target mask map for the difference information in the reference image, the target mask map being used to highlight the difference objects in the reference image relative to the similar image determined based on the difference information; perform textual expansion based on the difference objects and the reference image to obtain a difference description text; the difference description text is used to describe the content difference between the reference image and the similar image; and perform local adjustments on the similar image based on the target mask map, the difference description text, and the reference image to obtain a target image. Finally, the server returns the target image that has been locally fine-tuned for the similar image to the client, so that the image provided by the image search service can be more consistent with the actual demand (reference image), which is conducive to the development of the image search service.

[0131] It should be noted that the above are only examples and can also be applied to other image services, which will not be described in detail here.

[0132] For ease of understanding, each step of the image processing method will be described in detail below. It should be noted that the order of the following embodiments is not intended to limit the preferred order of the embodiments.

[0133] In the embodiments of the present application, the image processing apparatus will be described from the perspective of being integrated into a computer device such as a terminal or server. Referring to FIG2 , FIG2 is a schematic flow chart of the steps of an image processing method provided in the embodiments of the present application. In the embodiments of the present application, the image processing apparatus is specifically integrated into a server as an example. When the processor on the server executes the program instructions corresponding to the image processing method, the specific process is as follows:

[0134] 101. Obtain a reference image, obtain similarities between each image in a preset database and the reference image, and determine an image with the greatest similarity as a similar image to the reference image.

[0135] After obtaining the reference image, in order to obtain an image that meets the actual requirements of the reference image, the embodiment of the present application generally searches for an image that is most similar to the reference image from the existing image data, so that the similar image found can be further processed. The image processing process can be image adjustment, such as large-scale adjustment, local adjustment, etc., so that the information contained in the target image better matches the reference image, such as the obtained image is closer to the reference image in style and content, and is reliable.

[0136] The reference image can include any content, such as any one or more content, such as fruit, tableware, animals, people, and animations. It can also include other forms of content, which are not listed here. It should be noted that the reference image can serve as a basis for image adjustment during image processing, i.e., other images can be adjusted based on the reference image.

[0137] The similar image may be the image in the database that is most similar to the reference image, which can be understood as the image in the existing data that is most similar to the reference image in terms of image content or image style. It should be noted that the embodiment of the present application uses the similar image as the basic data for image adjustment, that is, image adjustment is performed based on the similar image.

[0138] To facilitate understanding of reference images and similar images, the following examples illustrate these two types of images. For example, in the image search business, a customer sends an example image to the image search platform through the client. The example image contains two cats lying down. This example image can be considered a reference image. Upon receiving the example image sent by the customer, the image search platform can search its database for the most similar image to the example image. This allows the platform to make further adjustments to the similar image based on the relevant information of the reference image to best meet the customer's image search needs.

[0139] In some embodiments, to find similar images to a reference image in an existing database, the similarity between any two images can be determined using a feature distance method, or the similarity between two images can be measured using a feature distance. For example, step 101 may include: determining a reference cluster center to which the reference image belongs; determining a feature category distance between each pre-constructed image cluster center in a preset database and the reference cluster center; based on the feature category distance, determining the similarity between each image corresponding to each image cluster center and the reference image, and determining the image with the greatest similarity as a similar image to the reference image.

[0140] It should be noted that different images contain different content information, and images can be classified based on the content information they contain. For example, images in the animal category can be categorized by animal species, such as cats, dogs, tigers, horses, pigeons, eagles, and other animals. For one or more images belonging to the same category, the cluster center corresponding to that category can be calculated based on the one or more images.

[0141] Among them, the reference cluster center can be a feature cluster center constructed based on one or more reference images, which represents the center point of the feature category between the one or more reference images and can be understood as the feature mean point. For example, when there is a reference image, the reference image can be converted into a pixel matrix, and the pixel matrix of the reference image can be regarded as the reference cluster center; when there are multiple reference images, the pixel matrix of each reference image can be determined separately, and the reference cluster centers of the multiple reference images can be calculated by combining each pixel matrix, such as taking the mean between the multiple pixel matrices as the reference cluster center. The above is only an example and is not a specific limitation for the implementation of this application.

[0142] The image cluster center can be the cluster center of the image set corresponding to each category in the existing database. Each image cluster center can change according to the update of the image set of each category in the database, that is, the image cluster center can be constructed in real time. For example, the preset database may include images of subjects such as food, animals, plants, vehicles, and accessories. Each subject may contain one or more image categories, and each image category corresponds to an image cluster center. Taking animals as an example, assuming that the image set includes a cat category, in this case, each image in the cat category is represented by each pixel matrix, and the mean of all pixel matrices in the cat category is calculated as the image cluster center corresponding to the cat category. Assuming that the animal category also includes a dog category, the calculation method of the dog category image cluster center is consistent with the calculation method of the "cat category image cluster center". It should be noted that for images of any category under other subjects, the calculation method of their image cluster centers is the same as above, and will not be repeated here.

[0143] In order to select similar images similar to the reference image from a preset database, after determining the reference cluster center corresponding to the reference image, the feature category distance between the reference cluster center and each image cluster center in the database can be calculated. Furthermore, similar images similar to the reference image can be selected based on the feature category distance. Specifically, the size of the feature category distance can be used to determine which image cluster center in the database is closer to the cluster center of the reference image, so as to determine the image category of the similar target image cluster center as the image category of the reference cluster center. Then, the feature distance between the reference image and each image under the image category of the target image cluster center is calculated. It should be noted that the size of the feature distance can reflect the similarity between any two images. Therefore, similar images similar to the reference image can be selected based on the size of the feature distance. For example, the image with the smallest feature distance to the reference image is selected as the similar image.

[0144] Through the above method, after obtaining the reference image, the image most similar to the reference image can be searched from the existing image data, so that further image adjustments can be made to the searched image to obtain an image that better meets the actual needs of the reference image and is reliable.

[0145] 102. Determine difference information between the reference image and the similar image.

[0146] In an embodiment of the present application, in order to adjust similar images more accurately, the difference between the reference image and the similar image can be determined, so that the difference between the two images can be used as the basis for adjustment in the image processing process, and the similar image can be adjusted to improve the accuracy of image adjustment.

[0147] Among them, the difference information can be information indicating the feature difference between the reference image and the similar image, which is not limited to difference information such as the number of objects (things) with differences in the image, the position of the objects and / or the posture of the objects. For example, the reference image contains two orange cats, the first orange cat is lying on the grass, and the second orange cat is in a running state in the area around the first orange cat. Assuming that the similar image contains two orange cats, one of the orange cats is lying on the grass, and the other orange cat is in a running state in an area farther away from the lying orange cat, the difference information between the reference image and the similar image can be the position difference between the two orange cats, that is, the position relationship; for another example, the reference image contains two orange cats and a blue cat, and the similar image contains two orange cats, then there is a difference object (a blue cat) in the difference information of the reference image relative to the similar image. It should be noted that the difference information can be generated by combining the difference object and the object position relationship. The above is only an example and is not a specific limitation for implementing this application.

[0148] In some embodiments, to obtain difference information between a reference image and similar images, the difference between the images may be determined based on the description text of the reference image and the description text of the similar images, thereby generating difference information. For example, step 102 may include:

[0149] (102.1) Obtaining a first description text corresponding to the reference image;

[0150] (102.2) Obtaining a second description text corresponding to the similar image;

[0151] (102.3) Generate difference information based on the difference between the first description text and the second description text.

[0152] Among them, the first description text can be an image content description text generated for the content information in the reference image. Since the content information of the reference image can include object information and environmental information of the object, the first description text is not limited to including a global description text for the overall content of the reference image and an object description text for the object in the reference image. For example, assuming that the reference image contains two orange cats and a blue cat, the global description text can be "There are two orange cats and a blue cat on the grass. The second orange cat is located in the grass area to the upper right of the first orange cat, and the blue cat is located in the grass area to the upper left of the first orange cat." The object description text for the first orange cat is "The orange cat is lying on the grass," the object description text for the second orange cat is "The orange cat is running on the grass," and the object description text for the blue cat is "The blue cat is rolling on the grass." The above global description text and object description text are only examples and are not intended to be a specific limitation for implementing this application. Any of the above description texts can also be described in more detail or more concisely according to actual circumstances.

[0153] The second description text may be an image content description text generated based on the content information in the similar images. Similarly, the second description text is not limited to including a global description text for the overall content of the similar images and an object description text for the objects in the similar images. For specific examples, please refer to the description of the first description text and are not limited here.

[0154] Specifically, in order to obtain the difference information between the reference image and the similar images, after respectively obtaining the first description text of the reference image and the second description text of the similar image, the difference in content information between the reference image and the similar images can be learned based on the description difference between the first description text and the second description text, such as determining whether there is a difference in the number of objects between the reference image and the similar images, determining whether there is a difference in the object position distribution between the reference image and the similar images, determining whether there is a difference in the object posture between the reference image and the similar images, etc. It should be noted that the difference between the reference image and the similar images may include one or more of the above situations, and may also include other difference situations; further, based on the differences determined above, the difference information is obtained.

[0155] For example, assuming that the first description text for the reference image is "There are two orange cats on the grass, the first orange cat is lying on the grass, the second orange cat is located in the upper right area of ​​the first orange cat, and the second orange cat is running on the grass", and assuming that the second description text for the similar image is "There are two orange cats on the grass, the first orange cat is lying on the grass, the second orange cat is located in the upper left area of ​​the first orange cat, and the second orange cat is running on the grass", it can be concluded that the difference between the first description text and the second description text is the difference in the position distribution of the second orange cat (object) in the image. Therefore, difference information can be generated based on these two differences. The difference information can be information about the second orange cat (difference object) in the reference image, such as the position information, shape information, size information, etc. of the second orange cat. In addition, since the upper left area of ​​the first orange cat in the reference image is grass, and there is a second orange cat in the upper left area of ​​the first orange cat in the similar image, the difference information may also include information about the grass (difference object) in the upper left area of ​​the first orange cat in the reference image.

[0156] For example, suppose the first description text for the reference image is "There are two orange cats and one blue cat on the grass. The first orange cat is lying on the grass, the second orange cat is located in the upper right area of ​​the first orange cat, the second orange cat is running on the grass, the blue cat is located in the upper left area of ​​the first orange cat, and the blue cat is rolling on the grass." Suppose the second description text for the similar image is "There are two orange cats on the grass. The first orange cat is lying on the grass, the second orange cat is located in the upper right area of ​​the first orange cat, and the second orange cat is running on the grass." It can be concluded that the difference between the first description text and the second description text is the difference in the number of cats (objects), the difference in the position distribution of cats in the image, etc. Therefore, difference information can be generated based on these two differences. The difference information can be information for the blue cat (difference object), such as the object description text of the blue cat and its position information in the reference image.

[0157] In some embodiments, since both the reference image and the similar image are image data, in order to accurately obtain the text information describing the reference image and the similar image, a first description text of the reference image and a second description text of the similar image can be obtained through image-to-text conversion. For example, taking the acquisition of the first description text as an example, step (102.1) may include:

[0158] (102.1.1) Performing a global description of the reference image using a first preset model to generate a global description text of the reference image;

[0159] (102.1.2) Processing the pixel region where each object in the reference image is located using the second preset model to obtain object description text corresponding to each object in the reference image;

[0160] (102.1.3) Determine a first description text corresponding to the reference image based on the global description text of the reference image and the object description text corresponding to each object.

[0161] The global description text can be understood as a text that comprehensively describes and summarizes the content information contained in the image. Through the global description text, the image can be quickly understood in a text-based manner. For example, taking the reference image as an example, assuming that the content of the reference image includes grass, a frisbee, and a pet dog jumping towards the frisbee, the global description text can be "The pet dog jumps on the grass to play with the frisbee." For another example, assuming that the content of the reference image includes grass, a lying orange cat, and a running blue cat, the global description text can be "The orange cat lies on the grass to rest, and the blue cat runs and plays on the grass"; the above examples.

[0162] Among them, the object description text can be a text used to describe the features of the corresponding object in the image, which may include descriptions of the object's color, shape, posture, position, and other aspects. For example, taking the reference image as an example, the object description text can describe the position, action, posture, shape, and other information of the corresponding object in the image. It should be noted that the object description text may include an object category label and object description information. The object category label is used to indicate the category or name to which the corresponding object belongs, and the object description information is used to specifically describe the object's action, position, shape, posture, and the like. For example, assuming that the content information contained in an image is "a brown dog playing with a frisbee on the grass", the object category label may be "dog" or "brown dog", and the object description information may be "brown dog playing with a frisbee on the grass". The above is only an example and is not a specific limitation for the implementation of this application.

[0163] Among them, the first preset model can be a model for global text description of the overall situation of the image. For example, the first preset model can be a pre-trained frozen image encoder and large language model (Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models, BLIP2), which introduces large-scale language models (Large Language Models, LLM).

[0164] Specifically, as shown in Figure 3, the first preset model may structurally include two parts: vision-and-language representation learning and vision-to-language generative learning. The vision-and-language representation learning structure includes an image encoder and a lightweight query transformer (Q-Former), while the vision-to-language generative learning structure includes large language models (LLMs). Taking the global description of a reference image as an example, in the first preset model, first, the reference image is input into the image encoder, and the image encoder encodes the reference image to obtain an image encoding result; then, the image encoding result is input into the lightweight query transformer, and the object category label in the reference image is determined, so as to be fused with the object category label in the lightweight query transformer to obtain a fused feature result; finally, the fused feature result is input into the large language model for language processing to output a global description text for the reference image.

[0165] The second preset model can be a model for providing a text description of an image region containing an object in an image. For example, the second preset model can be a Generative Region-to-Text Transformer (GRIT). Specifically, as shown in FIG4 , the second preset model can structurally include a visual encoder, a foreground object extractor for locating objects, and a text decoder. Taking the text description of the pixel region where each object is located in the reference image as an example, first, each object in the reference image is identified to determine the pixel region where each object is located in the reference image, and each object identified in the reference image is classified. Then, the reference image with the pixel region and category of each object determined is input into the second preset model. In the second preset model, the second preset model performs a region-to-language conversion process on the pixel region where each object is located, and outputs a converted reference image. The converted reference image contains a marking box for marking each object, and an object description text of the object in each marking box.

[0166] Furthermore, after obtaining the global description text and the object description text of the reference image, the global description text and the object description text may be merged to obtain a first description text for the reference image.

[0167] In the embodiment of the present application, the method for obtaining the second description text of the similar image can refer to the above-mentioned process of obtaining the "first description text of the reference image", which will not be described in detail here.

[0168] In the above manner, the difference between the reference image and the similar image can be determined, so that the difference between the two images can be used as an adjustment basis in the image processing process to more accurately adjust the similar image.

[0169] 103. Determine a target mask map for difference information in the reference image.

[0170] In the embodiments of the present application, after determining that similar images need to be adjusted, local adjustments can be used to perform image processing. To achieve local adjustments of similar images, a mask image targeting the difference information in the reference image is obtained. This mask image can then be used to perform local fine-tuning of the similar images, thereby improving the accuracy of subsequent image adjustments. The target mask image is used to highlight the objects in the reference image that differ from the similar image, as determined based on the difference information.

[0171] Among them, the target mask map can be a pixel area mask map generated based on the difference information of the reference image relative to the similar image, which is used to mask part of the pixel area in the similar image when fine-tuning the similar image, so that the masked pixel area of ​​the similar image is blank (no content).

[0172] In some embodiments, a target mask map may be constructed based on the difference objects in the reference image relative to the similar image. For example, step 103 may include:

[0173] (103.1) determining a difference object in the reference image relative to the similar image based on the difference information;

[0174] (103.2) Generate a target mask map based on the difference object in the reference image.

[0175] Specifically, to obtain a target mask map for difference information in the reference image, after obtaining the difference information, the difference objects contained in the reference image relative to the similar image can be determined based on the difference information, and the target mask map can be generated based on the determined difference objects. It should be noted that when generating the target mask map, an initial mask map of the same size as the reference image can be constructed. Then, using information related to the difference objects as indication information, the target mask map is generated according to the indication information, the initial mask map, and the reference image.

[0176] In some embodiments, a target mask for difference information in the reference image can be obtained through semantic segmentation. For example, step (103.2) may include: constructing an initial mask of the same size as the reference image; obtaining indicative information for the difference object in the reference image, including but not limited to pixel regions of the background of the difference object, a marker box, and object description text; and generating a target mask based on the initial mask, the indicative information, and the reference image using a semantic segmentation model.

[0177] Among them, the mask segmentation model (seg everything) is used to generate a mask map of any image according to the indication information. As shown in Figure 5, the model may structurally include an image encoder (image encoder), a convolution module (conv), a fusion module, an encoder for indication information (prompt encoder), and a decoder for the mask map (mask decoder); specifically, the reference image is input into the semantic segmentation model, and the reference image is encoded by the image encoder to obtain an image vector. At the same time, the initial mask map is feature extracted by the convolution module (conv) to obtain a mask map vector; then, the image vector and the mask map vector are fused to obtain image fusion features; in addition, the indication information is encoded by the encoder for indication information (prompt encoder), and the indication information may include a pixel area (point) of the background of the difference object, a marking box (box) and an object description text (text), to obtain the encoding features of the indication information, and the decoder for the mask map (mask A decoder) decodes the image fusion features and the encoded features of the indication information, and outputs one or more prediction mask maps; it should be noted that when only one prediction mask map is output, the prediction mask map is used as the target mask map; when multiple prediction mask maps are output, each prediction mask map has a corresponding score, and the prediction mask map with the largest score can be selected as the target mask map.

[0178] To facilitate understanding of the target mask generation principle, the target mask will be described below using a scenario example. Specifically, assuming that the reference image contains two orange cats, the first lying on the grass, and the second running to the upper right of the first; and the similar image contains two orange cats, the first lying on the grass, and the second running to the upper left of the first. The difference information includes information about the second orange cat in the reference image and information about the grass to the upper left of the first, such as location distribution information, size information, shape information, etc. After constructing an initial mask with the same size as the reference image, a target mask can be generated based on the difference information obtained above. This target mask is primarily used to mask the pixel area of ​​the second orange cat in the similar image and the pixel area of ​​the grass to the upper right of the first orange cat in the similar image during image processing. In the target mask, these two pixel areas are represented by "0", and all pixel areas other than these two areas are represented by "1".

[0179] Through the above method, a target mask map of the difference information between the reference image and the similar image can be obtained, and the target mask map is used as a necessary element when adjusting the similar image to participate in the local fine-tuning of the similar image, so as to improve the accuracy of the image adjustment later.

[0180] 104. Perform textual expansion based on the difference object and the reference image to obtain a difference description text.

[0181] In an embodiment of the present application, in order to achieve local adjustment of similar images, in addition to obtaining a mask image for the difference information in the reference image, it is also necessary to obtain relevant descriptive text for the difference information in the reference image, so that the relevant descriptive text of the difference information can be used as a guiding condition for image adjustment in the future, so as to perform local adjustment on similar images and improve the accuracy of image adjustment.

[0182] The difference description text can be a text describing the content differences between the reference image and the similar image. This text is primarily derived from a textual expansion of the difference information, richly and accurately representing the differences between the reference image and the similar image. This difference description text can serve as a guide for adjusting the similar image, instructing local adjustments to the similar image and improving the accuracy of image adjustments.

[0183] In some embodiments, since the difference information can reflect the difference between the reference image and the similar image, in order to obtain a description text that fully describes the difference between the two images, the description can be expanded based on the relationship between the difference object and other objects in the reference image and the object description text of the difference object to obtain a description text for the difference between the two images, that is, a difference description text. For example, step 104 may include:

[0184] (104.1) determining a difference object of the reference image relative to the similar image based on the difference information;

[0185] (104.2) determining object relationship information between the difference object and each object in the reference image;

[0186] (104.3) Obtaining a global description text of the reference image and a target object description text of the difference object;

[0187] (104.4) Perform text expansion based on the global description text, the target object description text, and the object relationship information to obtain a difference description text.

[0188] Among them, the object relationship information can be information representing the relationship between the difference object and any object, and the object relationship is not limited to including the position distribution relationship (such as distance, direction) and category relationship (whether they belong to the same species attributes) between the difference object and other objects. Exemplarily, the object elements of the reference image include grass, an orange cat and a blue cat. The orange cat is lying in the grass area at the center of the reference image, and the blue cat is located on the grass in the upper right area of ​​the orange cat; the object elements of the similar image also include grass, an orange cat and a blue cat, but the difference is that the blue cat in the similar image is located on the grass in the upper left area of ​​the orange cat; it can be obtained that the difference object of the reference image relative to the similar image can at least include the blue cat. Taking the blue cat as the difference object as an example, the object relationship information can include the position distribution relationship between the blue cat and the orange cat in the reference image (such as the blue cat is located in the upper right area of ​​the orange cat) and the category relationship between the blue cat and the orange cat (the blue cat and the orange cat belong to the same species category).

[0189] Specifically, to provide a rich description of the difference information between a reference image and a similar image, the difference object corresponding to the difference information in the reference image is first determined, and the object relationship information between the difference object and any other object in the reference image is obtained, such as determining the positional relationship between the current difference object and other objects in the image and whether they belong to the same object category (object category). Furthermore, referring to the acquisition of the "first description text" described above, a global description text of the reference image is obtained, as well as object description text associated with the difference object. Finally, based on the global description text of the reference image, the description text of the difference object, and the object relationship information between the difference object and other objects, a text expansion description is performed to fully describe the difference information between the reference image and the similar image, resulting in an expanded difference description text. This difference description text richly expresses the relationship between the difference object and other objects in the reference image, as well as the object state information of the difference object. Thus, compared to the object description text, this difference description text can more richly represent the relevant information of the difference object in the reference image, so that it can serve as a guide when subsequently adjusting the similar image, achieving accurate and reliable fine-tuning of local areas of the similar image.

[0190] Exemplarily, when performing text expansion on the difference information, an existing large-scale language processing model can be used. Specifically, the object relationship information, the global description text, and the object description text of the difference object can be transmitted to the large-scale language processing model, so that the large-scale language processing model can mine relevant information from the global description text and the object description text of the difference object based on the object relationship information, and further perform text expansion to generate a text containing a rich description of the difference object, namely, the difference description text, to be used as data for subsequent image adjustment.

[0191] In the above manner, a text rich in description of the difference information in the reference image can be obtained, so that the difference description text can be subsequently used as a guiding condition to indicate local adjustments to similar images, thereby improving the accuracy of image adjustment.

[0192] 105. According to the target mask image, the difference description text and the reference image, the similar image is locally adjusted to obtain the target image.

[0193] In an embodiment of the present application, in order to obtain a target image that is more consistent with the content information of a reference image, after obtaining a similar image similar to the reference image, the similar image can be used as the image to be adjusted. Then, the target mask image, the difference description text and the reference image can be used as guiding conditions for image processing, and the similar image can be adjusted based on the guiding conditions to locally fine-tune the similar image and generate a target image that is more consistent with the content requirements of the reference image and is reliable.

[0194] The target image may be an image not currently stored in the database and is primarily obtained by performing local fine-tuning on a similar image. Specifically, after finding the most similar image to the reference image in the preset database, since this similar image may actually differ from the reference image to a certain extent, in order to obtain a similar image that more closely matches the reference image, the similar image may be used as a base image. This base image is then further fine-tuned based on the reference image, a target mask image for differences, and text describing the differences, to obtain a target image that is more similar to the reference image.

[0195] It should be noted that when adjusting local areas of similar images, the target mask, difference description text and reference image are mainly used as guiding conditions for image adjustment. Among them, the main function of the target mask map is to affect the presentation of pixels belonging to the target pixel area in the similar image. The target pixel area can be a pixel area corresponding to a difference or non-difference in the reference image relative to the similar image. Among them, the adjustment of similar images can include two parts. Specifically, the first part is to use the difference description text as a guiding condition and combine the target mask map to perform local adjustments on the similar images; the second part is to use the reference image as a guiding (constraint) condition and combine the target mask map to perform local adjustments on the similar images. For ease of understanding, the adjustment of similar images will be described in detail below.

[0196] In the embodiments of the present application, when performing local fine-tuning on similar images, the similar images can first be masked, and then the masked results can be fine-tuned to obtain the target image. Furthermore, the image can also be masked during the fine-tuning process. It should be noted that the fine-tuning process described above is not limited to being achieved through noise processing. Specific optional implementations are described below.

[0197] (A) First, mask the similar images and then fine-tune the masked results:

[0198] In some embodiments, the difference description text and the reference image can be used as guiding conditions for the similar image, and local adjustments can be performed on the similar images respectively, so as to fuse the two adjustment results to obtain the target image. For example, step 105 can include:

[0199] (105.A.1) Perform local fine-tuning on the similar image based on the target mask image and the difference description text to obtain the first image;

[0200] (105.A.2) Based on the target mask and the reference image, locally fine-tune the similar image to obtain a second image;

[0201] (105.A.3) Fuse the first image with the second image to obtain a target image.

[0202] The first image may be an image obtained by locally adjusting the similar image according to the difference description text. The content in the image is different from that in the similar image. Some areas in the first image contain blank pixel areas, i.e., areas without content. Specifically, the pixel areas corresponding to the difference information in the reference image are blank pixel areas. For example, assuming that the content of the reference image and the similar image both contain a blue cat and an orange cat playing on the grass, but the difference between the two images is "the position of the blue cat in the image," when locally fine-tuning the similar image according to the difference description text, the pixel areas at the same position in the similar image may be adjusted according to the pixel areas where the blue cat is located in the reference image. The adjusted pixel areas are defined as difference pixel areas to obtain the first image. The difference pixel areas in the first image will be replaced by blank pixel areas, i.e., without content. In addition, when fine-tuning the similar image, the pixel areas where the blue cat originally existed in the similar image may also be adjusted so that the pixel areas where the blue cat originally existed in the similar image in the first image are replaced by blank pixel areas. For another example, assuming that the reference image and the similar image both contain the content of "a plate with food and cutlery", but the difference between the reference image and the similar image lies in the placement of the cutlery, then after locally adjusting the similar image, the pixel area at the same position in the similar image is adjusted according to the pixel area where the cutlery is located in the reference image, and the adjusted pixel area in the obtained first image is replaced by a blank pixel area.

[0203] The second image may be an image obtained by locally adjusting the similar image according to the guidance conditions of the reference image. Most areas in the second image are blank pixel areas, and only the pixel areas corresponding to the difference information in the reference image are non-blank pixel areas. The content presented by the pixel combination in the non-blank pixel area is the image content of the difference object corresponding to the difference information in the reference image. For example, assuming that the content of the reference image and the similar image both contain "a blue cat and an orange cat playing on the grass", but the difference between the two images is "the position of the blue cat in the image", therefore, when locally fine-tuning the similar image according to the reference image, the pixel area at the same position in the similar image can be adjusted according to the pixel area where the blue cat is located in the reference image. The non-blank pixel area in the obtained second image contains the pixel area where the blue cat is located in the reference image, and the content presented is the blue cat in the reference image and information such as the shape, posture, and movement of the blue cat. It can be understood that the middle part of the second image contains the image content of the orange cat and the grass. For another example, suppose that the reference image and the similar image both contain the content of "a plate with food and cutlery", but the difference between the reference image and the similar image lies in the placement of the cutlery. Then, after local adjustment of the similar image, the second image obtained only contains the image content of the cutlery in the reference image.

[0204] To obtain a target image that is more similar to the reference image, two stages may be included: image adjustment and image fusion. Specifically, the image adjustment stage may include two parts: difference description text guidance and reference image guidance. The first part may use the difference description text as a guiding condition for image adjustment, combined with the target mask image, to adjust the similar image to obtain a first image, such that the pixel areas corresponding to the difference information in the first image are blank pixels and do not contain image content. The second part may use the reference image as a guiding condition for image adjustment, combined with the target mask image, to adjust the similar image to obtain a second image, such that the second image only contains the image content of the pixel areas corresponding to the difference information. Furthermore, the obtained first image and the second image are superimposed and fused, such that the image content of the pixel areas corresponding to the difference information in the second image is superimposed and filled with the blank pixel areas corresponding to the difference information in the first image, achieving image content complementarity between the first and second images, thereby obtaining a locally fine-tuned target image. Compared to the similar image, this target image better meets the image content requirements of the reference image and is more similar to the reference image.

[0205] It should be noted that when performing local fine-tuning on similar images, this can be achieved through a conditional stable diffusion (SD) model. Specifically, similar images are input into the conditional stable diffusion model, and noise diffusion processing is performed on the similar images through the conditional stable diffusion model. Guiding conditions are introduced to assist in the noise diffusion process to instruct the image to accurately fine-tune the relevant pixel areas, thereby improving the accuracy of image adjustment.

[0206] In some embodiments, step (105.A.1) may include: using a first neural network model to perform local fine-tuning on the similar image based on the target mask image and the difference description text to generate the first image. Step (105.A.2) may include: using a second neural network model to perform local fine-tuning on the similar image based on the target mask image and the reference image to generate the second image. The first neural network model and the second neural network model are both conditional latent space diffusion (SD) models.

[0207] In some embodiments, in order to achieve local fine-tuning of similar images through a conditional latent space diffusion (Stable Diffusion, SD) model, it is necessary to train the conditional latent space diffusion model to obtain a first neural network model and a second neural network model for image fine-tuning, respectively.

[0208] For example, taking the training of the first neural network model as an example, before step (105.A.1), it can also include: obtaining a sample reference image and a sample similar image, as well as a first sample target image; based on the difference information between the sample reference image and the sample similar image, generating a sample target mask map and a sample difference description text of the sample reference image; using a preset model to perform local fine-tuning on the sample similar image based on the sample target mask map and the sample difference description text to generate a first predicted image; determining the prediction loss based on the difference between the first sample target image and the first predicted image; iteratively training the preset model based on the prediction loss until the preset model convergence condition is reached to obtain the first neural network model.

[0209] For another example, before step (105.A.2), it may also include: obtaining a sample reference image and a sample similar image, as well as a second sample target image; generating a sample target mask map of the sample reference image based on the difference information between the sample reference image and the sample similar image, and obtaining a sample target inverse mask map that is opposite to the sample target mask map; locally fine-tuning the sample similar image based on the sample target inverse mask map and the reference image through a preset model to generate a second predicted image; determining a prediction loss based on the difference between the second sample target image and the second predicted image; iteratively training the preset model based on the prediction loss until the preset model convergence condition is reached to obtain a second neural network model.

[0210] In some embodiments, local fine-tuning of similar images primarily involves introducing guiding conditions into the noise diffusion of the images, thereby indicating that local fine-tuning of target areas of the similar images is to be performed during the noise diffusion, thereby improving the accuracy of image adjustment. For example, using difference description text as a guiding condition for image adjustment, and adjusting similar images using a first neural network model, step (105.A.1) may include:

[0211] (105.A.1.1) Masking the similar image according to the target mask image to obtain a first similar mask image;

[0212] (105.A.1.2) Noising the first similar mask image to obtain a first similar noise map;

[0213] (105.A.1.3) Obtain the difference text vector corresponding to the difference description text;

[0214] (105.A.1.4) performing noise reduction on the first similar noise map according to the difference text vector to obtain a first feature map;

[0215] (105.A.1.5) Decode the first feature map to obtain a first image.

[0216] It should be noted that the role of the target mask image is to mask the target pixel area in the similar image at the same position as the area where the difference object is located in the reference image, so as to block the representation of the pixels in the target pixel area in the similar image. Therefore, before adjusting the similar image through the first neural network model, the similar image can first be masked according to the target mask image. The masking process can be to multiply the target mask image with the similar image so that each value in the target mask image is multiplied with the corresponding pixel in the similar image to obtain a first similar mask image. Then, the first similar mask image is input into the first neural network model to perform local fine-tuning on the first similar mask image through the first neural network model. For ease of understanding, the image adjustment process of the first neural network model can be introduced in conjunction with Figure 6. Specifically, the first neural network model encodes the first similar mask image and introduces the encoding result into the latent space. The encoding result is forward diffused in the latent space. The forward diffusion can be understood as a noise addition process to obtain a first similar noise image. Next, before backdiffusion of the first similar noise image, the difference description text serving as the guide condition is encoded to obtain a difference text vector, thereby introducing the difference description text into the latent space. Furthermore, the first similar noise image is backdiffused (denoised) in the latent space, and during the backdiffusion process, the difference text vector is integrated into the noise of the first similar noise image and backdiffused together to obtain a first feature map. Finally, the first feature map is decoded to restore the first feature map in the latent space to obtain the first image.

[0217] It should be noted that when the first similar noise map is locally fine-tuned through the first neural network model, it mainly includes two processes: forward diffusion and backward diffusion. Among them, the forward diffusion is specifically a process of noise-adding the image, which can be understood as a process of gradual noise addition, and the backward diffusion process is a process of denoising the noise map, which can be understood as a process of gradual noise reduction.

[0218] In some embodiments, to perform forward diffusion on similar images, the similar images need to be converted into vectors, imported into a latent space, and then forward diffused in the latent space. For example, taking the forward diffusion of a model guided by difference description text as an example, step (105.A.1.2) may include: encoding the first similarity mask image to obtain an encoded feature map; and performing noise processing on the encoded feature map to obtain a first similarity noise map.

[0219] Specifically, after the first similarity mask image is input into the first neural network model, the first neural network model performs image feature encoding processing on the first similarity mask image in the pixel space to obtain an encoded feature map, which is a vector feature matrix of the first similarity mask image; then, the encoded feature map is transmitted to the latent space, and the encoded feature map is forward diffused in the latent space. The forward diffusion process is a noise processing process, which mainly involves gradually adding noise to the encoded feature map, and after multiple time steps of noise processing, the first similarity mask map is completely noised, thereby obtaining a completely noisy first similarity noise map.

[0220] In some embodiments, after the first similar noise map is obtained through forward noise addition processing, the first similar noise map is subjected to reverse noise reduction processing, and the feature vector corresponding to the guiding condition is incorporated during the noise reduction process to obtain a noise map after noise reduction. For example, taking the difference text vector corresponding to the difference description text as the guiding condition as an example, step (105.A.1.4) may include: performing noise reduction processing on the first similar noise map multiple times in a row, and incorporating the difference text vector into each noise reduction process of the first similar noise map through an attention mechanism to obtain a first feature map. The first feature map contains semantic information associated with the difference text vector.

[0221] Specifically, in order to perform reverse diffusion processing on the completely noisy first similar noise image, it is necessary to perform noise reduction processing on the first similar noise image for multiple rounds (time steps), and add the difference text vector corresponding to the difference description text in each noise reduction process to achieve accurate fine-tuning of the similar image until the first feature map is obtained after noise reduction for a preset number of time steps.

[0222] For example, to facilitate understanding of the noise reduction process of each round, the first round of noise reduction process is taken as an example. Since the first similar noise map is a completely noisy noise map, assuming that it has undergone T time steps of noise addition, the first similar noise map can be represented as "ZT" or "XT". During the first round of noise reduction, the first similar noise map "ZT" is subjected to noise reduction, and the difference text vector is incorporated into the noise reduction process through the attention mechanism. In conjunction with Figure 6, each round of noise reduction is implemented in the denoising network layer (Denoising U-Net), specifically referring to Figure 7. The denoising network layer (Denoising U-Net) structurally includes a residual network layer (ResNet) and an attention module, wherein the residual network layer (ResNet) is mainly used for feature extraction to implement the gradual noise reduction process, and the attention module is used to fuse the feature vector corresponding to the guidance condition (such as the text difference vector) with the noise feature map to achieve indication and guidance in image fine-tuning.

[0223] Specifically, the denoising network layer (Denoising U-Net) can be composed of two residual network layers and two attention modules, and its structure is specifically "residual network layer-attention module-residual network layer-attention module". The following will introduce the denoising process of a denoising round in combination with the specific results of the denoising network layer, as follows: First, the first similar noise image is feature extracted by the first residual network layer, and the first feature result of the feature extraction is transmitted to the first attention module, and the difference text vector is transmitted to the first attention module, and the difference text vector is integrated into the first feature result through the first attention module. For example, the difference text vector can be calculated by the attention mechanism, and the attention calculation result is fused with the first feature result to obtain a first initial fusion result. Further, the first initial fusion result is feature extracted by the second residual network layer to obtain a second feature result, and the second feature result is transmitted to the second attention module, and the difference text vector is transmitted to the second attention module, and the difference text vector is integrated into the second feature result through the second attention module to obtain a first fused noise image.

[0224] According to the above example, after multiple rounds ("T-1" times) of noise reduction processing, the final "T-1" fused noise map is obtained, and the "T-1" fused noise map is the first feature map.

[0225] Furthermore, after obtaining the first feature map, the decoding process of the first feature map can be specifically: decoding the first feature map in the latent space through the decoding module to restore the first feature map to a pixel matrix in the pixel space, so that the first neural network outputs the first image after local fine-tuning.

[0226] In some embodiments, the reference image is used as a guide condition for image adjustment, and the similar image is adjusted using a second neural network model. For example, step (105.A.2) may include: inverting the target mask image to obtain a target inverse mask image corresponding to the target mask image; masking the similar image according to the target inverse mask image to obtain a second similar mask image; performing noise processing on the second similar mask image to obtain a second similar noise image, wherein the time step of the second similar noise image is adjacent to the time step of the first similar noise image; performing noise reduction processing on the second similar noise image according to the feature map corresponding to the reference image to obtain a second feature map; and decoding the second feature map to obtain a second image.

[0227] It should be noted that the purpose of local fine-tuning of the similar image using the reference image as a guiding condition is: after image fine-tuning, a second image containing a representation of the difference object is obtained; since the role of the target mask map is to mask the same target pixel area in the similar image as the area where the difference object is located in the reference image, the target mask map needs to be inverted. The inversion process is to replace the original "0" in the target mask map with "1", and at the same time, replace the original "1" in the target mask map with "0" to obtain a target inverse mask map. The role of the target inverse mask map is to allow the representation of pixels in the target pixel area where the difference object is located, and reject the representation of other pixels in non-target pixel areas.

[0228] Furthermore, after obtaining the target inverse mask image, first, the similar image is masked according to the target inverse mask image. The masking process can be to multiply the target inverse mask image with the similar image so that each value in the target mask image is multiplied by the corresponding pixel in the similar image to obtain a second similar mask image. Then, the second similar mask image is input into the second neural network model to perform local fine-tuning on the second similar mask image through the second neural network model. For ease of understanding, the image adjustment process of the second neural network model can be introduced in conjunction with Figure 6. Specifically, the second neural network model encodes the second similar mask image and introduces the encoding result into the latent space. The encoding result is forward diffused in the latent space. The forward diffusion can be understood as a noise addition process to obtain a second similar noise image. Next, before backdiffusion of the second similar noise image, the reference image serving as the guide condition is subjected to image encoding processing to obtain a feature map (i.e., a vector matrix) corresponding to the reference image, thereby introducing the feature map corresponding to the reference image into the latent space. Furthermore, the second similar noise image is backdiffused in the latent space, and during the backdiffusion process, the feature map corresponding to the reference image is integrated into the noise of the second similar noise image and backdiffused together to obtain a second feature map. Finally, the second feature map is decoded to restore the first feature map in the latent space to the pixel space for representation, thereby outputting the second image.

[0229] It should be noted that when the feature map of the reference image is used as a constraint condition, when the second similar noise map is back-diffused, the specific processing process is the same as the steps of the above-mentioned "back-diffusion processing process of the first similar noise map in the denoising network layer (Denoising U-Net). The only difference is the "feature map of the reference image" and the "difference text vector of the difference description text". The specific back-diffusion process can be understood in conjunction with Figures 6 and 7 and the above content, and will not be described here one by one.

[0230] (B) Masking the image during fine-tuning:

[0231] It should be noted that when performing local fine-tuning on similar images, this can be achieved through a conditional stable diffusion (SD) model. Specifically, the similar images and the target mask map are input into the conditional latent space diffusion model, and the similar images are subjected to noise diffusion processing through the conditional latent space diffusion model. In the noise diffusion process, guiding conditions (such as difference description text and reference image) are introduced to assist in instructing the image to perform precise fine-tuning on the relevant pixel areas, thereby improving the accuracy of image adjustment.

[0232] In some implementations, a similar image to be fine-tuned is first subjected to noise processing for multiple consecutive time steps, two noise images with adjacent time steps are obtained, and the noise images are fine-tuned based on the target mask image, the difference description text, and the reference image, so as to fuse the fine-tuning results to obtain the target image. For example, step 105 may include:

[0233] (105.B.1) Performing noise processing on the similar image and obtaining a first similar noise map and a second similar noise map at adjacent time steps during the noise processing;

[0234] (105.B.2) De-noising the first similar noise image based on the target mask image and the difference description text to obtain a first image;

[0235] (105.B.3) De-noising the second similar noise image based on the target mask image and the reference image to obtain a second image;

[0236] (105.B.4) Fuse the first image with the second image to obtain a target image.

[0237] Specifically, a conditional latent space diffusion (SD) model can be used to achieve local fine-tuning of similar images. The similar images and target masks are transmitted to the diffusion model. The diffusion model encodes the similar images to obtain feature maps, which are then imported into the latent space for forward diffusion. The forward diffusion process is a gradual noise addition process over multiple time steps, with each time step considered as a noise addition process, until a completely noisy noise map is obtained. Then, the completely noisy noise map and the noise map of the adjacent previous time step can be taken. Specifically, the noise map of the adjacent previous time step can be used as the first similar noise map, and the completely noisy noise map can be used as the second similar noise map. Furthermore, based on the target mask map and the difference description text, the first similar noise map is subjected to back diffusion processing. This back diffusion process is a continuous multiple noise reduction process to obtain the first image. The second similar noise map is then subjected to back diffusion processing based on the target mask map and the reference image to obtain the second image. Finally, the first image and the second image are fused to obtain the target image.

[0238] Exemplarily, as shown in Figure 6, the similar image X, the target mask map, the difference description text and the reference image are transmitted to a conditional latent space diffusion (Stable Diffusion, SD) model, wherein the diffusion model encodes the similar image X to import the encoding processing result (feature map Z) into the latent space, and through forward diffusion (noising) processing, it is assumed that through step-by-step noise processing of T time steps, a completely noisy noise map "ZT" is obtained, which can also be expressed as "XT". Thus, the noise map "ZT" is taken as the second similar noise map, and the noise map "ZT-1" of the adjacent time step is taken as the first similar noise map. Furthermore, based on the target mask image and the difference description text, the first similar noise image is subjected to backdiffusion processing, which involves noise reduction processing over a corresponding number of time steps, such as gradually reducing noise over "T-1" time steps, and the first image is decoded. Similarly, the second similar noise image is subjected to backdiffusion processing, which involves noise reduction processing over a corresponding number of time steps, such as gradually reducing noise over "T-1" time steps, to obtain a second feature map "Z", and the second image is decoded. Finally, the first and second images are fused, such as by overlaying or splicing, to obtain the adjusted target image.

[0239] It should be noted that the noise image at the adjacent time step to the first similar noise image is taken as the second similar noise image, and the reference image is introduced into the denoising to guide the denoising, so that the features of the reference image and the similar image at the adjacent time steps "T-1" and "T" can be guaranteed to be consistent, such as the size and dimension, and have reliability.

[0240] In some embodiments, the first image is fine-tuned using the difference description text as a guide condition. For example, step (105.B.2) may include: masking the first similar noise image according to the target mask image to obtain a first masked noise image; obtaining a difference text vector corresponding to the difference description text, and performing noise reduction processing on the first similar noise image according to the difference text vector to obtain a first feature image; and decoding the first feature image to obtain the first image.

[0241] Exemplarily, the conditional latent space diffusion (Stable Diffusion, SD) model obtains the first similar noise map "ZT-1" after multiple noise addition processes, and multiplies the target mask map with the first similar noise map to obtain the first mask noise map; at the same time, using the difference description text as a guiding condition, the difference description text is subjected to text encoding processing to obtain a difference text vector; then, based on the difference text vector, the first mask noise map is gradually denoised. Specifically, during the noise reduction process of each time step, the difference text vector is introduced into the noise of the time step through the attention mechanism, and the noise reduction is continued for "T-1" time steps until it is completely denoised to obtain the first feature map, such as "Z" in Figure 6; finally, the first feature map is decoded to obtain the first image. It should be noted that the above is only an example, and the first feature map can also be masked after complete denoising. The timing of the mask processing process is not specifically limited here.

[0242] In some embodiments, the second image is obtained by fine-tuning the reference image as a guide condition. Step (105.B.3) may include: inverting the target mask image to obtain a target inverse mask image corresponding to the target mask image; masking the second similar noise image according to the target inverse mask image to obtain a second masked noise image; masking the second masked noise image according to the feature image corresponding to the reference image to obtain a second feature image; and decoding the second feature image to obtain the second image.

[0243] In an exemplary embodiment, after a conditional latent space diffusion (SD) model undergoes multiple denoising processes to obtain a second similar noise map "ZT", it is necessary to perform masking on the second similar noise map. Since it is necessary to extract information about the difference objects in the reference image and mask information about other non-difference objects in the reference and similar images, the target mask map needs to be inverted to obtain a target inverse mask map that is the opposite of the target mask map. The target inverse mask map is then multiplied by the second similar noise map to obtain a second masked noise map. Simultaneously, using the reference image as a guide condition, image encoding is performed on the reference image to obtain a feature map of the reference image. Furthermore, based on the feature map of the reference image, the second masked noise map is gradually denoised. Specifically, during the denoising process at each time step, the feature map of the reference image is introduced into the noise at that time step through an attention mechanism. This denoising process is continued for "T" consecutive time steps until complete denoising is achieved, resulting in a second feature map, as shown by "Z" in Figure 6. Finally, this second feature map is decoded to obtain the second image. It should be noted that the above is only an example, and the second feature map may be masked after complete denoising. The timing of the masking process is not specifically limited here.

[0244] In some embodiments, the training process of the conditional stable diffusion (SD) model is specifically as follows: obtaining a sample reference image, a sample similar image, and a sample target image, and obtaining a sample difference description text and a sample target mask map of the sample reference image relative to the sample similar image; then, transmitting the sample reference image, the sample similar image, the sample difference description text, and the sample target mask map to a preset SD model, using the sample difference description text and the sample reference image as guiding conditions, respectively, and locally fine-tuning the sample similar image in combination with the sample target mask map to obtain a predicted target image; then, obtaining the difference between the predicted target image and the sample target image to construct a prediction loss, and iteratively training the preset model based on the prediction loss until the preset model convergence condition is reached, thereby obtaining a trained target model, i.e., a conditional latent space diffusion model.

[0245] From the above, it can be seen that the embodiment of the present application can first obtain a similar image that is similar to the reference image from the existing data, and then, based on the difference information between the reference image and the similar image, generate a target mask map, and expand the difference information to enrich the difference description text representing the difference information. Finally, the existing similar image is locally fine-tuned by combining the target mask, the difference description text and the reference image to obtain the fine-tuned target image; in this way, the difference between the images can be expanded and described, and the expanded description text of the difference and the reference image can be used as constraints to locally adjust the similar image, thereby improving the accuracy of the image adjustment and making the adjusted image effect consistent with the actual needs, so as to facilitate the subsequent development of other businesses.

[0246] The method described in the above embodiment will be further described in detail below with examples.

[0247] The embodiment of the present application takes image processing as an example to further describe the image processing method provided in the embodiment of the present application.

[0248] Figure 8 is a schematic diagram of another step flow of the image processing method provided in an embodiment of the present application, Figure 9 is a schematic diagram of the framework structure of the image processing system provided in an embodiment of the present application, Figure 10 is a schematic diagram of the structure of the residual network layer provided in an embodiment of the present application, Figure 11 is a schematic diagram of a scenario of aggregating difference information to generate difference description text provided in an embodiment of the present application, and Figure 12 is a schematic diagram of a scenario of the image fine-tuning process provided in an embodiment of the present application. For ease of understanding, the embodiments of the present application are described in conjunction with Figures 3-12.

[0249] In the embodiments of the present application, the image processing apparatus will be described from the perspective of an image processing device, which can be integrated into a computer device such as a server. For example, when a processor on the computer device executes a program corresponding to the image processing method, the specific process of the image processing method is as follows:

[0250] 201. Obtain a reference image, obtain the similarity between each image in a preset database and the reference image, and determine the image with the greatest similarity as a similar image similar to the reference image.

[0251] In an embodiment of the present application, in order to obtain a target image that is more similar to a reference image, a similar image that is most similar to the reference image can be searched from a preset database so that further image processing can be performed on the searched similar image, such as performing local fine-tuning on the similar image to obtain a target image that is more consistent with the reference image.

[0252] The reference image can be an image containing any content. For example, in the case of an image query service platform, a client sends one or more example images to the platform, which serve as reference images for embodiments of the present application. Upon receiving the example images, the platform can search for similar images from an existing database, so that the similar images can be used as a basis for local adjustments to the similar images based on the differences between the similar images and the reference images.

[0253] The similar image may be an image in the database that is most similar to the reference image, which may be understood as an image in the existing data that is most similar to the reference image in terms of image content or image style.

[0254] Specifically, in order to select similar images that are similar to the reference image from a preset database, the process of obtaining similar images can be as follows: first, the reference cluster center corresponding to the reference image can be determined, and the feature category distance between the reference cluster center and each image cluster center in the existing database can be calculated respectively. Further, similar images that are similar to the reference image can be selected based on the feature category distance. Specifically, the size of the feature category distance can be used to determine which image cluster center in the database the cluster center of the reference image is closer to, so as to determine the image category of the similar target image cluster center as the image category of the reference cluster center; then, the feature distance between the reference image and each image under the image category of the target image cluster center is calculated. It should be noted that the size of the feature distance can reflect the similarity between any two images. Therefore, similar images that are similar to the reference image can be selected based on the size of the feature distance. For example, the image with the smallest feature distance to the reference image is selected as the similar image.

[0255] 202. Determine difference information between the reference image and the similar image.

[0256] The difference information may be information indicating feature differences between the reference image and the similar image, and is not limited to information including the number of objects (things) with differences in the image, object positions and / or object postures.

[0257] Specifically, in order to obtain the difference information between the reference image and the similar image, the first description text of the reference image can be obtained by image-text conversion, and the second description text of the similar image can be obtained by image-text conversion; further, based on the difference between the first description text and the second description text, the difference information between the reference image and the similar image is generated.

[0258] To facilitate understanding of the first description text and the second description text, the first description text is taken as an example for description. Specifically, the first description text may include a global description text for the overall image content in the reference image and an object description text for each object in the image.

[0259] The global description text generation approach can be implemented based on pre-trained frozen image encoders and large language models (Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models, BLIP2). Specifically, the model can structurally include two parts: vision-and-language representation learning and vision-to-language generative learning. The vision-and-language representation learning structure includes an image encoder and a lightweight query transformer (Q-Former), while the vision-to-language generative learning structure includes large language models (LLM). The global description text generation process is as follows: the reference image is input into the image encoder, and the reference image is encoded by the image encoder to obtain the image encoding result; then, the image encoding result is input into a lightweight query transformer, and the object category label in the reference image is determined, so as to be fused with the object category label in the lightweight query transformer to obtain a fused feature result; finally, the fused feature result is input into a large-scale language model for language processing to output the global description text for the reference image.

[0260] Among them, the object description text generation approach can be implemented based on the Generative Region-to-Text Transformer (GRIT), which can structurally include a visual encoder, a foreground object extractor for locating objects, and a text decoder. Specifically, the object description text generation process is as follows: each object in the reference image is identified to determine the pixel region where each object is located in the reference image, and the category of each object is marked. The reference image with the pixel region and category of each object determined is input into the generative transformer from the image region to text to convert the pixel region between region and language, and output the converted reference image, which contains a marker box for marking each object and the object description text of the object in each marker box.

[0261] In this way, the difference between the reference image and the similar image can be determined, so that the difference between the two images can be used as an adjustment basis in the image processing process, and the similar image can be adjusted to improve the accuracy of image adjustment.

[0262] 203. Determine a target mask map for difference information in the reference image, and obtain a target anti-mask map that is opposite to the target mask map.

[0263] In the embodiment of the present application, local adjustment is mainly used to process similar images. Therefore, it is necessary to obtain a mask map for difference information in the reference image, so as to use the mask map to participate in local fine-tuning of the similar images, so as to subsequently improve the accuracy of image adjustment.

[0264] It should be noted that the target mask map can be a pixel area mask map generated based on the difference information between the reference image and the similar image, and is used to mask the target pixel area in the similar image at the same position as the area where the difference object is located in the reference image, so as to block the representation of the pixels in the target pixel area in the similar image, so that the target pixel area in the similar image appears blank, that is, content. The target mask map is mainly represented by "0" and "1", wherein the value in the target pixel area is "0", and the value of other areas except the target pixel area is "1".

[0265] In addition, the target inverse mask map is a mask map that is the opposite of the target mask map and is used to allow the representation of pixels in the target pixel area where the difference object is located, and reject the representation of other pixels in the non-target pixel area. It can be understood that in the target inverse mask map, the value in the target pixel area is "1", and the value of other areas except the target pixel area is "0".

[0266] In this way, the target mask map and the target inverse mask map for the difference object in the reference image can be obtained respectively as necessary elements when adjusting the similar image, and respectively participate in the local fine-tuning of the similar image to improve the accuracy of the image adjustment later.

[0267] 204. Perform textual expansion based on the difference object and the reference image to obtain a difference description text.

[0268] In an embodiment of the present application, relevant descriptive text for the difference information in the reference image can be obtained, so that the relevant descriptive text of the difference information can be used as a guiding condition for image adjustment to perform local adjustments on similar images and improve the accuracy of image adjustment.

[0269] The difference description text can be a text describing the content differences between the reference image and the similar image. This text is primarily derived from a textual expansion of the difference information, richly and accurately representing the differences between the reference image and the similar image. This difference description text can serve as a guide for adjusting the similar image, instructing local adjustments to the similar image and improving the accuracy of image adjustments.

[0270] Specifically, in order to provide a rich description of the difference information between the reference image and the similar image, first, the difference object corresponding to the difference information in the reference image can be determined, and the object relationship information between the difference object and any other object in the reference image can be obtained, such as determining the positional relationship between the current difference object and other objects in the image, whether they belong to the same object category (object category), etc.; further, referring to the acquisition of the aforementioned "first description text", the global description text of the reference image is obtained, and the object description text associated with the difference object is obtained; finally, based on the global description text of the reference image, the description text of the difference object, and the object relationship information between the difference object and other objects, the text is expanded and described, thereby achieving a full description of the difference information between the reference image and the similar image, and obtaining an expanded difference description text. It should be noted that the difference description text richly expresses the relationship between the difference object and other objects in the reference image and the object state information of the difference object. Therefore, compared with the object description text, the difference description text can more richly represent the relevant information of the difference object in the reference image.

[0271] In the above manner, a text rich in description of the difference information in the reference image can be obtained, so that the difference description text can be subsequently used as a guiding condition to indicate local adjustments to similar images, thereby improving the accuracy of image adjustment.

[0272] It should be noted that when adjusting the local area of ​​a similar image, the adjustment process includes two parts: first, using the difference description text as a guiding condition, combining the target mask image to perform local adjustments on the similar image to obtain a first image, specifically refer to the following step 205; second, using the reference image as a guiding (constraint) condition, combining the target inverse mask image to perform local adjustments on the similar image to obtain a second image, specifically refer to the following step 206.

[0273] 205. Based on the target mask image and the difference description text, the similar image is locally fine-tuned by the first neural network model to generate a first image.

[0274] In an embodiment of the present application, when performing local fine-tuning on similar images, the first neural network model can be a conditional stable diffusion (SD) model. Specifically, the similar images are input into the conditional stable diffusion model, and noise diffusion processing is performed on the similar images through the conditional latent space diffusion model. Guiding conditions are introduced to assist in the noise diffusion process to instruct the image to perform precise fine-tuning on the relevant pixel area, thereby improving the accuracy of image adjustment.

[0275] Specifically, before adjusting the similar image through the latent space diffusion model, the similar image is first masked according to the target mask image. The masking process can be to multiply the target mask image with the similar image so that each value in the target mask image is multiplied by the corresponding pixel in the similar image to obtain a first similar mask image. Then, the first similar mask image is input into the latent space diffusion model to perform local fine-tuning on the first similar mask image through the latent space diffusion model. For ease of understanding, the image adjustment process of the first neural network model can be introduced in conjunction with Figures 6 and 7. Specifically, the latent space diffusion model encodes the first similar mask image and introduces the encoding result (first encoding feature map) into the latent space. The encoding result is forward diffused in the latent space. The forward diffusion can be understood as a process of gradual noise addition over multiple time steps to obtain a completely noisy first similar noise map. Next, before back-diffusion of the first similar noise map, the difference description text serving as the guide condition is encoded to obtain a difference text vector, thereby introducing the difference description text into the latent space. Furthermore, the first similar noise map is back-diffused in the latent space through a denoising network layer (Denoising U-Net). This back-diffusion process can be understood as performing multiple (multiple time-step) denoising operations. During each denoising process, the noise map at the current time step is feature extracted through a residual network layer (ResNet), and the difference text vector is incorporated into the feature vector through an attention mechanism. After denoising for multiple time steps, a first feature map is obtained. Finally, the first feature map is decoded to restore the first feature map in the latent space to pixel space for pixel representation, thereby obtaining the first image.

[0276] The first image obtained after local adjustment according to the difference description text is different from the similar image. Specifically, the pixel area in the first image corresponding to the difference object in the reference image is a blank pixel area, that is, there is no image content.

[0277] 206. Based on the target inverse mask image and the reference image, the similar image is locally fine-tuned by the second neural network model to generate a second image.

[0278] Specifically, the target inverse mask image is first multiplied with the similar image so that each value in the target mask image is multiplied by the corresponding pixel in the similar image, resulting in a second similar mask image. This second similar mask image is then input into a second neural network model, where it is encoded in pixel space to obtain a second encoded feature map. The second encoded feature map is then introduced into the latent space and subjected to a stepwise noise addition process over multiple time steps to obtain a fully noisy second similar noise map. Furthermore, image encoding is performed on a reference image serving as a guide condition to obtain a feature map (i.e., a vector matrix) corresponding to the reference image. The second similar noise map is then subjected to a noise reduction process over multiple time steps in the latent space. During each time step, the feature map of the reference image is repeatedly incorporated into the noise reduction process using an attention mechanism. After these multiple time steps of noise reduction, a second feature map is obtained. Finally, the second feature map is decoded, specifically by restoring the second feature map from the latent space to pixel space for pixel representation to obtain the second image.

[0279] Among them, the second image obtained after local adjustment according to the reference image is different from the similar image. Specifically, in the second image, the pixel area corresponding to the non-difference information in the similar image is a blank pixel area, while the pixel area corresponding to the difference information is not a blank pixel area, that is, the second image only contains the image content of the pixel area corresponding to the difference information.

[0280] It should be noted that, regarding the training process of the first neural network model and the second neural network model, please refer to the description of the aforementioned embodiment for details, which will not be repeated here.

[0281] 207. Fuse the first image and the second image to obtain a target image.

[0282] In an embodiment of the present application, after obtaining a first image adjusted with the difference description text as a guiding condition and a second image adjusted with the reference image as a guiding condition, the obtained first image and the second image can be superimposed and fused so that the image content of the pixel area corresponding to the difference information in the second image is superimposed and filled with the blank pixel area corresponding to the difference information in the first image, thereby achieving image content complementarity between the first image and the second image, so as to obtain a target image after local fine-tuning. Compared with the similar image, the target image is more in line with the image content requirements of the reference image, is more similar to the reference image, and is reliable.

[0283] In some embodiments, steps 205 to 206 may be implemented through the following process: specifically, a conditional latent space diffusion (Stable Diffusion, SD) model may be used to implement local fine-tuning of similar images, wherein the similar images and the target mask are transmitted to the diffusion model. The diffusion model encodes the similar images to obtain feature maps, and imports the feature maps into the latent (latent) space for forward diffusion. The forward diffusion process is a gradual noise addition process that undergoes multiple time steps, where each time step is considered as a noise addition process, until a completely noisy noise map is obtained. Then, the completely noisy noise map and the noise map of the adjacent previous time step may be taken. Specifically, the noise map of the adjacent previous time step may be taken as the first similar noise map, and the completely noisy noise map may be taken as the second similar noise map. Furthermore, based on the target mask map and the difference description text, the first similar noise map is subjected to back diffusion processing, wherein the back diffusion processing process is a continuous multiple noise reduction process to obtain the first image, and based on the target mask map and the reference image, the second similar noise map is subjected to back diffusion processing to obtain the second image. Finally, the first image and the second image are fused to obtain the target image. For the description of this embodiment, please refer to the description of the above embodiment (B), which will not be repeated here.

[0284] To facilitate understanding of the embodiments of the present application, the embodiments of the present application will be described using a specific application scenario example. Specifically, the application scenario example is described by executing the above steps 201-207 and combining Figures 3-12.

[0285] Specifically, this image processing method is mainly used for the scene of local fine-tuning of the image. The scene examples of this image processing are as follows:

[0286] 1. As shown in Figure 9, the framework of the image processing system includes: a training set (preset database), a feature extraction layer of the residual network (ResNet50), a semantic segmentation layer (Segment everything, GRIT, and BLip2), a large-scale language processing model, and a latent space diffusion model (SD) for fine-tuning training.

[0287] For ease of understanding, the image processing process is summarized in combination with the processing layers in Figure 9, as follows:

[0288] (1) Obtain similar images from the training set (the existing set of images) that are similar to the example image. These similar images are images that are highly similar to the example image but may have different or missing key local information (a certain difference object). It should be noted that this process of obtaining similar images can be implemented using the feature extraction layer of the residual network (ResNet50).

[0289] As shown in Figure 10, the ResNet50 serves as the backbone network (basic network), which structurally includes an encoder. The program is a convolutional neural network (CNN). The feature extraction module of the convolutional neural network (CNN) consists of 3 convolutional layers and 6 Resblocks. For the input image (such as the reference image), after three convolutional layers, the width (w) and height (h) of the image are reduced to 1 / 4 of the original, and the number of channels changes from 3 to 128, forming a feature map of w / 4*h / 4*128. This feature map will then pass through a subnetwork composed of 6 ResBlocks to generate a new high-level semantic feature map. Among them, each ResBlock is structurally composed of two convolutional layers and a pass-through (identity) layer. It should be noted that after 6 ResBlocks, the high-level semantic feature map obtained is w*h*c (for example, w / 64*h / 64*1024).

[0290] Specifically, the example image provided by the client is first input into the feature extraction layer of the residual network (ResNet50) to obtain the high-level semantic feature map of the reference image and, based on this high-level semantic information, to obtain the feature mean of the reference image. Similarly, the image data in the training set is traversed in the same manner to obtain the high-level semantic feature map of each image in the training set and its feature mean. Then, the feature dimensions of the high-level semantic feature map of the reference image and the high-level semantic feature map of each image in the training set are reduced, such as through global max pooling, to reduce the feature map dimension to 128.

[0291] Furthermore, we calculate the cluster centers Ac of each category in the training set and the cluster centers Bc of the example images, and use the cluster centers to calculate the similarity between the example images and each category in the training set. The specific formula is as follows:

[0292] Finally, for each example image, determine whether its similarity (pred(B)*similarity(Ac,B)) with the image cluster center Ac of each category in the training set is greater than the similarity between the customer's example image cluster center and the image cluster center of each category in the training set. Images with a similarity greater than this are identified as similar images for subsequent adjustments.

[0293] (2) Obtain the semantic information of the example image (reference image) and similar images respectively to determine the difference information between the two.

[0294] The global description of the reference image is obtained through pre-training of frozen image encoders and large language models (BLIP2). Similarly, the global description of similar images is obtained.

[0295] The object description text of each object in the reference image is obtained through the Generative Region-to-Text Transformer (GRIT). Similarly, the object description text of each object in the similar image is obtained.

[0296] Furthermore, the difference information between the reference image and the similar images may be determined based on the differences in the global description text and the object description text between the reference image and the similar images.

[0297] In addition, a mask segmentation model (seg everything) is used to obtain a target mask map for the object corresponding to the difference information.

[0298] (3) As shown in FIG11 , the difference information is transmitted to a large-scale language processing model for summary processing, and the model is guided to generate a key description of the difference information (i.e., difference description text). Specifically, the large-scale language processing model is allowed to infer the relationship between objects in the image (objects in the image) and the information of the objects to obtain high-quality text for the difference objects, i.e., difference description text. It should be noted that the large-scale language processing model can be any type of language processing model, such as "Chat-Gpt", which is not limited here.

[0299] (4) Local fine-tuning of similar images mainly includes two parts. Specifically, first, local fine-tuning of similar images using the difference description text as prompt information; second, local fine-tuning of similar images using the reference image as prompt information. The detailed introduction of image fine-tuning is as follows:

[0300] Figure 12 shows a scenario diagram of the image fine-tuning process. Specifically, the goal of image fine-tuning is to use a similar image as the input image for the upstream process and, based on the example image provided by the downstream customer, adjust the similar image to a target image that is more similar to the example image. As shown in the figure, the target image does not contain the person's hands, and the knife and fork are placed on the plate, making it more similar to the example image.

[0301] First, similar images are input into the on-line image processing and subjected to noise diffusion. Diffusion (e.g., denoising or noise reduction) introduces a text description of the differences as a prompt, which can be understood as a guiding condition. It should be noted that during noise diffusion, a noise diffusion map at a random time step (defined as the first noise map) can be taken and combined with the target mask map to obtain a preliminary image related to the example image through reverse diffusion, namely the first image. It should be noted that the on-line diffusion process can be expressed as follows:

[0302] Then, the lower path performs noise diffusion on similar images in the same manner as the upper path, and takes the noise diffusion map of the next time step relative to the noise diffusion map of the upper path (defined as the second noise map), and uses the example image (reference image) as a constraint in the noise reduction diffusion, combines the second noise map with the inverse mask map of the target mask map (mask), and obtains the second image after denoising. It should be noted that two noise maps with adjacent time steps can be selected as the first noise map and the second noise map, respectively, so as to ensure that the features of the first image obtained by the upper path and the second image obtained by the lower path can be consistent, such as the same image size, so that the first image and the second image can be accurately fused later. It should be noted that the lower path diffusion process can be expressed as follows:

[0303] Finally, the first image is fused with the second image to obtain the target image. The image fusion process can be expressed as follows:

[0304] By executing the above scenario steps (1) to (4), the following can be achieved: find pictures that are relatively similar to the customer data in the training set, and then input the customer example pictures into multiple large models (segment everything, blip2, grit, etc.) to output the corresponding prompt (picture description). Then, use chatgpt to describe these descriptive words, and focus on guiding chatgpt to add details to several key features identified by humans, so as to obtain the final descriptive words; finally, use the SD model to focus on the differences between similar images and the example pictures provided by the customer, so that the image can be partially regenerated to obtain the target image.

[0305] Through the above application scenario examples, the following effects can be achieved: using the differences between the example image and similar images as the guiding condition, and performing local fine-tuning on a small number of images provided by the customer, the accuracy of image fine-tuning is improved to obtain a target image that is more consistent with the example image.

[0306] From the above, it can be seen that the embodiment of the present application can first obtain a similar image similar to the reference image from the existing data, and then, based on the difference information between the reference image and the similar image, generate a target mask map, and expand the difference information to enrich the difference description text representing the difference information. Finally, the existing similar image is locally fine-tuned by combining the target mask, the difference description text and the reference image to obtain the fine-tuned target image; in this way, the difference between the images can be expanded and described, and the expanded description text of the difference and the reference image can be used as constraints to locally adjust the similar image, thereby improving the accuracy of the image adjustment and making the adjusted image effect consistent with the actual needs, so as to facilitate the subsequent development of other businesses.

[0307] To better implement the above method, the present application also provides an image processing device. For example, as shown in FIG13 , the image processing device may include an acquisition unit 401 , a first determination unit 402 , a second determination unit 403 , an expansion unit 404 , and an adjustment unit 405 .

[0308] An acquiring unit 401 is configured to acquire a reference image, obtain the similarity between each image in a preset database and the reference image, and determine the image with the greatest similarity as a similar image to the reference image;

[0309] A first determining unit 402 is configured to determine difference information between a reference image and a similar image;

[0310] The second determining unit 403 is configured to determine a target mask map for the difference information in the reference image; the target mask map is configured to highlight the difference object in the reference image relative to the similar image determined based on the difference information;

[0311] An expansion unit 404 is configured to perform text expansion based on the difference object and the reference image to obtain a difference description text; the difference description text is used to describe the content difference between the reference image and the similar image;

[0312] The adjustment unit 405 is configured to perform local adjustments on the similar images according to the target mask image, the difference description text and the reference image to obtain a target image; the target image is an image that meets the content requirements of the reference image.

[0313] In some embodiments, the adjustment unit 405 is further used to: perform noise processing on the similar image, and obtain a first similar noise map and a second similar noise map at adjacent time steps in the noise processing; perform denoising processing on the first similar noise map according to the target mask map and the difference description text to obtain a first image; perform denoising processing on the second similar noise map according to the target mask map and the reference image to obtain a second image; and fuse the first image with the second image to obtain a target image.

[0314] In some embodiments, the adjustment unit 405 is further used to: mask the first similar noise image according to the target mask image to obtain a first mask noise image; obtain the difference text vector corresponding to the difference description text, and perform noise reduction processing on the first similar noise image according to the difference text vector to obtain a first feature image; and decode the first feature image to obtain a first image.

[0315] In some embodiments, the adjustment unit 405 is further used to: invert the target mask image to obtain a target inverse mask image corresponding to the target mask image; mask the second similar noise image according to the target inverse mask image to obtain a second mask noise image; perform noise reduction processing on the second mask noise image according to the feature image corresponding to the reference image to obtain a second feature image; and decode the second feature image to obtain a second image.

[0316] In some embodiments, the adjustment unit 405 is further used to: perform local fine-tuning on the similar image according to the target mask image and the difference description text to obtain a first image; perform local fine-tuning on the similar image according to the target mask image and the reference image to obtain a second image; and fuse the first image with the second image to obtain a target image.

[0317] In some embodiments, the adjustment unit 405 is further used to: mask the similar image according to the target mask image to obtain a first similar mask image; perform noise addition processing on the first similar mask image to obtain a first similar noise image; obtain a difference text vector corresponding to the difference description text; perform noise reduction processing on the first similar noise image according to the difference text vector to obtain a first feature image; and perform decoding processing on the first feature image to obtain a first image.

[0318] In some embodiments, the noise reduction process is a continuous multiple noise reduction process; the adjustment unit 405 is also used to: incorporate the difference text vector into the noise reduction process of the first similar noise image each time through the attention mechanism to obtain a first feature map; the first feature map contains semantic information associated with the difference text vector.

[0319] In some embodiments, the adjustment unit 405 is further configured to: perform encoding processing on the first similar mask image to obtain an encoded feature map; and perform noise processing on the encoded feature map to obtain a first similar noise map.

[0320] In some embodiments, the adjustment unit 405 is further configured to: perform local fine-tuning on the similar image based on the target mask image and the difference description text using the first neural network model to generate the first image;

[0321] The image processing device also includes a training unit, which is used to: obtain a sample reference image and a sample similar image, as well as a first sample target image; generate a sample target mask map and a sample difference description text of the sample reference image based on the difference information between the sample reference image and the sample similar image; perform local fine-tuning on the sample similar image based on the sample target mask map and the sample difference description text through a preset model to generate a first predicted image; determine the prediction loss based on the first sample target image and the first predicted image; iteratively train the preset model based on the prediction loss until the preset model convergence condition is reached to obtain a first neural network model.

[0322] In some embodiments, the adjustment unit 405 is further used to: invert the target mask image to obtain a target inverse mask image corresponding to the target mask image; mask the similar image according to the target inverse mask image to obtain a second similar mask image; perform noise addition processing on the second similar mask image to obtain a second similar noise image; perform noise reduction processing on the second similar noise image according to the feature image corresponding to the reference image to obtain a second feature image; and decode the second feature image to obtain a second image.

[0323] In some embodiments, the expansion unit 404 is further used to: determine the difference object of the reference image relative to the similar image based on the difference information; determine the object relationship information between the difference object and each object in the reference image; obtain the global description text of the reference image and the target object description text of the difference object; and perform text expansion based on the global description text, the target object description text and the object relationship information to obtain the difference description text.

[0324] In some embodiments, the acquisition unit 401 is further used to: determine the reference cluster center to which the reference image belongs; determine the feature category distance between each image cluster center pre-constructed in a preset database and the reference cluster center; based on the feature category distance, determine the similarity between the image corresponding to each image cluster center and the reference image, and determine the image with the greatest similarity as a similar image similar to the reference image.

[0325] In some embodiments, the first determining unit 402 is further configured to: obtain a first description text corresponding to the reference image; obtain a second description text corresponding to the similar image; and generate difference information based on the difference between the first description text and the second description text.

[0326] In some embodiments, the first determination unit 402 is further used to: globally describe the reference image through a first preset model to generate a global description text of the reference image; process the pixel area where each object in the reference image is located through a second preset model to obtain an object description text corresponding to each object in the reference image; and determine the first description text corresponding to the reference image based on the global description text of the reference image and the object description text corresponding to each object.

[0327] In some embodiments, the second determining unit 403 is further configured to: determine a different object in the reference image relative to the similar image according to the difference information; and generate a target mask image based on the different object in the reference image.

[0328] From the above, it can be seen that the embodiment of the present application can obtain similar images that are similar to the reference image from existing data, and then, based on the difference information between the reference image and the similar image, generate a target mask map, and expand the difference information to enrich the difference description text representing the difference information. Finally, the existing similar image is locally fine-tuned by combining the target mask, the difference description text and the reference image to obtain the fine-tuned target image; in this way, the difference between the images can be expanded and described, and the expanded description text and the reference image for the difference can be used as constraints to locally adjust the similar image, thereby improving the accuracy of the image adjustment and making the adjusted image effect consistent with the actual needs, so as to facilitate the subsequent development of other businesses.

[0329] The present application also provides a computer device, as shown in FIG14 , which shows a schematic diagram of the structure of the computer device involved in the present application embodiment. Specifically:

[0330] The computer device may include components such as a processor 501 with one or more processing cores, a memory 502 with one or more computer-readable storage media, a power supply 503, and an input unit 504. Those skilled in the art will appreciate that the computer device structure shown in FIG14 does not limit the computer device and may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0331] Processor 501 is the control center of the computer device. It connects all components of the computer device using various interfaces and circuits. It executes software programs and / or modules stored in memory 502 and accesses data stored in memory 502 to perform various computer functions and process data. Optionally, processor 501 may include one or more processing cores. Preferably, processor 501 integrates an application processor and a modem processor. The application processor primarily handles the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 501.

[0332] The memory 502 can be used to store software programs and modules. The processor 501 executes various functional applications and image processing processes by running the software programs and modules stored in the memory 502. The memory 502 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 502 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory 502 may also include a memory controller to provide the processor 501 with access to the memory 502.

[0333] The computer device also includes a power supply 503 for supplying power to various components. Preferably, the power supply 503 can be logically connected to the processor 501 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The power supply 503 can also include one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0334] The computer device may further include an input unit 504 , which may be configured to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0335] Although not shown, the computer device may further include a display unit, etc., which will not be described in detail here. Specifically, in the embodiment of the present application, the processor 501 in the computer device will load the executable files corresponding to the processes of one or more application programs into the memory 502 according to the following instructions, and the processor 501 will run the application programs stored in the memory 502 to implement various functions as follows:

[0336] Obtain a reference image, obtain the similarity between each image in a preset database and the reference image, and determine the image with the greatest similarity as a similar image similar to the reference image; determine the difference information between the reference image and the similar image; determine a target mask map for the difference information in the reference image; the target mask map is used to highlight the difference object in the reference image determined according to the difference information relative to the similar image; perform textual expansion based on the difference object and the reference image to obtain a difference description text; the difference description text is used to describe the content difference between the reference image and the similar image; perform local adjustment on the similar image according to the target mask map, the difference description text and the reference image to obtain a target image; the target image is an image that meets the content requirements of the reference image.

[0337] The specific implementation of the above operations can be found in the previous embodiments and will not be described in detail here.

[0338] It can be seen that this solution can obtain similar images that are similar to the reference image from existing data, and then generate a target mask map based on the difference information between the reference image and the similar image, and expand the difference information to enrich the difference description text representing the difference information. Finally, the existing similar image is locally fine-tuned by combining the target mask, the difference description text and the reference image to obtain the fine-tuned target image; in this way, the difference between the images can be expanded and described, and the similar images can be locally adjusted based on the expanded description text of the difference and the reference image as constraints, thereby improving the accuracy of the image adjustment and making the adjusted image effect consistent with the actual needs, so as to facilitate the subsequent development of other businesses.

[0339] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be accomplished by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0340] To this end, an embodiment of the present application provides a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute the steps of any of the image processing methods provided in the embodiments of the present application. For example, the instructions can execute the following steps:

[0341] Obtain a reference image and obtain a similar image that is similar to the reference image; determine difference information between the reference image and the similar image; determine a target mask map for the difference information in the reference image; expand based on the difference information to obtain a difference description text; locally adjust the similar image according to the target mask map, the difference description text and the reference image to obtain a target image.

[0342] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0343] The computer-readable storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0344] Since the instructions stored in the computer-readable storage medium can execute the steps in any image processing method provided in the embodiments of the present application, the beneficial effects that can be achieved by any image processing method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0345] According to one aspect of the present application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations provided in the above embodiments.

[0346] The above is a detailed introduction to an image processing method, device, equipment and computer-readable storage medium provided in the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An image processing method, characterized in that: include: Acquire a reference image, acquire the similarity between each image in a preset database and the reference image, and determine the image with the greatest similarity as a similar image similar to the reference image; Determining difference information between the reference image and the similar image; Determining a target mask map for the difference information in the reference image; the target mask map is used to highlight the difference object in the reference image determined according to the difference information relative to the similar image; Based on the difference object and the reference image, a textual expansion expression is performed to obtain a difference description text; the difference description text is used to describe the content difference between the reference image and the similar image; According to the target mask image, the difference description text and the reference image, the similar image is locally adjusted to obtain a target image; the target image is an image that meets the content requirements of the reference image.

2. The method according to claim 1, characterized in that The step of locally adjusting the similar image according to the target mask image, the difference description text and the reference image to obtain the target image includes: Performing noise processing on the similar image, and obtaining a first similar noise image and a second similar noise image at adjacent time steps in the noise processing; According to the target mask image and the difference description text, performing denoising processing on the first similar noise image to obtain a first image; According to the target mask image and the reference image, performing denoising processing on the second similar noise image to obtain a second image; The first image is fused with the second image to obtain a target image.

3. The method according to claim 2, characterized in that The step of performing denoising on the first similar noise image according to the target mask image and the difference description text to obtain a first image includes: Performing mask processing on the first similar noise image according to the target mask image to obtain a first mask noise image; Obtaining a difference text vector corresponding to the difference description text, and performing noise reduction processing on the first similar noise graph according to the difference text vector to obtain a first feature graph; The first feature map is decoded to obtain a first image.

4. The method according to claim 2, characterized in that: The step of performing denoising processing on the second similar noise image according to the target mask image and the reference image to obtain a second image includes: Inverting the target mask image to obtain a target inverse mask image corresponding to the target mask image; performing mask processing on the second similar noise image according to the target inverse mask image to obtain a second mask noise image; performing noise reduction processing on the second mask noise map according to the feature map corresponding to the reference image to obtain a second feature map; The second feature map is decoded to obtain a second image.

5. The method according to claim 1, characterized in that According to the target mask image, the difference description text and the reference image, the similar image is locally adjusted to obtain a target image, including: According to the target mask image and the difference description text, locally fine-tune the similar image to obtain a first image; According to the target mask image and the reference image, locally fine-tuning the similar image to obtain a second image; The first image is fused with the second image to obtain a target image.

6. The method according to claim 5, characterized in that The method of performing local fine-tuning on the similar image according to the target mask image and the difference description text to obtain a first image includes: Performing mask processing on the similar image according to the target mask image to obtain a first similar mask image; Performing noise processing on the first similar mask image to obtain a first similar noise image; Obtain a difference text vector corresponding to the difference description text; Performing noise reduction processing on the first similar noise graph according to the difference text vector to obtain a first feature graph; The first feature map is decoded to obtain a first image.

7. The method according to claim 6, characterized in that The denoising process is a continuous multiple denoising process; the denoising process is performed on the first similar noise graph according to the difference text vector to obtain a first feature graph, including: The difference text vector is incorporated into the process of each denoising process on the first similar noise map through an attention mechanism to obtain a first feature map; the first feature map contains semantic information associated with the difference text vector.

8. The method according to claim 6, characterized in that The step of performing noise processing on the first similar mask image to obtain a first similar noise map includes: Performing encoding processing on the first similar mask image to obtain an encoding feature map; Noise processing is performed on the encoding feature map to obtain a first similar noise map.

9. The method according to claim 5, characterized in that The method of performing local fine-tuning on the similar image according to the target mask image and the difference description text to obtain a first image includes: Based on the target mask image and the difference description text, the similar image is locally fine-tuned by a first neural network model to generate the first image; Before locally fine-tuning the similar image based on the target mask image and the difference description text by using the first neural network model to generate the first image, the method further includes: Acquire a sample reference image, a sample similar image, and a first sample target image; Based on the difference information between the sample reference image and the sample similar image, generating a sample target mask image and a sample difference description text of the sample reference image; Based on the sample target mask image and the sample difference description text, the sample similar image is locally fine-tuned by a preset model to generate a first predicted image; constructing a prediction loss based on a difference between the first sample target image and the first predicted image; The preset model is iteratively trained based on the prediction loss until a preset model convergence condition is reached to obtain a trained first neural network model.

10. The method according to claim 6, characterized in that The step of locally fine-tuning the similar image according to the target mask image and the reference image to obtain a second image includes: Inverting the target mask image to obtain a target inverse mask image corresponding to the target mask image; Performing mask processing on the similar image according to the target inverse mask image to obtain a second similar mask image; Performing noise processing on the second similar mask image to obtain a second similar noise map, wherein a time step of the second similar noise map is adjacent to a time step of the first similar noise map; Performing noise reduction processing on the second similar noise map according to the feature map corresponding to the reference image to obtain a second feature map; The second feature map is decoded to obtain a second image.

11. The method according to claim 10, characterized in that The denoising process is a continuous multiple denoising process; the denoising process is performed on the second similar noise map according to the feature map corresponding to the reference image to obtain the second feature map, including: The feature map corresponding to the reference image is integrated into the denoising process of the second similar noise map each time through the attention mechanism to obtain a second feature map; the second feature map contains semantic information associated with the feature map corresponding to the reference image.

12. The method according to claim 1, characterized in that The step of performing textual expansion based on the difference object and the reference image to obtain a difference description text includes: Determining, according to the difference information, a difference object between the reference image and the similar image; determining object relationship information between the difference object and each object in the reference image; Acquire a global description text of the reference image and a target object description text of the difference object; A text description is performed based on the global description text, the target object description text and the object relationship information to obtain a difference description text.

13. The method according to claim 1, characterized in that The obtaining of the similarity between each image in the preset database and the reference image, and determining the image with the greatest similarity as a similar image similar to the reference image, comprises: Determining a reference cluster center to which the reference image belongs; Determine the feature category distance between each image cluster center pre-constructed in a preset database and the reference cluster center; Based on the feature category distance, the similarity between the image corresponding to each image cluster center and the reference image is determined, and the image with the maximum similarity is determined as a similar image similar to the reference image.

14. The method according to claim 1, characterized in that The determining the difference information between the reference image and the similar image comprises: Obtaining a first description text corresponding to the reference image; Obtaining a second description text corresponding to the similar image; Based on the difference between the first description text and the second description text, difference information is generated.

15. The method according to claim 14, characterized in that The obtaining of a first description text corresponding to the reference image includes: Performing a global description of the reference image by using a first preset model to generate a global description text of the reference image; Processing the pixel area where each object in the reference image is located by a second preset model to obtain object description text corresponding to each object in the reference image; According to the global description text of the reference image and the object description text corresponding to each object, a first description text corresponding to the reference image is determined.

16. The method according to claim 1, characterized in that The determining a target mask map for the difference information in the reference image includes: Determining, based on the difference information, a difference object in the reference image relative to the similar image; Based on the difference objects in the reference image, a target mask map is generated.

17. An image processing device, characterized in that: include: An acquisition unit, configured to acquire a reference image, acquire the similarity between each image in a preset database and the reference image, and determine an image with the greatest similarity as a similar image similar to the reference image; A first determining unit, configured to determine difference information between the reference image and the similar image; A second determining unit is used to determine a target mask map for the difference information in the reference image; the target mask map is used to highlight the difference object in the reference image determined according to the difference information relative to the similar image; An expansion unit, configured to perform text expansion based on the difference object and the reference image to obtain a difference description text; the difference description text is used to describe the content difference between the reference image and the similar image; The adjustment unit is used to locally adjust the similar image according to the target mask image, the difference description text and the reference image to obtain a target image; the target image is an image that meets the content requirements of the reference image.

18. A computer device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program, and the processor is used to execute the computer program in the memory to implement the steps of the image processing method according to any one of claims 1 to 16.

19. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the image processing method according to any one of claims 1 to 16.

Images