Image processing method and device

Abstract

The embodiment of the invention provides an image processing method and device.The image processing method comprises the steps that in response to an editing instruction for an original image, the original image is updated into an edited image through an image editing model; inputting the original image and the edited image into an image evaluation model for processing to obtain image evaluation information corresponding to a plurality of image evaluation dimensions; dividing the editing image into an editing area and a non-editing area, and determining editing evaluation information corresponding to the editing area and non-editing evaluation information corresponding to the non-editing area; and according to the region information of the editing region, fusing the image evaluation information, the editing evaluation information and the non-editing evaluation information to obtain target evaluation information corresponding to the editing image.

Description

TECHNICAL FIELD

[0001] Embodiments of the present specification relate to the technical field of image processing, in particular to an image processing method and device. BACKGROUND

[0002] With the development of multi-modal large language models (MLLMs), the technology of editing images through natural language instructions has made significant progress. Users can input natural language instructions for images while uploading images, so that multi-modal large language models can edit the uploaded images based on the natural language instructions, such as text editing in images, object material replacement in images, local color adjustment, style transfer, background adjustment, perspective transformation, etc., which can all be achieved through multi-modal large language models. However, how to comprehensively and accurately evaluate the effects of these editing models so that the evaluation results align with users' subjective feelings is still a great challenge. Existing evaluation benchmarks have obvious limitations, covering limited editing types; and the evaluation dimensions are single, which can easily ignore key factors in images, and the perception of image details is rough, which cannot identify logical fallacies or minor flaws. In addition, there is a systematic deviation between the automatic evaluation indicators and users' subjective aesthetics, resulting in a virtual high score that cannot truly reflect the evaluation information of the image, so an effective solution is urgently needed to solve the above problems. SUMMARY

[0003] Therefore, the embodiments of the present specification provide an image processing method. One or more embodiments of the present specification also relate to an image processing device, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defects in the prior art.

[0004] According to a first aspect of the embodiments of the present specification, an image processing method is provided, comprising: updating an original image to an edited image using an image editing model in response to an editing instruction for the original image; inputting the original image and the edited image into an image evaluation model for processing to obtain image evaluation information corresponding to a plurality of image evaluation dimensions respectively; dividing the edited image into an edited region and a non-edited region, and determining edited evaluation information corresponding to the edited region and non-edited evaluation information corresponding to the non-edited region; fusing the image evaluation information, the edited evaluation information, and the non-edited evaluation information according to the region information of the edited region to obtain target evaluation information corresponding to the edited image.

[0005] According to a second aspect of the embodiments of the present specification, another image processing method is provided, applied to a server, comprising: receiving an editing instruction submitted by a client and carrying an original image, and updating the original image to an edited image by using an image editing model in response to the editing instruction; inputting the original image and the edited image into an image evaluation model for processing to obtain image evaluation information corresponding to a plurality of image evaluation dimensions respectively; dividing the edited image into an edited region and a non-edited region, and determining edited evaluation information corresponding to the edited region and non-edited evaluation information corresponding to the non-edited region; fusing the image evaluation information, the edited evaluation information and the non-edited evaluation information according to region information of the edited region to obtain target evaluation information corresponding to the edited image; updating the edited image to a target image based on the target evaluation information, and sending the target image to the client.

[0006] According to a third aspect of the embodiments of the present specification, an image processing apparatus is provided, comprising: an updating module configured to update an original image to an edited image by using an image editing model in response to an editing instruction for the original image; a processing module configured to input the original image and the edited image into an image evaluation model for processing to obtain image evaluation information corresponding to a plurality of image evaluation dimensions respectively; a determining module configured to divide the edited image into an edited region and a non-edited region, and determine edited evaluation information corresponding to the edited region and non-edited evaluation information corresponding to the non-edited region; a fusing module configured to fuse the image evaluation information, the edited evaluation information and the non-edited evaluation information according to region information of the edited region to obtain target evaluation information corresponding to the edited image.

[0007] According to a fourth aspect of the embodiments of the present specification, another image processing apparatus is provided, applied to a server, comprising: a receiving module configured to receive an editing instruction submitted by a client and carrying an original image, and update the original image to an edited image by using an image editing model in response to the editing instruction; a processing module configured to input the original image and the edited image into an image evaluation model for processing to obtain image evaluation information corresponding to a plurality of image evaluation dimensions respectively; a dividing module configured to divide the edited image into an edited region and a non-edited region, and determine edited evaluation information corresponding to the edited region and non-edited evaluation information corresponding to the non-edited region; a fusion module configured to fuse the image evaluation information, the editing evaluation information and the non-editing evaluation information according to the region information of the editing region, to obtain target evaluation information corresponding to the edited image; a sending module configured to update the edited image to a target image based on the target evaluation information, and send the target image to the client.

[0008] According to a fifth aspect of the embodiments of the present specification, a computing device is provided, comprising: a memory and a processor; The memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions, and the computer executable instructions, when executed by the processor, implement the steps of the image processing method.

[0009] According to a sixth aspect of the embodiments of the present specification, a computer readable storage medium is provided, which stores computer executable instructions, and the instructions, when executed by a processor, implement the steps of the image processing method.

[0010] According to a seventh aspect of the embodiments of the present specification, a computer program product is provided, comprising a computer program or instructions, and the computer program or instructions, when executed by a processor, implement the steps of the image processing method.

[0011] The image processing method provided by the embodiment can accurately evaluate the image processed by the image editing model. The original image can be updated to an edited image by using an image editing model in response to an editing instruction for the original image. After obtaining the edited image, the original image and the edited image can be input into an image evaluation model for processing. The image evaluation model can perform basic evaluation operations on the images from multiple image evaluation dimensions, and then obtain image evaluation information corresponding to each image evaluation dimension. The evaluation result of the model is only associated with the set multiple image evaluation dimensions. In order to improve the strictness of the evaluation result, the edited image can be divided into an editing region and a non-editing region, and local evaluation can be performed on the editing region and the non-editing region. Then, editing evaluation information corresponding to the editing region and non-editing evaluation information corresponding to the non-editing region can be determined. On this basis, the image evaluation information, the editing evaluation information and the non-editing evaluation information can be fused in combination with the region information of the editing region to determine the target evaluation information corresponding to the edited image, and to implement comprehensive evaluation of multiple evaluation information. When the image is evaluated and processed, the editing effect can be automatically evaluated from multiple dimensions and multiple angles, thereby effectively improving the evaluation accuracy, so as to use accurate evaluation results for subsequent service processing. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1is a flowchart of an image processing method provided by one embodiment of the present specification; Figure 2 is a schematic diagram of an image in an image processing method provided by one embodiment of the present specification; Figure 3 is a flowchart of another image processing method provided by one embodiment of the present specification; Figure 4 is a flowchart of a processing procedure of an image processing method provided by one embodiment of the present specification; Figure 5 is a structural schematic diagram of an image processing apparatus provided by one embodiment of the present specification; Figure 6 is a structural schematic diagram of another image processing apparatus provided by one embodiment of the present specification; Figure 7 is a structural block diagram of a computing device provided by one embodiment of the present specification. DETAILED DESCRIPTION

[0013] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present specification. However, the present specification can be practiced without the specific details, other than in the examples, set forth in this description. Those skilled in the art, in light of the description, can implement the present specification without limiting to the specific details disclosed in this description.

[0014] The terminology used in one or more embodiments of the present specification is for the purpose of describing particular embodiments only and is not intended to be limiting of one or more embodiments of the present specification. As used in one or more embodiments of the present specification and the accompanying claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in one or more embodiments of the present specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0015] It will be understood that, although the terms first, second, etc. can be used herein to describe various information, these terms are not intended to denote a temporal or chronological order. Rather, these terms are used solely to distinguish one from another only. For example, without departing from the scope of one or more embodiments of the present specification, first can be termed second, and similarly, second can be termed first. The term "if' as used herein, can be interpreted as meaning "when" or "upon" or "in response to determining," depending on the context.

[0016] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0017] The technical solutions provided in this application can employ deep learning models with relatively large parameter scales. However, this large model is merely an example; this application does not limit the number of model parameters supported by the deep learning model used, aiming to meet actual needs. The deep learning models involved in this application can be artificial intelligence-based language models (LM) or multimodal models (MM).

[0018] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0019] A multimodal large language model (MLLM) is an artificial intelligence model that can receive and understand multiple types of input information (such as text, images, audio, video, etc.) and generate coherent and relevant text output.

[0020] GroundingSAM is an advanced vision technology framework that combines GroundingDINO and Segment Anything Model (SAM) for image instance segmentation of open-vocabulary.

[0021] To address the aforementioned technical problems, this specification provides an image processing method. This specification also relates to an image processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0022] The image processing method provided in this embodiment, in order to accurately evaluate the image processed by the image editing model, can first respond to the editing instruction for the original image and update the original image into an edited image using the image editing model. After obtaining the edited image, the original image and the edited image can be input into the image evaluation model for processing. The image evaluation model can perform basic evaluation operations on the image from multiple image evaluation dimensions, thereby obtaining image evaluation information corresponding to each image evaluation dimension. The evaluation result of the model is only associated with the set multiple image evaluation dimensions. In order to improve the rigor of the evaluation result, the edited image can be divided into edited regions and non-edited regions, and local evaluation can be performed on the edited regions and non-edited regions to determine the editing evaluation information corresponding to the edited regions and the non-editing evaluation information corresponding to the non-edited regions. On this basis, the image evaluation information, editing evaluation information and non-editing evaluation information can be fused by combining the regional information of the edited region to achieve the determination of the target evaluation information corresponding to the edited image by comprehensively considering multiple evaluation information. This enables automated multi-dimensional and multi-angle evaluation of the editing effect when evaluating the image, thereby effectively improving the evaluation accuracy so that the accurate evaluation results can be used for subsequent service processing.

[0023] See Figure 1 , Figure 1 A flowchart of an image processing method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0024] Step S102: In response to the editing instruction for the original image, the original image is updated to an edited image using an image editing model.

[0025] The image processing method provided in this embodiment can be applied to scenarios where the edited image is evaluated after outputting an edited image from any image editing model. For example, after evaluating the edited image, the image can be optimized to provide the user with a high-quality target image; or after evaluating the edited image, the image editing model can be optimized based on the evaluation results to enable the image editing model to have more accurate image editing capabilities; or after evaluating the edited image, sample pairs for training the image editing model can be constructed based on the evaluation results. This embodiment does not limit the application scenarios of the image processing method.

[0026] Specifically, the original image refers to the image that needs to be processed by the image editing model at the current moment. The image editing model's editing operations on the original image can be determined according to the editing instructions. The image editing model specifically refers to a multimodal large language model, and the editing instructions specifically refer to the natural language instructions used when editing the original image, which are input by the user according to actual needs. The edited image specifically refers to the image obtained after the image editing model processes the original image according to the editing instructions.

[0027] Based on this, in order to accurately evaluate the image processed by the image editing model, we can first respond to the editing instructions for the original image and use the image editing model to update the original image into an edited image. After obtaining the edited image, we can first input the original image and the edited image into the image evaluation model for processing. The image evaluation model can perform basic evaluation operations on the image from multiple image evaluation dimensions, thereby obtaining the image evaluation information corresponding to each image evaluation dimension. The evaluation results of the model are only associated with the set multiple image evaluation dimensions. In order to improve the rigor of the evaluation results, the edited image can be divided into edited regions and non-edited regions, and local evaluation can be performed on the edited regions and non-edited regions. This can determine the editing evaluation information corresponding to the edited region and the non-editing evaluation information corresponding to the non-edited region. On this basis, we can combine the regional information of the edited region to fuse the image evaluation information, editing evaluation information and non-editing evaluation information, so as to determine the target evaluation information corresponding to the edited image by comprehensively considering multiple evaluation information. This enables automated multi-angle evaluation of the editing effect when evaluating the image, thereby effectively improving the evaluation accuracy, so as to use the accurate evaluation results for subsequent service processing.

[0028] In practical applications, editing commands include, but are not limited to, text editing commands, material editing commands, local color adjustment commands, mask or graffiti area editing commands, style transfer commands, background adjustment commands, atmosphere and color editing commands, perspective change commands, theme addition commands, theme deletion commands, theme customization commands, character retouching commands, action editing commands, multi-target editing commands, etc. This embodiment does not impose any limitations on these commands.

[0029] Text editing commands refer to the dual operations of visually adjusting and semantically modifying text elements contained within an image. Their core lies in achieving an organic unity between text information and the image scene in terms of both visual consistency and semantic rationality. Examples include: Replacement: modifying the semantics of existing text (e.g., changing "CLOSED" on a billboard to "OPEN"); Addition: generating new text in blank areas (e.g., adding an event date to a poster); Deletion: removing specified text without damaging the background (e.g., removing a logo from a T-shirt). Visual style adjustments include, for example, material simulation: changing the font style to a metallic / glass / neon effect; artistic style: changing the font to a calligraphy / graffiti / glitch art style; and repair and enhancement: completing missing strokes / characters, improving the clarity of blurry text, and removing mosaic effects.

[0030] Material editing commands refer to the process of altering the physical properties and visual characteristics of an object's surface in an image through algorithms. While maintaining the target object's geometry and lighting conditions, they remodel its surface texture (such as roughness, reflectivity, and transparency), ensuring that the target material achieves physical consistency with the original image environment in dimensions such as color distribution, microstructure, and optical response. Examples include changing surface pattern styles (wood flooring -> marble texture); controlling the specular / diffuse reflection ratio (plastic -> metal (enhancing specular reflection intensity)); changing from ordinary glass to frosted glass (reducing transmittance); and maintaining consistent seam direction when changing fabric materials.

[0031] Local color adjustment commands refer to the editing process of selectively modifying the hue, brightness, saturation, and other color attributes of specific areas in an image. Its core lies in accurately locating the target area through semantic understanding and combining it with visual algorithms for non-destructive adjustments, ensuring that the modified area blends naturally with the overall image's lighting and texture details. For example, when a user inputs the command "deepen the red tones of the sunset," the model needs to identify the sky area, enhance the red channel values, and maintain the smoothness of the cloud transitions.

[0032] Region editing commands using masks or graffiti refer to local image modification techniques guided by user-provided spatial positioning signals (masks / graffiti) and natural language instructions. The core of this technique lies in using masks to define the boundaries of the editing area, combining text instructions to control semantic and visual feature changes within the area, and simultaneously maintaining the integrity of the non-editing area through the cross-modal attention mechanism of a diffusion model. For example, if a user selects a clothing area in an image using graffiti and inputs "change to dark blue," the model must complete the color replacement while preserving the clothing texture and folds, without affecting the surrounding skin and background areas.

[0033] Style transfer instructions refer to the technique of converting an image as a whole into a specific artistic style through natural language instructions. Its core lies in reconstructing visual elements (such as brushstrokes, tones, and textures) to match the aesthetic characteristics of the target style while maintaining the original image content structure. This process needs to balance semantic preservation and stylization intensity. For example, when converting a landscape photo into an "oil painting style," it is necessary to reconstruct swirling brushstrokes and contrasting colors while preserving the outline of the mountains.

[0034] Background adjustment commands are a global editing technique that uses natural language commands to replace content or modify environmental features in non-subject areas (or specified background areas) of an image. Its core principle is to maintain harmony between the foreground subject and the background in terms of lighting, perspective, and semantic logic. For example, the command "change the background to the foot of a snow-capped mountain" requires identifying the boundaries of the foreground subject, generating vegetation and lighting that match the topographical features of the snow-capped mountain, and adjusting the subject's projection direction to match the light source position of the new background.

[0035] Atmosphere and color editing commands refer to techniques that use natural language commands to coordinate and adjust the overall or partial color tone and lighting atmosphere of an image. The core of these commands lies in reconstructing color distribution and lighting systems through semantic understanding, ensuring consistency between the image's emotional expression and visual characteristics. For example, the command "create a retro film feel" requires reducing contrast, adding grain noise, and adjusting the color cast to a warm yellow tone, while maintaining the clarity of the subject's details.

[0036] Viewpoint transformation commands refer to the technology of globally adjusting the three-dimensional spatial viewpoint of an image through natural language commands. Its core lies in reconstructing a 2D image into a 3D scene representation, and then changing the viewpoint by modifying the virtual camera parameters, while maintaining the geometric consistency between objects in the scene. For example, the command "show the room from a top-down angle" requires reconstructing the three-dimensional spatial structure, adjusting the camera's pitch angle, and automatically filling in the details of occluded areas.

[0037] Thematic addition is a technique in image editing that uses user-guided commands to add new target objects (such as animals, objects, or people) to an original image in a semantically logical and visually plausible manner. Its core requirement is that the added object must maintain consistency with the original image's scene in terms of style, lighting, perspective, and other visual attributes, while also satisfying spatial layout requirements to achieve a "seamless insertion" editing effect.

[0038] Subject deletion is a technique in image editing that uses user commands to precisely identify and remove specified objects, while intelligently repairing the affected areas to seamlessly integrate them with their surroundings. This operation must maintain the integrity of the image structure, the continuity of texture, and the consistency of lighting and shadows, avoiding any traces of manual repair.

[0039] Theme customization instructions refer to the process of seamlessly applying instructions to other scenes or images based on a specific object. Its core is to extract theme features through decoupling design, allowing adjustments to visual presentation while preserving the object's identity.

[0040] Portrait retouching commands are a technology that uses user natural language or interactive commands to intelligently edit parts or the entire portrait in an image. Its core lies in accurately identifying specific body parts (such as clothing, face, and limbs) and maintaining consistency in lighting, texture, and perspective based on context. This technology uses AI models to achieve non-destructive editing, ensuring the generated results are natural and conform to physical laws.

[0041] Motion editing commands refer to the technical process of modifying the motion state, trajectory, or interaction of specific objects (such as people, animals, or objects) in an image using natural language commands, while keeping other parts of the scene unchanged. Its core is to achieve precise local adjustments by decoupling the dynamic attributes of objects (such as posture, direction of movement, and interactive actions).

[0042] Multi-object editing commands refer to the technical process of simultaneously modifying the dynamic attributes (such as actions, positions, and interaction relationships) and static features (such as colors and shapes) of multiple objects using natural language commands, while maintaining the consistency of scene logic to achieve complex editing. Its core lies in decoupling and reorganizing the semantic relationships between multiple objects, supporting cross-object coordinated control.

[0043] Furthermore, to enable the image editing model to provide stable and accurate image editing operations, the predictive ability of the model can be evaluated by combining a set of masked images. In this embodiment, after the step of updating the original image to the edited image using the image editing model in response to the editing instruction for the original image is executed, the method further includes: Obtain a set of masked images associated with the image editing model; evaluate the image editing model using the masked image sample data contained in the set of masked images to obtain model evaluation information corresponding to the image editing model; optimize the image editing model based on the model evaluation information until a target image editing model that meets the optimization stopping condition is obtained.

[0044] Specifically, the mask image set refers to a collection of mask image sample data. Mask image sample data specifically refers to a single set of sample data consisting of a sample image, the edited mask image, and relevant editing instructions. This data is used to evaluate the image editing model. By inputting a mask image and processing it according to the editing instructions, the generated image is compared with the sample image to determine the model's prediction performance. In practical applications, the mask image set can be pre-built and stored in a database for use during model evaluation, or multiple images can be collected before constructing the mask image set. This embodiment does not impose any limitations on this. Correspondingly, model evaluation information refers to information evaluating the model's predictive ability. It can reflect the model's current predictive ability from multiple dimensions, such as the effect of image background adjustment, image font adjustment, theme adjustment, and color adjustment, and is used for model optimization. The optimization stopping condition specifically refers to the conditions for stopping the optimization of the target image editing model, including but not limited to loss value comparison conditions, iteration count conditions, and validation set verification conditions. This embodiment does not impose any limitations on this. The target image editing model is the optimized image editing model that can be deployed in service scenarios.

[0045] Therefore, when validating an image editing model, a set of masked image samples can be used. Specifically, a set of masked images associated with the image editing model can be obtained first. Then, the masked image sample data contained in the set can be used to evaluate the image editing model. During evaluation, the masked images can be input into the model and processed according to the corresponding editing instructions. After obtaining the model's output image, it is compared with the sample images. After multiple rounds of processing, the model evaluation information corresponding to the image editing model can be obtained. Subsequently, the image editing model can be optimized based on the model evaluation information until a target image editing model that meets the optimization stopping condition is obtained.

[0046] In summary, by using masked image sets to standardize and data-drivenize model evaluation, the bias of human judgment is effectively eliminated, the automation level and naturalness of image editing tasks are significantly improved, the efficiency of model iteration is accelerated, and the editing output is guaranteed to achieve professional-grade quality in detail fusion and visual coherence, providing scalable technical support for real-time image processing applications.

[0047] Furthermore, in order to evaluate the model using a high-quality set of mask images, any mask image sample data in the set can be constructed as follows. In this embodiment, the construction of any mask image sample data contained in the mask image set includes: A target editing instruction is determined from a set of mask editing instructions, and a target image subset associated with the target editing instruction is determined from multiple image subsets; an image to be edited is extracted from the target image subset, and the image to be edited is masked according to the target editing instruction to obtain a mask image; based on the image to be edited, the mask image, and the target editing instruction, target mask image sample data is constructed.

[0048] Specifically, the mask editing instruction set refers to the set of natural language instructions recorded for editing images. Correspondingly, the image subset refers to the set obtained by dividing multiple images according to set attributes. Since different images have different characteristics—for example, image A contains multiple colors, while image B has a clear background—image A can be stored in a color image set, and image B in a background image set. When constructing target mask image sample data, color masking can be performed based on images in the color image set to test the image editing model's ability to adjust local colors, and background replacement can be performed based on images in the background image set to test the image editing model's ability to adjust backgrounds. Correspondingly, the target editing instruction is any available editing instruction randomly selected from the mask editing instructions, and the target image subset is the image subset matching the target editing instruction. This can be understood as any image in this image subset being masked according to the target editing instruction. The image to be edited is any image randomly selected from this set. The mask image specifically refers to the image obtained after processing the image according to the editing instructions, used to combine the image to be edited and the target editing instruction to form a target mask image sample data.

[0049] Based on this, for any mask image sample data contained in the mask image set, a target editing instruction can be determined from the mask editing instruction set, and a target image subset associated with the target editing instruction can be determined from multiple image subsets. Then, the image to be edited can be extracted from the target image subset, and masking processing can be performed on the image to be edited according to the target editing instruction to obtain a mask image. This allows for the construction of target mask image sample data based on the image to be edited, the mask image, and the target editing instruction. The above process can be repeated for any editing instruction in the mask editing instruction set, thereby constructing a mask image set with higher sample richness.

[0050] In summary, by dynamically and intelligently associating instructions with image subsets, the accurate and automated construction of mask samples is achieved, significantly reducing the complexity of data preparation, improving the strong matching between samples and editing tasks, optimizing model training efficiency and the ability to understand editing intentions, ensuring that image editing results maintain high accuracy and high adaptability in diverse scenarios, and providing reliable data-driven support for real-time editing applications.

[0051] Furthermore, the construction of the mask editing instruction set determines the quality of the mask image sample data; therefore, it can be divided based on set attributes, and the image subsets will also be divided according to these attributes. In this embodiment, the construction of the mask editing instruction set includes: Multiple initial editing instructions are obtained, and each initial editing instruction is optimized to obtain multiple intermediate editing instructions. According to the instruction attribute information corresponding to each intermediate editing instruction, the multiple intermediate editing instructions are divided into a first editing instruction set and a second editing instruction set. The first editing instruction set and the second editing instruction set are used as the mask editing instruction set. The construction of the plurality of image subsets includes: acquiring a plurality of sample images, and dividing the plurality of sample images into the plurality of image subsets according to the image attribute information corresponding to the plurality of sample images respectively.

[0052] Specifically, multiple initial editing instructions refer to initial editing instructions that have not yet been optimized. Correspondingly, instruction optimization refers to the operation of improving the clarity and accuracy of initial editing instructions. Instruction attribute information specifically refers to the attributes representing the form of each editing instruction, such as whether it is clear and direct or blurry and indirect. Correspondingly, image attribute information specifically refers to the attribute description information corresponding to each image, used to indicate the type of each image and facilitate the construction of image subsets.

[0053] Based on this, for the mask editing instruction set, multiple initial editing instructions can be obtained first. These initial instructions can then be optimized to obtain multiple intermediate editing instructions. Subsequently, based on the instruction attribute information corresponding to each intermediate editing instruction, these instructions can be divided into a first editing instruction set and a second editing instruction set. These first and second editing instruction sets can then be used as the mask editing instruction set. Similarly, for the multiple sub-image sets, multiple sample images can be obtained. These sample images can then be divided into multiple image subsets based on their corresponding image attribute information to construct the mask image set.

[0054] In practical applications, when acquiring sample images, images with a resolution of ≥1024×1024 can be selected to avoid compression artifacts or blurring and ensure that details are editable. At the same time, the image source can be controlled: real-shot images (images not synthesized by generative models) can be used, covering a variety of scenes such as natural scenes (e.g., natural lighting, complex backgrounds) and artificial scenes (e.g., interior design, artworks), and a few challenging scenes (occlusion, dense scenes, etc.) can also be included. In addition, the diversity of editing objects can be controlled: including a variety of text, people, animals, objects, angles, actions, time, scenes, etc. The focus should be on a wide range of categories and should not be limited to editing certain categories.

[0055] After obtaining multiple sample images, they can be divided as follows to construct image subsets corresponding to different attributes: 1. Text editing: Clear text areas: The image must contain recognizable text elements (such as signs, posters, book covers), and the text must be complete, unobstructed, and have a sufficiently high resolution; Diverse fonts / typography: covering different font styles (serif / sans-serif), font sizes, colors, and backgrounds (solid colors / complex textures); Multilingual support: Includes text in Chinese, English, and other common languages, covering both horizontal and vertical layouts.

[0056] 2. Material transformation: The target object must have a clear and uniform material texture (such as wood grain tabletops, metal utensils, and fabric sofas), clearly showing the material grain and surface characteristics.

[0057] 3. Local color adjustments: High color contrast areas: The target editing area is clearly distinguished from the surrounding colors (such as solid-color clothing, monochrome vehicles, and monochromatic buildings). Environmental interference control: Avoid specular reflections or gradient color areas, and ensure that color modification instructions can be clearly executed.

[0058] 4. Style transfer: The image should contain a subject with a recognizable geometric shape or clear outline (such as a building or a person) to facilitate the attachment of stylistic features; the original image should maintain a neutral tone to avoid interference from inherent colors in the stylization effect evaluation.

[0059] 5. Background Adjustment: The subject should have sharp edges and a clear color / brightness contrast with the background (e.g., a light-colored subject against a dark background); it should include various background types (natural / urban / indoor / abstract / monochrome / seasonal, etc.), and the subject should maintain a harmonious proportion under different backgrounds.

[0060] 6. Atmosphere and color scheme editing: It needs to contain high-definition images of various editable scenes, such as stages, beaches, grasslands, city street scenes, etc.

[0061] 7. Viewpoint Change: The target object must have clear geometric features and may contain some reference objects in the background.

[0062] 8. Adding, deleting, and customizing themes: It has a clear and well-defined theme, such as people, anime characters, animals, objects, etc. (It can include a single theme or multiple themes).

[0063] 9. Portrait retouching: We need high-resolution human portrait images with rich expressions and poses, and facial features facing the camera without obstruction.

[0064] 10. Motion Editing: It needs to include a variety of target actions, such as human movements and animal movements, and can involve various scenarios such as sports and daily life.

[0065] 11. Multi-target editing: High-quality images containing multiple interacting themes are needed, such as two people shaking hands, a person holding a pet cat, a dog carrying a ball, etc. The subjects should be clearly visible and facing the camera.

[0066] 12. Editing based on masked or graffiti areas: The data that requires local guidance can be combined and constructed as described above; only further masking annotation of the image is needed.

[0067] After dividing the image into multiple subsets as described above, mask editing instructions can be constructed. A large language model combined with manual input can be used to design editing instructions for the data, ensuring that the editing reflects the user's actual needs. It should be noted that both clear and direct editing instructions and ambiguous or indirect instructions should be included. 1. First, a large language model is used to extract image content and provide several types of editing instructions.

[0068] 2. Next, the generated suggested instructions are collected, selected, and rewritten. This rewriting includes rewriting clear instructions as well as rewriting vague instructions for non-professional users.

[0069] 3. Finally, after obtaining all the editing instructions, the rationality and accuracy of all evaluation units are checked and screened using a combination of large language models and manual methods, thereby obtaining a high-quality set of mask editing instructions.

[0070] Furthermore, when constructing masked image sample data, a combination of a large language model, GroundingSAM, and manual input can be used to construct the ground truth mask values. 1. First, the clear instructions in each evaluation unit are analyzed using a large language model to extract content description units related to editing.

[0071] 2. Next, GroundingSAM and description units are used to perform preliminary mask calculations on the image to obtain the instance mask (mask image).

[0072] 3. Finally, the mask is appropriately expanded and simplified according to the instructions, instance mask, and image to avoid the main target falling outside the mask area after reasonable editing, which would lead to an incorrect preservation evaluation.

[0073] Based on the above processing, a high-quality set of masked images can be obtained. Then, the masked image sample data contained in the set can be used to evaluate the image editing model, thereby ensuring the deployment of a high-quality model in the service environment.

[0074] In summary, by using attribute-driven instruction classification and image subset partitioning, precise coordination between editing instructions and image data is achieved, significantly improving instruction execution accuracy and data utilization efficiency, reducing training complexity, ensuring the stability and consistency of output in multi-task editing, and providing an efficient and scalable instruction management mechanism for intelligent image editing systems.

[0075] Step S104: Input the original image and the edited image into the image evaluation model for processing to obtain image evaluation information corresponding to multiple image evaluation dimensions.

[0076] Specifically, after obtaining the edited image output by the image editing model, in order to evaluate the image from multiple dimensions, the original image and the edited image can be input into the image evaluation model for processing. This allows the image evaluation model to evaluate the edited image from multiple image evaluation dimensions, thereby obtaining image evaluation information corresponding to each image evaluation dimension.

[0077] Specifically, the image evaluation model refers to a model that can evaluate an edited image from multiple image evaluation dimensions and output corresponding evaluation information. It can be implemented using a multimodal large language model, or using a CNN (Convolutional Neural Network) model (CNN models include image preprocessing layers, convolutional layers, pooling layers, and fully connected layers), a rule engine (based on rules set by brightness, contrast, edge sharpness, etc.), a quality evaluation algorithm (such as SSIM (Structural Similarity Index Measure), PSNR (Peak Signal-to-Noise Ratio), MSE (Mean Squared Error), etc.), a deep learning model (such as LPIPS (Learned Perceptual Image Patch Similarity), or a hybrid model (a model combining CNN and rule engine). In specific implementation, the choice can be made according to actual needs, and this embodiment does not impose any limitations.

[0078] Accordingly, multiple image evaluation dimensions include, but are not limited to, instruction execution dimension, quality evaluation dimension, and / or reasonableness evaluation dimension, etc., and this embodiment does not impose any limitations on them. Image evaluation information specifically refers to the descriptive information representing the evaluation results of the image in the corresponding dimension.

[0079] Furthermore, when using an image editing model for multi-dimensional image evaluation, it can be achieved from the instruction execution dimension, the rationality evaluation dimension, and the quality evaluation dimension. In this embodiment, the step of inputting the original image and the edited image into the image evaluation model for processing to obtain image evaluation information corresponding to multiple image evaluation dimensions includes: The original image and the edited image are input into an image evaluation model; the image evaluation model is used to compare the original image and the edited image, and the image evaluation information corresponding to the instruction execution dimension is determined based on the comparison result; the image evaluation model is used to perform a reasonable evaluation on the edited image to obtain the image evaluation information corresponding to the reasonable evaluation dimension; the image evaluation model is used to perform a quality evaluation on the edited image, and the image evaluation information corresponding to the quality evaluation dimension is determined based on the evaluation result; the image evaluation information corresponding to the instruction execution dimension, the image evaluation information corresponding to the reasonable evaluation dimension, and the image evaluation information corresponding to the quality evaluation dimension are used as the image evaluation information corresponding to multiple image evaluation dimensions respectively.

[0080] Specifically, the instruction execution dimension measures whether the editing instructions in the edited result are accurately executed, ensuring that the model accurately understands and executes the user's intent. The reasonableness assessment dimension evaluates the realism, naturalness, and logical rationality of the generated image, ensuring that the edited content conforms to physical laws, common sense, and visual coherence, avoiding distortions, counterintuitive elements, or logical contradictions. The quality assessment dimension evaluates the visual fidelity of the edited image, including sharpness, artifacts, naturalness, and the presence of editing traces.

[0081] Therefore, to ensure accuracy when using image evaluation for multi-dimensional evaluation, it can be implemented from the instruction execution dimension, quality evaluation dimension, and rational evaluation dimension. On one hand, the original image and the edited image can be input into the image evaluation model; the model compares the original and edited images, and the image evaluation information corresponding to the instruction execution dimension can be determined based on the comparison results. On the other hand, the image evaluation model can be used to perform a rational evaluation on the edited image, obtaining the image evaluation information corresponding to the rational evaluation dimension; it can also be used to perform a quality evaluation on the edited image, determining the image evaluation information corresponding to the quality evaluation dimension based on the evaluation results. Subsequently, the image evaluation information corresponding to the instruction execution dimension, the rational evaluation dimension, and the quality evaluation dimension can be used as the image evaluation information corresponding to each of the multiple image evaluation dimensions for subsequent determination of target evaluation information.

[0082] In summary, by using an image evaluation model to achieve automated quantitative evaluation of instruction execution, content rationality, and visual quality in three dimensions, the model effectively avoids inconsistencies caused by subjective human judgment, significantly improves evaluation accuracy and real-time performance, provides an objective and comprehensive quality feedback loop for image editing tasks, significantly enhances the professional applicability and user satisfaction of editing results, and promotes the intelligent upgrade of image processing technology.

[0083] Furthermore, when performing image evaluation processing for quality assessment dimensions, visual and semantic information can be combined. In this embodiment, the process of using the image evaluation model to perform quality assessment on the edited image and determining the image evaluation information corresponding to the quality assessment dimension based on the evaluation results includes: The image evaluation model is used to evaluate the quality of the edited image in the semantic sub-dimension to obtain image evaluation sub-information corresponding to the semantic sub-dimension; the image evaluation model is used to evaluate the quality of the edited image in the visual sub-dimension according to preset visual indicators to obtain image evaluation sub-information corresponding to the visual sub-dimension; the image evaluation sub-information corresponding to the semantic sub-dimension and the image evaluation sub-information corresponding to the visual sub-dimension are fused to obtain image evaluation information corresponding to the quality evaluation dimension.

[0084] Specifically, the semantic sub-dimension refers to the dimension for evaluating the quality of edited images from a semantic perspective. Correspondingly, the visual metrics refer to the defined indicators for evaluating edited images from a visual perspective, such as sharpness, color, brightness, and size. In other words, the visual sub-dimension refers to the dimension for evaluating the quality of edited images from a visual perspective.

[0085] Therefore, in order to accurately determine the evaluation information of the image quality dimension, a comprehensive evaluation can be performed by combining the visual and semantic sub-dimensions. Specifically, an image evaluation model can be used to evaluate the quality of the edited image in the semantic sub-dimension to obtain the image evaluation sub-information corresponding to the semantic sub-dimension; simultaneously, the edited image can be evaluated in the visual sub-dimension according to preset visual indicators to obtain the image evaluation sub-information corresponding to the visual sub-dimension; then, the image evaluation sub-information corresponding to the semantic and visual sub-dimensions can be fused to obtain the image evaluation information corresponding to the quality evaluation dimension.

[0086] In practical applications, image evaluation from the perspective of instruction execution essentially measures whether the editing instructions in the edited result are accurately completed, ensuring that the model accurately understands and executes the user's intent. Existing methods only use simple rule design to score the instruction compliance of the edited result. These coarse rules make it difficult for the model to capture subtle local editing differences between images, and the scoring criteria depend on model preferences, making it difficult to align with human perception. Therefore, precise evaluation of the instruction execution dimension can be achieved by designing rules that guide the model to focus on editing areas of interest, emphasizing the comparison of differences between two images. Detailed evaluation criteria can be provided for each scoring level to align with user perception; furthermore, an inference stage can be added to enhance the model's instruction understanding ability. Through these improvements, instruction compliance can be analyzed and evaluated in detail, progressing from coarse to fine. This achieves a comprehensive and detailed reflection of the model's ability to parse text instructions, a core indicator for evaluating editing effectiveness. Automated scoring quantifies the model's performance under different instructions.

[0087] Image evaluation, when conducted across reasonable evaluation dimensions, essentially assesses the realism, naturalness, and logical rationality of the generated image. This ensures that the edited content conforms to physical laws, common sense, and visual coherence, avoiding distortions, counterintuitive elements, or logical contradictions. Current models commonly suffer from violations of physical laws (e.g., gravity violations, perspective errors), lack of common sense (inconsistent seasonal characteristics, incorrect chemical reactions), object deformation (abnormal limb structures, distorted textures), and logical contradictions (disordered timelines, conflicting spatial relationships). This embodiment, by setting a new prompt, evaluates whether the edited image conforms to factual logic from different perspectives. It reveals the shortcomings of the editing model at the cognitive reasoning level, providing a comprehensive evaluation of the rationality of the editing results to align with practical applications.

[0088] Image quality assessment essentially evaluates the visual fidelity of the edited image, including sharpness, artifacts, naturalness, and the presence of editing artifacts. Existing methods typically use statistical metrics such as LPIPS and SSIM for evaluation alone, or use model design rules for autonomous scoring. However, the former does not align well with user perception, while the latter often reflects inherent aesthetic biases, tending towards unrealistic AI-style images. This embodiment alleviates these problems by utilizing a comprehensive evaluation. User-perceived alignment metrics capture the overall image structure and assess the retention of high-frequency details, while a model is combined to perform semantic-level quality assessment. Finally, these two scores are combined to achieve multi-faceted cross-evaluation. This avoids technical defects (such as blurring or distortion) introduced by editing operations, ensuring that the output meets basic visual standards. High-quality editing results are a prerequisite for practical applications.

[0089] In summary, by deeply co-evaluating and fusing semantic and visual sub-dimensions, the semantic coherence and visual fluency of image editing results are significantly improved. This effectively avoids the one-sidedness of a single evaluation indicator, achieves multi-dimensional and accurate quantification of editing quality, provides highly reliable data support for model optimization, and ensures that the output images have both detail precision and overall aesthetic appeal in professional scenarios, greatly enhancing the practical value and market competitiveness of the image editing system.

[0090] Step S106: Divide the edited image into edited regions and non-edited regions, and determine the edit evaluation information corresponding to the edited regions and the non-edit evaluation information corresponding to the non-edited regions.

[0091] Specifically, after the model obtained above evaluates the image from multiple dimensions, the edited image can be divided into edited and non-edited regions. At this point, the edited and non-edited regions can be evaluated separately to determine the editing evaluation information corresponding to the edited region and the non-editing evaluation information corresponding to the non-edited region. Subsequently, the coarse-grained evaluation information and the fine-grained evaluation information can be combined to complete the overall evaluation of the image, thereby improving the accuracy of image evaluation.

[0092] Specifically, the edited region refers to the image area in the edited image that is edited by the image editing model driven by editing instructions, while the non-edited region is the area in the edited image that is not edited by the image editing model. Editing evaluation information specifically refers to the image evaluation information corresponding to the edited region, and non-editing evaluation information specifically refers to the image evaluation information corresponding to the non-edited region. In practice, the evaluation methods for the two regions can be implemented through a model or by setting rules; this embodiment does not impose any limitations. Furthermore, the division between the edited and non-edited regions can be achieved through an object detection model to ensure the accuracy of the division.

[0093] Furthermore, the evaluation of editable and non-editable regions can be achieved by combining models. In this embodiment, determining the editing evaluation information corresponding to the editable region and the non-editing evaluation information corresponding to the non-editable region includes: The edited region is input into an image topic detection model for processing to obtain image evaluation information corresponding to the topic evaluation dimension, and the image evaluation information corresponding to the topic evaluation dimension is used as the editing evaluation information; the non-edited region is input into an image preservation detection model for processing to obtain image evaluation information corresponding to the image preservation dimension, and the image evaluation information corresponding to the image preservation dimension is used as the non-editing evaluation information.

[0094] Specifically, the image subject detection model refers to a model that detects subject targets in the edited region before and after editing, while the image dimension detection model refers to a model that detects the degree of detail preservation in the non-edited region before and after editing.

[0095] Based on this, in order to achieve accurate detection of edited and non-edited regions, two different models can be used. Specifically, the edited region can be input into the image topic detection model for processing to obtain the image evaluation information corresponding to the topic evaluation dimension, and then the image evaluation information corresponding to the topic evaluation dimension can be used as the editing evaluation information. At the same time, the non-edited region can be input into the image preservation detection model for processing to obtain the image evaluation information corresponding to the image preservation dimension, and then the image evaluation information corresponding to the image preservation dimension can be used as the non-editing evaluation information.

[0096] In practical applications, the evaluation of edited and non-edited regions involves assessing the detail retention of the subject before and after editing, as well as the detail retention of the non-edited regions. Existing methods either ignore the evaluation of these regions or only use models for coarse semantic-level evaluation. This results in a high score despite poor editing performance, even if the editing instructions have been followed. This embodiment combines masking to separate the edited and non-edited regions of the image, and uses CLIP (Contrastive Language-Image Pretraining) and DINO (DIstillation with NO labels) to evaluate the similarity of the subject. It uses SSIM (Structural Similarity Index) and LPIPS (Learned Perceptual Image Patch Similarity) to evaluate the detail fidelity of the non-edited regions. This allows for a comprehensive evaluation of the edited image from different perspectives, leveraging the advantages of various metrics to mitigate the limitations of semantic-level one-sided evaluation and comprehensively assess the detail retention of the edited image.

[0097] In summary, by employing a targeted evaluation mechanism that combines topic detection in edited regions with maintenance detection in non-edited regions, the scope of the editing intent can be accurately pinpointed while effectively suppressing regional interference. This ensures the consistency of the edited content's theme and the integrity of the original image's non-edited regions, significantly improving the natural integration and visual credibility of the editing results. It provides a high-precision, low-risk evaluation and optimization path for professional image processing.

[0098] Step S108: According to the region information of the edited region, the image evaluation information, the edit evaluation information, and the non-edit evaluation information are fused to obtain the target evaluation information corresponding to the edited image.

[0099] Specifically, after obtaining image evaluation information, editing evaluation information and non-editing evaluation information corresponding to multiple image evaluation dimensions, the image evaluation information, editing evaluation information and non-editing evaluation information can be fused according to the regional information of the editing area to comprehensively consider the proportion of the editing area in the image, thereby obtaining more accurate target evaluation information.

[0100] Specifically, "regional information" refers to the attribute information corresponding to the editing region, including but not limited to the theme, color, background, and region proportion of the editing region. "Target evaluation information" is the evaluation information obtained after comprehensively evaluating the edited image. It can be in the form of a score or descriptive text. When generating target evaluation information, image evaluation information, editing evaluation information, and non-editing evaluation information can be combined to generate a score value corresponding to the edited image. The size of the score value reflects the quality of the edited image; for example, the quantitative indicator can be a comprehensive score from 0 to 100. A visualization report corresponding to the edited image can also be generated by combining image evaluation information, editing evaluation information, and non-editing evaluation information. This visualization report can record the evaluation results for each dimension, thereby improving the interpretability of the edited image evaluation. For example, an evaluation report including a heatmap can be generated, where highlighted areas show the quality distribution of the editing region. A scoring matrix can also be output, with different matrix elements representing different dimension scores. Before-and-after comparison images and annotations of key evaluation points can also be provided to ensure the interpretability of the target evaluation information. In practical applications, the specific form of the target evaluation information can be set according to actual needs; this embodiment does not impose any limitations.

[0101] Furthermore, the target evaluation information can be achieved by weighted fusion based on the proportion information of the edited region. In this embodiment, the step of fusing the image evaluation information, the edit evaluation information, and the non-edit evaluation information according to the regional information of the edited region to obtain the target evaluation information corresponding to the edited image includes: The region information corresponding to the editing region is determined, and the image proportion information corresponding to the editing region is determined based on the region information; an information fusion weight is constructed based on the image proportion information, and the image evaluation information, the editing evaluation information and the non-editing evaluation information are fused according to the information fusion weight to obtain the target evaluation information corresponding to the edited image.

[0102] Specifically, image proportion information refers to the ratio of the editing area to the target image. Fusion weight refers to the weights calculated based on the image proportion information; in other words, the larger the proportion, the higher the weight.

[0103] Based on this, when fusing evaluation information across multiple dimensions, we can first determine the regional information corresponding to the editing region. At this point, we can determine the image proportion information corresponding to the editing region based on the regional information. Then, we can construct information fusion weights based on the image proportion information and fuse the image evaluation information, editing evaluation information, and non-editing evaluation information according to the information fusion weights. This will enable us to construct the target evaluation information corresponding to the edited image based on the fusion results.

[0104] In practical applications, the proportion of the edited and non-edited regions in the original image can be calculated based on the mask and the sigmoid function. Then, the weights w can be dynamically calculated, and the mask proportions are non-linearly mapped using the sigmoid function: w = σ(k*(m - m0)), where m is the mask proportion, m0 is the reasonable edited region threshold (50%), and k is the slope coefficient. As m approaches 0, the following weights automatically decay. By controlling the slope coefficient, evaluation imbalances in extreme cases (editing extremely large or small regions) can be mitigated, thus providing more accurate target evaluation information.

[0105] For example, see Figure 2 The diagram illustrates that after obtaining the original image, the received editing instruction is "{delete trees from the image}". An image editing model can then be used to process the original image according to this instruction, resulting in an edited image. To provide users with high-quality feedback, the edited image can be evaluated in multiple dimensions. By comparing the original and edited images, we can measure whether the editing instructions were accurately executed, thus obtaining instruction compliance assessment information. We can also assess the realism, naturalness, and logical rationality of the generated image, ensuring that the edited content conforms to physical laws, common sense, and visual coherence, avoiding distortions, counterintuitive elements, or logical contradictions, thus obtaining image rationality assessment information. Furthermore, we can evaluate the visual fidelity of the edited image, including sharpness, artifacts, naturalness, and the presence of editing traces, thus obtaining image quality assessment information.

[0106] Furthermore, the edited image can be divided into edited and non-edited regions. CLIP & DINO can then be used to evaluate the similarity of the subject, while SSIM & LPIPS can be used to evaluate the detail fidelity of the non-edited region. This allows for the assessment of the detail retention of the subject before and after editing, as well as the non-edited region.

[0107] After obtaining the above evaluation information, the multi-dimensional evaluation information can be fused to obtain the target evaluation information corresponding to the edited image. If the target evaluation information indicates that the non-edited area of ​​the edited image has low preservation, the edited image can be optimized based on this information, and then a higher quality target image can be fed back to the user.

[0108] In summary, by dynamically generating fusion weights based on the proportion of the editing area, intelligent weighted integration of multi-dimensional evaluation information is achieved, significantly improving the accuracy and decision-making value of target evaluation information, effectively avoiding the risk of global distortion, and ensuring that the editing results achieve a balance between local detail optimization and overall visual coordination, thus providing a high-precision and quantifiable evaluation and optimization basis for image editing systems.

[0109] The image processing method provided in this embodiment, in order to accurately evaluate the image processed by the image editing model, can first respond to the editing instruction for the original image and update the original image into an edited image using the image editing model. After obtaining the edited image, the original image and the edited image can be input into the image evaluation model for processing. The image evaluation model can perform basic evaluation operations on the image from multiple image evaluation dimensions, thereby obtaining image evaluation information corresponding to each image evaluation dimension. The evaluation result of the model is only associated with the set multiple image evaluation dimensions. In order to improve the rigor of the evaluation result, the edited image can be divided into edited regions and non-edited regions, and local evaluation can be performed on the edited regions and non-edited regions to determine the editing evaluation information corresponding to the edited regions and the non-editing evaluation information corresponding to the non-edited regions. On this basis, the image evaluation information, editing evaluation information and non-editing evaluation information can be fused by combining the regional information of the edited region to achieve the determination of the target evaluation information corresponding to the edited image by comprehensively considering multiple evaluation information. This enables automated multi-angle evaluation of the editing effect when evaluating the image, thereby effectively improving the evaluation accuracy so that the accurate evaluation results can be used for subsequent service processing.

[0110] See Figure 3 , Figure 3 A flowchart of another image processing method according to an embodiment of this specification is shown. The method is applied to a server and specifically includes the following steps.

[0111] Step S302: Receive the editing instruction submitted by the client carrying the original image, and in response to the editing instruction, update the original image to the edited image using the image editing model.

[0112] Step S304: Input the original image and the edited image into the image evaluation model for processing to obtain image evaluation information corresponding to multiple image evaluation dimensions.

[0113] Step S306: Divide the edited image into edited regions and non-edited regions, and determine the edit evaluation information corresponding to the edited regions and the non-edited evaluation information corresponding to the non-edited regions.

[0114] Step S308: According to the region information of the edited area, the image evaluation information, the edit evaluation information and the non-edit evaluation information are fused to obtain the target evaluation information corresponding to the edited image.

[0115] Step S310: Update the edited image to the target image based on the target evaluation information, and send the target image to the client.

[0116] This embodiment provides another image processing method. Any content not described in detail can refer to the same or corresponding descriptions in the above embodiments. This embodiment will not elaborate further.

[0117] Specifically, the server refers to the server providing image editing services, and the client refers to the terminal held by the user using the image editing services. Correspondingly, the target image refers to the image obtained after updating the edited image based on the target evaluation information. It can be understood that the target evaluation information can reflect the evaluation results of the edited image in different dimensions. When the evaluation results do not meet the set conditions, the edited image can be further optimized using the image editing model to obtain a high-quality target image to provide feedback to the user.

[0118] The following is in conjunction with the appendix Figure 4 Taking the application of the image processing method provided in this specification in an image generation scenario as an example, the image processing method will be further explained. Among them, Figure 4 A flowchart illustrating the processing steps of an image processing method according to an embodiment of this specification is shown, specifically including the following steps.

[0119] Step S402: In response to the editing instruction for the original image, the original image is updated to an edited image using an image editing model.

[0120] Step S404: Input the original image and the edited image into the image evaluation model.

[0121] Step S406: Compare the original image and the edited image using an image evaluation model, and determine the image evaluation information corresponding to the instruction execution dimension based on the comparison results.

[0122] Step S408: Use the image evaluation model to reasonably evaluate the edited image and obtain the image evaluation information corresponding to the reasonable evaluation dimensions.

[0123] Step S410: Use the image evaluation model to evaluate the quality of the edited image in the semantic sub-dimension and obtain the image evaluation sub-information corresponding to the semantic sub-dimension.

[0124] Step S412: Evaluate the quality of the edited image in the visual sub-dimension according to the preset visual indicators, and obtain the image evaluation sub-information corresponding to the visual sub-dimension.

[0125] Step S414: The image evaluation sub-information corresponding to the semantic sub-dimension and the image evaluation sub-information corresponding to the visual sub-dimension are fused to obtain the image evaluation information corresponding to the quality evaluation dimension.

[0126] Step S416: The image evaluation information corresponding to the instruction execution dimension, the image evaluation information corresponding to the reasonable evaluation dimension, and the image evaluation information corresponding to the quality evaluation dimension are used as the image evaluation information corresponding to the multiple image evaluation dimensions respectively.

[0127] Step S418: Divide the edited image into edited and non-edited regions.

[0128] Step S420: Input the editing area into the image topic detection model for processing to obtain the image evaluation information corresponding to the topic evaluation dimension, and use the image evaluation information corresponding to the topic evaluation dimension as the editing evaluation information.

[0129] Step S422: Input the non-edited region into the image preservation detection model for processing to obtain the image evaluation information corresponding to the image preservation dimension, and use the image evaluation information corresponding to the image preservation dimension as the non-editing evaluation information.

[0130] Step S424: Determine the region information corresponding to the editing region, and determine the image proportion information corresponding to the editing region based on the region information.

[0131] Step S426: Construct information fusion weights based on image proportion information, and fuse image evaluation information, editing evaluation information and non-editing evaluation information according to the information fusion weights to obtain target evaluation information corresponding to the edited image.

[0132] Step S428: Update the edited image to the target image based on the target evaluation information, and send the target image to the client for display.

[0133] In summary, to accurately evaluate images processed by the image editing model, we can first respond to editing instructions on the original image and update it into an edited image using the image editing model. After obtaining the edited image, we can input both the original and edited images into the image evaluation model for processing. This allows the model to perform basic evaluation operations on the image from multiple image evaluation dimensions, thereby obtaining image evaluation information corresponding to each dimension. The model's evaluation results are only associated with the set multiple image evaluation dimensions. To improve the rigor of the evaluation results, the edited image can be divided into edited and non-edited regions, and local evaluations can be performed on both regions to determine the editing evaluation information corresponding to the edited region and the non-editing evaluation information corresponding to the non-edited region. Based on this, we can combine the regional information of the edited region to fuse the image evaluation information, editing evaluation information, and non-editing evaluation information, thereby comprehensively determining the target evaluation information corresponding to the edited image. This enables automated multi-angle evaluation of the editing effect during image evaluation processing, effectively improving the evaluation accuracy and allowing for subsequent service processing using accurate evaluation results.

[0134] Corresponding to the above method embodiments, this specification also provides embodiments of image processing apparatus. Figure 5 A schematic diagram of the structure of an image processing apparatus according to one embodiment of this specification is shown. Figure 5 As shown, the device includes: The update module 502 is configured to update the original image into an edited image using an image editing model in response to an editing instruction for the original image; Processing module 504 is configured to input the original image and the edited image into an image evaluation model for processing, and obtain image evaluation information corresponding to multiple image evaluation dimensions respectively; The determining module 506 is configured to divide the edited image into edited regions and non-edited regions, and determine the editing evaluation information corresponding to the edited regions and the non-editing evaluation information corresponding to the non-edited regions; The fusion module 508 is configured to fuse the image evaluation information, the editing evaluation information, and the non-editing evaluation information according to the region information of the editing region to obtain the target evaluation information corresponding to the edited image.

[0135] In an optional embodiment, the step of inputting the original image and the edited image into an image evaluation model for processing to obtain image evaluation information corresponding to multiple image evaluation dimensions includes: The original image and the edited image are input into an image evaluation model; the image evaluation model is used to compare the original image and the edited image, and the image evaluation information corresponding to the instruction execution dimension is determined based on the comparison result; the image evaluation model is used to perform a reasonable evaluation on the edited image to obtain the image evaluation information corresponding to the reasonable evaluation dimension; the image evaluation model is used to perform a quality evaluation on the edited image, and the image evaluation information corresponding to the quality evaluation dimension is determined based on the evaluation result; the image evaluation information corresponding to the instruction execution dimension, the image evaluation information corresponding to the reasonable evaluation dimension, and the image evaluation information corresponding to the quality evaluation dimension are used as the image evaluation information corresponding to multiple image evaluation dimensions respectively.

[0136] In an optional embodiment, the step of using the image evaluation model to perform quality evaluation on the edited image and determining the image evaluation information corresponding to the quality evaluation dimension based on the evaluation result includes: The image evaluation model is used to evaluate the quality of the edited image in the semantic sub-dimension to obtain image evaluation sub-information corresponding to the semantic sub-dimension; the image evaluation model is used to evaluate the quality of the edited image in the visual sub-dimension according to preset visual indicators to obtain image evaluation sub-information corresponding to the visual sub-dimension; the image evaluation sub-information corresponding to the semantic sub-dimension and the image evaluation sub-information corresponding to the visual sub-dimension are fused to obtain image evaluation information corresponding to the quality evaluation dimension.

[0137] In an optional embodiment, determining the editing evaluation information corresponding to the edited area and the non-editing evaluation information corresponding to the non-edited area includes: The edited region is input into an image topic detection model for processing to obtain image evaluation information corresponding to the topic evaluation dimension, and the image evaluation information corresponding to the topic evaluation dimension is used as the editing evaluation information; the non-edited region is input into an image preservation detection model for processing to obtain image evaluation information corresponding to the image preservation dimension, and the image evaluation information corresponding to the image preservation dimension is used as the non-editing evaluation information.

[0138] In an optional embodiment, fusing the image evaluation information, the editing evaluation information, and the non-editing evaluation information according to the region information of the edited region to obtain the target evaluation information corresponding to the edited image includes: The region information corresponding to the editing region is determined, and the image proportion information corresponding to the editing region is determined based on the region information; an information fusion weight is constructed based on the image proportion information, and the image evaluation information, the editing evaluation information and the non-editing evaluation information are fused according to the information fusion weight to obtain the target evaluation information corresponding to the edited image.

[0139] In an optional embodiment, after the step of updating the original image to an edited image using an image editing model in response to an editing instruction for the original image is performed, the method further includes: Obtain a set of masked images associated with the image editing model; evaluate the image editing model using the masked image sample data contained in the set of masked images to obtain model evaluation information corresponding to the image editing model; optimize the image editing model based on the model evaluation information until a target image editing model that meets the optimization stopping condition is obtained.

[0140] In an optional embodiment, the construction of any mask image sample data contained in the mask image set includes: A target editing instruction is determined from a set of mask editing instructions, and a target image subset associated with the target editing instruction is determined from multiple image subsets; an image to be edited is extracted from the target image subset, and the image to be edited is masked according to the target editing instruction to obtain a mask image; based on the image to be edited, the mask image, and the target editing instruction, target mask image sample data is constructed.

[0141] In one optional embodiment, the construction of the mask editing instruction set includes: Multiple initial editing instructions are obtained, and each initial editing instruction is optimized to obtain multiple intermediate editing instructions. Based on the instruction attribute information corresponding to each intermediate editing instruction, the multiple intermediate editing instructions are divided into a first editing instruction set and a second editing instruction set. The first editing instruction set and the second editing instruction set are used as the mask editing instruction set. The construction of the multiple image subsets includes: obtaining multiple sample images, and dividing the multiple sample images into the multiple image subsets based on the image attribute information corresponding to each sample image.

[0142] The image processing apparatus provided in this embodiment, in order to accurately evaluate the image processed by the image editing model, can first respond to the editing instruction for the original image and update the original image into an edited image using the image editing model. After obtaining the edited image, the original image and the edited image can be input into the image evaluation model for processing. The image evaluation model can perform basic evaluation operations on the image from multiple image evaluation dimensions, thereby obtaining image evaluation information corresponding to each image evaluation dimension. The evaluation result of the model is only associated with the set multiple image evaluation dimensions. In order to improve the rigor of the evaluation result, the edited image can be divided into edited regions and non-edited regions, and local evaluation can be performed on the edited regions and non-edited regions to determine the editing evaluation information corresponding to the edited regions and the non-editing evaluation information corresponding to the non-edited regions. On this basis, the image evaluation information, editing evaluation information and non-editing evaluation information can be fused by combining the regional information of the edited region to achieve the determination of the target evaluation information corresponding to the edited image by comprehensively considering multiple evaluation information. This enables automated multi-dimensional and multi-angle evaluation of the editing effect when evaluating the image, thereby effectively improving the evaluation accuracy so that the accurate evaluation results can be used for subsequent service processing.

[0143] The above is an illustrative scheme of an image processing apparatus according to this embodiment. It should be noted that the technical solution of this image processing apparatus and the technical solution of the image processing method described above belong to the same concept. For details not described in detail in the technical solution of the image processing apparatus, please refer to the description of the technical solution of the image processing method described above.

[0144] Corresponding to the above method embodiments, this specification also provides another embodiment of an image processing apparatus. Figure 6 A schematic diagram of another image processing apparatus provided in one embodiment of this specification is shown. Figure 6 As shown, this device is used on the server side and includes: The receiving module 602 is configured to receive an editing instruction carrying an original image submitted by a client, and in response to the editing instruction, update the original image to an edited image using an image editing model; Processing module 604 is configured to input the original image and the edited image into an image evaluation model for processing, and obtain image evaluation information corresponding to multiple image evaluation dimensions respectively; The segmentation module 606 is configured to divide the edited image into edited regions and non-edited regions, and determine the edit evaluation information corresponding to the edited regions and the non-edit evaluation information corresponding to the non-edited regions; The fusion module 608 is configured to fuse the image evaluation information, the editing evaluation information, and the non-editing evaluation information according to the region information of the editing region to obtain the target evaluation information corresponding to the edited image; The sending module 610 is configured to update the edited image to the target image based on the target evaluation information, and send the target image to the client.

[0145] The above is an illustrative scheme of another image processing apparatus according to this embodiment. It should be noted that the technical solution of this image processing apparatus and the technical solution of the image processing method described above belong to the same concept. For details not described in detail in the technical solution of the image processing apparatus, please refer to the description of the technical solution of the image processing method described above.

[0146] Figure 7 A structural block diagram of a computing device 700 according to one embodiment of this specification is shown. The components of the computing device 700 include, but are not limited to, a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and a database 750 is used to store data.

[0147] The computing device 700 also includes an access device 740, which enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 740 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0148] In one embodiment of this specification, the above-described components of the computing device 700 and Figure 7Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 7 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0149] The computing device 700 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 700 can also be a mobile or stationary server.

[0150] The processor 720 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described image processing method.

[0151] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the image processing method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the image processing method described above.

[0152] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described image processing method.

[0153] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the image processing method described above belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the image processing method described above.

[0154] An embodiment of this specification also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described image processing method.

[0155] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the image processing method described above belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the image processing method described above.

[0156] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0157] The computer program / instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0158] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0159] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0160] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. An image processing method, comprising: In response to an editing instruction for the original image, the original image is updated to an edited image using an image editing model; The original image and the edited image are input into the image evaluation model for processing to obtain image evaluation information corresponding to multiple image evaluation dimensions. The edited image is divided into edited and non-edited regions, and the edit evaluation information corresponding to the edited regions and the non-edited evaluation information corresponding to the non-edited regions are determined. Based on the region information of the edited area, the image evaluation information, the edit evaluation information, and the non-edit evaluation information are fused to obtain the target evaluation information corresponding to the edited image.

2. The image processing method according to claim 1, wherein inputting the original image and the edited image into an image evaluation model for processing to obtain image evaluation information corresponding to multiple image evaluation dimensions respectively includes: The original image and the edited image are input into the image evaluation model; The original image and the edited image are compared using the image evaluation model, and the image evaluation information corresponding to the instruction execution dimension is determined based on the comparison results. The image evaluation model is used to reasonably evaluate the edited image, and image evaluation information corresponding to the reasonable evaluation dimensions is obtained. The image evaluation model is used to evaluate the quality of the edited image, and the image evaluation information corresponding to the quality evaluation dimension is determined based on the evaluation results. The image evaluation information corresponding to the instruction execution dimension, the image evaluation information corresponding to the reasonable evaluation dimension, and the image evaluation information corresponding to the quality evaluation dimension are used as the image evaluation information corresponding to multiple image evaluation dimensions respectively.

3. The image processing method according to claim 2, wherein the step of using the image evaluation model to perform quality evaluation on the edited image and determining the image evaluation information corresponding to the quality evaluation dimension based on the evaluation result includes: The image evaluation model is used to evaluate the quality of the edited image in the semantic sub-dimension to obtain the image evaluation sub-information corresponding to the semantic sub-dimension. The quality of the edited image is evaluated in the visual sub-dimension according to the preset visual indicators, and the image evaluation sub-information corresponding to the visual sub-dimension is obtained. The image evaluation sub-information corresponding to the semantic sub-dimension and the image evaluation sub-information corresponding to the visual sub-dimension are fused to obtain the image evaluation information corresponding to the quality evaluation dimension.

4. The image processing method according to claim 1, wherein determining the editing evaluation information corresponding to the edited region and the non-editing evaluation information corresponding to the non-edited region includes: The edited area is input into the image topic detection model for processing to obtain image evaluation information corresponding to the topic evaluation dimension. The image evaluation information corresponding to the topic evaluation dimension is used as the editing evaluation information. The unedited region is input into the image preservation detection model for processing to obtain image evaluation information corresponding to the image preservation dimension. The image evaluation information corresponding to the image preservation dimension is used as the unedited evaluation information.

5. The image processing method according to claim 1, wherein fusing the image evaluation information, the editing evaluation information, and the non-editing evaluation information according to the region information of the edited region to obtain the target evaluation information corresponding to the edited image includes: Determine the region information corresponding to the editing region, and determine the image proportion information corresponding to the editing region based on the region information; Information fusion weights are constructed based on the image proportion information, and the image evaluation information, the edit evaluation information, and the non-edit evaluation information are fused according to the information fusion weights to obtain the target evaluation information corresponding to the edited image.

6. The image processing method according to any one of claims 1 to 5, further comprising, after the step of updating the original image to an edited image using an image editing model in response to an editing instruction for the original image, the method includes: Obtain the set of mask images associated with the image editing model; The image editing model is evaluated using the mask image sample data contained in the mask image set to obtain the model evaluation information corresponding to the image editing model; The image editing model is optimized based on the model evaluation information until a target image editing model that meets the optimization stopping condition is obtained.

7. The image processing method according to claim 6, wherein constructing any mask image sample data contained in the mask image set includes: Determine the target editing instruction from the set of mask editing instructions, and determine the target image subset associated with the target editing instruction from multiple image subsets; Extract the image to be edited from the target image subset, and perform masking processing on the image to be edited according to the target editing instructions to obtain a masked image; Based on the image to be edited, the mask image, and the target editing instructions, target mask image sample data is constructed.

8. The image processing method according to claim 7, wherein the construction of the mask editing instruction set comprises: Multiple initial editing instructions are obtained, and each of the multiple initial editing instructions is optimized to obtain multiple intermediate editing instructions; Based on the instruction attribute information corresponding to the multiple intermediate editing instructions, the multiple intermediate editing instructions are divided into a first set of editing instructions and a second set of editing instructions; The first set of editing instructions and the second set of editing instructions are used as the mask editing instruction set; The construction of the plurality of image subsets includes: Multiple sample images are acquired, and the multiple sample images are divided into multiple image subsets according to the image attribute information corresponding to each of the multiple sample images.

9. An image processing method, applied on a server, comprising: Receive an editing instruction carrying the original image submitted by the client, and in response to the editing instruction, update the original image to the edited image using an image editing model; The original image and the edited image are input into the image evaluation model for processing to obtain image evaluation information corresponding to multiple image evaluation dimensions. The edited image is divided into edited and non-edited regions, and the edit evaluation information corresponding to the edited regions and the non-edited evaluation information corresponding to the non-edited regions are determined. Based on the region information of the edited area, the image evaluation information, the edit evaluation information, and the non-edit evaluation information are fused to obtain the target evaluation information corresponding to the edited image; Based on the target evaluation information, the edited image is updated to the target image, and the target image is sent to the client.

10. An image processing apparatus, comprising: The update module is configured to update the original image into an edited image using an image editing model in response to an editing instruction for the original image; The processing module is configured to input the original image and the edited image into the image evaluation model for processing, and obtain image evaluation information corresponding to multiple image evaluation dimensions respectively; The determination module is configured to divide the edited image into edited regions and non-edited regions, and determine the edit evaluation information corresponding to the edited regions and the non-edit evaluation information corresponding to the non-edited regions; The fusion module is configured to fuse the image evaluation information, the editing evaluation information, and the non-editing evaluation information according to the region information of the editing region to obtain the target evaluation information corresponding to the edited image.

11. An image processing apparatus, applied on a server side, comprising: The receiving module is configured to receive an editing instruction carrying the original image submitted by the client, and in response to the editing instruction, update the original image to an edited image using an image editing model; The processing module is configured to input the original image and the edited image into the image evaluation model for processing, and obtain image evaluation information corresponding to multiple image evaluation dimensions respectively; The segmentation module is configured to divide the edited image into edited regions and non-edited regions, and determine the edit evaluation information corresponding to the edited regions and the non-edit evaluation information corresponding to the non-edited regions; The fusion module is configured to fuse the image evaluation information, the editing evaluation information, and the non-editing evaluation information according to the region information of the editing region to obtain the target evaluation information corresponding to the edited image; The sending module is configured to update the edited image to the target image based on the target evaluation information, and send the target image to the client.

12. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 9.

13. A computer-readable storage medium, characterized in that, It stores computer-executable instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 9.

14. A computer program product, characterized in that, It includes a computer program or instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 9.

Images