Text-driven image editing quality evaluation method, device and storage medium

Abstract

The application discloses a text-driven image editing quality evaluation method and device and a storage medium, relates to the technical field of artificial intelligence, and comprises the following steps: inputting first image features of a source image and second image features of an edited image into a source image and edited image interaction module to confirm the connection between the source image and the edited image, and obtaining an image connection score; inputting the second image features into a native quality evaluation module to evaluate the quality of the edited image, and obtaining an edited quality score; inputting text features of edited text and third image features of the edited image into a multi-modal attention interaction module to confirm the consistency between the text and the image, and obtaining a text consistency score; and obtaining an image editing quality score based on the image connection score, the edited quality score and the text consistency score. The application can more comprehensively and accurately reflect the actual effect of image editing by evaluating the image editing quality score from multiple dimensions.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a text-driven image editing quality evaluation method, device and storage medium. BACKGROUND

[0002] After editing the source image through the editing text instruction, text-driven image editing quality evaluation needs to be performed to measure whether the edited image meets the expectation of the image editing of the editing text instruction, and whether the expected change is completed compared with the source image.

[0003] In the existing evaluation method, CLIP-V (CLIP-based visual positioning model) measures the effect of image editing by calculating the cosine similarity between the edited image and the source image. FID (Fréchet Inception Distance) measures the similarity of two images in the feature space by calculating the Fréchet distance between the source image and the edited image. The two indicators of MSE (Mean Squared Error) and SSIM (Structural Similarity Index) represent the variance (MSE) and the overall structural similarity (SSIM) between the edited image and the source image at the pixel level. However, the overall effect of image editing is the result of the joint action of multiple factors, and the existing evaluation method only starts from a single dimension and does not cover the alignment relationship between text and image. It is difficult to comprehensively and comprehensively evaluate the overall effect of image editing, and it is impossible to fully reflect the true situation of image editing quality.

[0004] The above content is only used to assist in understanding the technical solutions of the present application, and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0005] The main purpose of the present application is to provide a text-driven image editing quality evaluation method, device and storage medium, aiming to solve the technical problem of how to comprehensively and accurately evaluate the actual effect of image editing.

[0006] To achieve the above-mentioned purpose, the present application provides a text-driven image editing quality evaluation method, which comprises:

[0007] The first image feature of the source image and the second image feature of the edited image are input into the source image and edited image interaction module to confirm the contact of the source image and the edited image, and the image contact score is obtained.

[0008] The second image feature is input into the original quality evaluation module to evaluate the quality of the edited image, and the editing quality score is obtained.

[0009] inputting the text feature of the edited text and the third image feature of the edited image into a multi-modal attention interaction module to perform consistency confirmation of the text and the image, to obtain a text consistency score;

[0010] obtaining an image editing quality score based on the image association score, the editing quality score and the text consistency score.

[0011] In an embodiment, before the step of inputting the first image feature of the source image and the second image feature of the edited image into a source image and edited image interaction module to perform association confirmation of the source image and the edited image, to obtain an image association score, the method further comprises:

[0012] inputting the source image into a pre-trained first visual backbone network model to obtain the first image feature;

[0013] inputting the edited image into a pre-trained second visual backbone network model to obtain the second image feature.

[0014] In an embodiment, the step of inputting the first image feature of the source image and the second image feature of the edited image into a source image and edited image interaction module to perform association confirmation of the source image and the edited image, to obtain an image association score, comprises:

[0015] concatenating the first image feature and the second image feature in the channel dimension to obtain a first concatenated feature;

[0016] performing feature fusion on the first concatenated feature through a self-attention mechanism and a transformer network to obtain a first fused feature;

[0017] performing linear combination and weighted calculation on the first fused feature based on a pre-set first weight parameter, to obtain the image association score.

[0018] In an embodiment, the step of inputting the second image feature into a native quality evaluation module to perform quality evaluation of the edited image, to obtain an editing quality score, comprises:

[0019] constructing a multi-layer feedforward network composed of linear layers;

[0020] inputting the second image feature into the multi-layer feedforward network to perform regression calculation, to obtain the editing quality score.

[0021] In an embodiment, before the step of inputting the text feature of the edited text and the third image feature of the edited image into a multi-modal attention interaction module to perform consistency confirmation of the text and the image, to obtain a text consistency score, the method further comprises:

[0022] inputting the edited text into a pre-trained text feature extractor to perform feature extraction, to obtain the text feature;

[0023] inputting the edited image into a pre-trained convolutional neural network model to perform feature extraction, to obtain the third image feature.

[0024] In an embodiment, the step of inputting the text feature of the edited text and the third image feature of the edited image into a multi-modal attention interaction module to confirm the consistency of the text and the image, to obtain a text consistency score, comprises:

[0025] performing a flattening operation on the third image feature in a spatial dimension, and performing a flattening operation on the text feature in a length dimension;

[0026] concatenating the third image feature and the text feature after the flattening operation in a channel dimension, to obtain a second concatenated feature;

[0027] performing linear combination and weighting calculation on the second concatenated feature based on a preset second weight parameter, to obtain the text consistency score.

[0028] In an embodiment, the step of obtaining an image editing quality score based on the image relevance score, the editing quality score, and the text consistency score comprises:

[0029] performing a concatenation operation on the image relevance score, the editing quality score, and the text consistency score, to obtain a third concatenated feature;

[0030] performing linear combination and weighting calculation on the third concatenated feature based on a preset third weight parameter, to obtain the image editing quality score.

[0031] In an embodiment, the method further comprises:

[0032] constructing a source image dataset;

[0033] editing a source image in the source image dataset according to a preset edited text instruction set and an image editing method, to obtain a first edited image;

[0034] constructing an edited image dataset according to each first edited image and an evaluation score of the first edited image by an observer.

[0035] In addition, to achieve the above object, the present application also provides a text-driven image editing quality evaluation device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the text-driven image editing quality evaluation method as described above.

[0036] In addition, to achieve the above object, the present application also provides a storage medium, which is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the text-driven image editing quality evaluation method as described above.

[0037] The present application provides a text-driven image editing quality evaluation method, which can measure the image editing quality from multiple different and key dimensions by respectively obtaining an image relevance score, an editing quality score and a text consistency score. The image relevance score focuses on the correlation between the source image and the edited image, the editing quality score focuses on the quality of the edited image itself, and the text consistency score focuses on the degree of fit between the edited text and the edited image at the semantic level. The scores of these dimensions are integrated to obtain the image editing quality score, which avoids the one-sidedness of evaluation from a single angle, can more comprehensively and accurately reflect the actual effect of image editing, and makes the final evaluation result more persuasive. In addition, the way of obtaining different dimension scores in modules and integration is closer to the subjective evaluation idea of human beings, so that the image editing quality score output by the model can better align with human perception, and the rationality and credibility of the evaluation result are improved. BRIEF DESCRIPTION OF DRAWINGS

[0038] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and serve to explain the principles of the present application together with the specification.

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without creative labor.

[0040] Figure 1 A flowchart is provided for the first embodiment of the text-driven image editing quality evaluation method of the present application;

[0041] Figure 2 A general detailed flowchart is provided for the text-driven image editing quality evaluation method of the present application;

[0042] Figure 3A flowchart provided for the second implementation of the text-driven image editing quality evaluation method of the present application text;

[0043] Figure 4 A flowchart provided for the third embodiment of the text-driven image editing quality evaluation method of the present application text;

[0044] Figure 5 A device structure diagram of a hardware operating environment involved in the text-driven image editing quality evaluation method of the embodiments of the present application. DETAILED DESCRIPTION

[0045] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and do not limit the present application.

[0046] In order to better understand the technical solutions of the present application, the following will be described in detail in conjunction with the drawings of the specification and specific embodiments.

[0047] The main solution of the embodiments of the present application is: input the first image feature of the source image and the second image feature of the edited image into the source image and edited image interaction module to confirm the connection of the source image and the edited image, and obtain an image connection score; input the second image feature into the original quality evaluation module to evaluate the quality of the edited image, and obtain an edited quality score; input the text feature of the edited text and the third image feature of the edited image into the multi-modal attention interaction module to confirm the consistency of the text and the image, and obtain a text consistency score; based on the image connection score, the edited quality score and the text consistency score, obtain an image editing quality score.

[0048] After editing the source image through the edited text instruction, text-driven image editing quality evaluation is needed to measure whether the edited image meets the expectations of the edited text instruction for image editing, and whether the edited image and the source image have completed the expected changes, etc.

[0049] In the existing evaluation method, CLIP-V measures the effect of image editing by calculating the cosine similarity between the edited image and the source image. FID measures the similarity of two images in the feature space by calculating the Fréchet distance between the source image and the edited image. The two indicators of MSE and SSIM represent the variance (MSE) and the overall structure similarity (SSIM) between the edited image and the source image at the pixel level. However, the overall effect of image editing is the result of the joint action of many factors, and the existing evaluation method only starts from a single dimension and does not cover the text and image alignment relationship dimension. It is difficult to comprehensively and comprehensively evaluate the overall effect of image editing, and it is impossible to fully reflect the true situation of image editing quality.

[0050] The present application provides a text-driven image editing quality evaluation method, which can measure image editing quality from multiple different and key dimensions by obtaining image association score, editing quality score and text consistency score respectively. Image association score focuses on the association between source image and edited image, editing quality score focuses on the quality of edited image itself, and text consistency score focuses on the degree of fit between edited text and edited image at the semantic level. The scores of these dimensions are integrated to obtain the image editing quality score, avoiding the one-sidedness of evaluation from a single angle, and can more comprehensively and accurately reflect the actual effect of image editing, so that the final evaluation result is more persuasive. In addition, the way of obtaining different dimension scores and integrating them in modules is closer to the subjective evaluation idea of human beings, so that the image editing quality score output by the model can better align with human perception, improving the rationality and credibility of the evaluation result.

[0051] It should be noted that the execution subject of the present embodiment can be a computing service device with network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device, a device, etc. capable of realizing the above functions. The text-driven image editing quality evaluation device is taken as an example to illustrate the present embodiment and the following embodiments.

[0052] Based on this, the present application provides a text-driven image editing quality evaluation method, which is described with reference to Figure 1 , Figure 1 The flowchart of the first embodiment of the text-driven image editing quality evaluation method of the present application is shown in the figure.

[0053] In the present embodiment, the text-driven image editing quality evaluation method comprises steps S100-S400:

[0054] In step S100, the first image features of the source image and the second image features of the edited image are input into the source image and edited image interaction module to confirm the association of the source image and the edited image, and obtain the image association score.

[0055] It should be noted that the source image refers to the original image without editing, and the edited image refers to the image after some editing or processing operation. Such editing can be cropping, adjusting brightness or contrast, adding filters, repairing defects, etc., or more complex operations such as image synthesis, style transfer, etc. The edited image is the result of processing the source image and contains different features or information from the source image.

[0056] In this embodiment, the first image feature of the source image and the second image feature of the edited image are input into the source image and edited image interaction module to confirm the connection between the source image and the edited image and obtain an image connection score. The source image and edited image interaction module is a linear layer and a self-attention layer connected in series, which can dynamically model the source feature and the target feature. The image connection score is usually used to determine whether the edited image still retains the important features of the source image.

[0057] In a feasible implementation, step S100 further includes:

[0058] The source image is input into the pre-trained first visual backbone network model to obtain the first image feature.

[0059] The edited image is input into the pre-trained second visual backbone network model to obtain the second image feature.

[0060] It should be noted that the visual backbone network model ViT (Vision Transformer) is a network model based on the Transformer architecture applied to the visual field. It regards the image as a sequence of image blocks (patches), just like processing the sequence of words in the text. Each image block is converted into a vector representation through linear embedding and other operations, and then input into the encoder of the Transformer. The Transformer encoder contains components such as multi-head attention mechanism, feedforward neural network, and layer normalization. The multi-head attention mechanism can enable the model to focus on the association between different parts of the image from multiple different “perspectives”, and mine the importance of each element in the image and their mutual influence, such as determining the correlation between the main character and the surrounding buildings in an image containing a person and a background building, so as to comprehensively extract image features.

[0061] In this embodiment, the source image and the edited image are input into the quality evaluation model. The source image and the edited image are respectively input into two visual quality encoders for image feature extraction. The visual quality encoder is a visual backbone network ViT pre-trained on a large-scale dataset. The visual backbone network is pre-trained on a large number of image datasets such as ImageNet, which can ensure correct extraction of the representation of the source image and the edited image in the hidden space. The two visual quality encoders have the same structure but do not share parameters.

[0062] In step S200, the second image feature is input into the native quality evaluation module for quality evaluation of the edited image to obtain an edited quality score.

[0063] In this embodiment, the image visual encoder CLIP (Contrastive Language-Image Pre-training) is used to extract the hidden space features of the input edited image (i.e., the second image features) by taking advantage of its priori and excellent multi-modal alignment properties on the pre-trained image quality evaluation dataset. The obtained hidden space features are input into the linear layer of the native quality evaluation module to obtain the evaluation of the native quality dimension. Optionally, the image encoder of CLIP can be a pre-trained ViT or CNN model.

[0064] In this embodiment, the edited images in the edited image dataset, which contains edited images and their corresponding native quality scores, are used for training and verification of the model. The native quality scores are obtained by conducting subjective experiments and collecting human feedback. When using the edited image dataset containing quality scores, CLIP can associate the general visual features of images (such as object shape, color distribution, texture structure, etc.) learned in pre-training with specific quality evaluation information. For example, CLIP may have already known that certain specific texture structures or color combinations are generally related to high-quality images, and in this image quality evaluation dataset, if these features correspond to high native quality scores, CLIP can further strengthen this association, thereby better utilizing its priori knowledge to understand the essential features of image quality.

[0065] In this embodiment, the edited image is input into the image visual encoder of the CLIP model. The image encoder of the CLIP model processes the image. Exemplarily, in the case of ViT as the image encoder, the image is divided into image blocks, after linear projection and position encoding, the feature extraction is performed through the Transformer Encoder, and finally the hidden space feature representation of the image is obtained.

[0066] In step S300, the text features of the edited text and the third image features of the edited image are input into the multi-modal attention interaction module to confirm the consistency of the text and the image, and obtain the text consistency score.

[0067] In this embodiment, the edited text is input into the text encoder of CLIP to obtain text features aligned with visual information, and the edited image is input into the corresponding ResNet50 to obtain visual features aligned with text information (i.e., third image features). Then, the text features and the third image features are input into a multi-modal attention interaction module, which is an IP-IQA-based self-attention and pooling module described below. Based on the self-attention mechanism and the pooling layer, the text features and the visual features are respectively flattened in the spatial dimension (length dimension for the text features) after passing through the Transformer block and then input into the pooling layer. The compressed text features and the visual features are concatenated in the channel dimension by the pooling layer, and finally a linear layer is used for regression to obtain a score measuring the semantic consistency of the text and the edited image.

[0068] In a possible implementation, step S300 can further include the following steps before step S300:

[0069] inputting the edited text into a pre-trained text feature extractor to extract features and obtain the text features;

[0070] inputting the edited image into a pre-trained convolutional neural network model to extract features and obtain the third image features.

[0071] It should be noted that CLIP (Contrastive Language-Image Pretraining) is a multi-modal machine learning model. It aims to learn to understand image content and match these contents with corresponding natural language descriptions by training on a large number of text-image pairs.

[0072] In addition, it should be noted that ResNet50 is a deep convolutional neural network architecture with strong image feature extraction capability.

[0073] In this embodiment, the edited text is input into the text encoder of CLIP, and the input edited text is mapped to a specific vector representation, i.e., text features. In this process, the text encoder converts the information such as words and grammatical structures in the text into feature vectors that can be compared and associated with visual information based on the language semantic understanding and visual information alignment mode learned by the text encoder.

[0074] Exemplarily, for the editing text "change this man into Iron Man", the semantic information in the text is first analyzed, and the semantics implied by key nouns such as "man" and "Iron Man" and the word "change" representing the change of action are converted into corresponding feature vectors. The text encoder will extract the feature vectors that can reflect the key semantic change from the original man to Iron Man according to the language understanding mode learned in the past and the alignment mode with visual information, and these vectors are internally related to the differences in visual presentation between man and Iron Man in reality, for example, the unique armor shape, color and other visual elements of Iron Man will be reflected in the vectors, preparing for subsequent matching with the visual features of the edited image.

[0075] In this embodiment, the edited result image corresponding to the editing text is input into ResNet50 to obtain the visual features aligned with the text information. For the edited result image, ResNet50 gradually extracts different levels of features in the image according to a series of operations such as convolution layers and pooling layers. From low-level edge and texture features to high-level object shape and semantic category related features, etc. will be mined out.

[0076] Exemplarily, for the edited image (presenting the image of Iron Man), ResNet50 starts to exert its powerful image feature extraction capability. Through the convolution layer, the local features at different positions in the image are captured, such as the low-level features of the texture on the Iron Man's armor and the edges of each component. With the deepening of the network, high-level features are gradually integrated through operations such as pooling layers, such as identifying the outline of the entire Iron Man, its iconic energy reactor and other features related to semantic categories, and also extracting the position of Iron Man in the picture and the relative relationship with the background. These extracted visual features can correspond to the "Iron Man" related semantic elements mentioned in the editing text, such as the appearance features of the armor echoing the Iron Man image described in the text, facilitating subsequent matching and consideration with text features.

[0077] In step S400, an image editing quality score is obtained based on the image relevance score, the editing quality score and the text consistency score.

[0078] In this embodiment, the image relevance score, the editing quality score and the text consistency score are input into a multi-dimensional comprehensive regressor. The multi-dimensional comprehensive regressor is a linear layer for splicing features of different dimensions, and the final score is obtained by regression.

[0079] In an optional implementation, step S400 includes:

[0080] The image relevance score, the editing quality score and the text consistency score are spliced to obtain a third spliced feature.

[0081] Based on the preset third weight parameter, the third spliced features are linearly combined and weighted to obtain the image editing quality score.

[0082] In this embodiment, the scores in three different dimensions of image association score, editing quality score and text consistency score are spliced to form a comprehensive feature vector. The spliced feature vector is input into a multi-dimensional comprehensive regressor. After receiving the spliced feature vector, the multi-dimensional comprehensive regressor performs linear combination and weighted calculation on the features in each dimension according to the pre-trained weight parameters, and finally outputs a scalar value, which is the image editing quality score obtained by the multi-dimensional comprehensive regressor.

[0083] Please refer to Figure 2 In this embodiment, the image features of the source image and the edited image are extracted by two visual quality encoders, and the two image features are input into the source image and edited image interaction module to obtain the image association score. The image features of the edited image are input into the native quality evaluation module to obtain the editing quality score. The image features of the edited image extracted by the visual feature extractor and the text features of the edited text extracted by the text feature extractor are input into the multi-modal attention interaction module to obtain the text consistency score. By obtaining the image association score, the editing quality score and the text consistency score, the image editing quality is measured from multiple different and key dimensions, which can more comprehensively and accurately reflect the overall quality level of the edited image.

[0084] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as the above embodiment one can refer to the above introduction, and the subsequent will not be described in detail. On this basis, please refer to Figure 3 The load parameters are the load values of the central processor and the graphics processor, and step S100 can include steps S110-S130:

[0085] Step S110, the first image features and the second image features are spliced in the channel dimension to obtain the first spliced features.

[0086] In this embodiment, when the visual backbone network model ViT extracts the corresponding correlation features from the source image and the edited image respectively, the shape of the feature vector of the first image feature extracted from the source image is [batch_size, num_channels1, height1, width1], wherein batch_size represents the number of images processed at a time, num_channels1 represents the number of channels of the feature, height1 and width1 represent the dimensions of the feature in the spatial dimension. The shape of the feature vector of the second image feature extracted from the edited image is [batch_size, num_channels2, height2, width2]. After the first image feature and the second image feature are extracted, a concatenation operation is performed in the channel dimension, and the two features are combined together along the channel direction to form a new feature vector, and the shape becomes [batch_size, num_channels1+num_channels2, height, width], height and width can take the appropriate spatial dimension size or the size after certain adjustment.

[0087] For example, if the first image feature has 32 channels and the second image feature has 64 channels, a new feature vector with 96 channels will be obtained after concatenation, which integrates the feature information of the source image and the edited image. By integrating the two in the same dimension, it is convenient to input them as a whole to the subsequent network layer for unified processing, to mine the deeper relationship between them and the feature representation after fusion.

[0088] In step S120, the first concatenation feature is fused by a self-attention mechanism and a transformer network to obtain a first fusion feature.

[0089] In this embodiment, the first spliced features obtained after splicing first enter the linear layer. The linear layer is essentially a fully connected neural network layer, which performs linear transformation on the input feature vector, changes the dimension and numerical distribution of the feature vector according to certain rules through the pre-learned weight matrix and bias vector. For example, it can map high-dimensional spliced features to an intermediate-dimensional space, preliminarily integrate and adjust the features, extract feature representations more suitable for subsequent self-attention mechanism processing, and also help the transition and adaptation of features between different network layers, so that the features meet the expected input form of the entire network structure. The features processed by the linear layer then enter the self-attention module. The self-attention mechanism can again focus on the correlation between different elements in the feature vector, and redistribute the weights according to the correlation between the elements, so that the model pays more attention to important parts of the fused features. For example, in the spliced image features, the self-attention mechanism gives higher weights to important feature elements that reflect the changes in the main body of the image, the differences in the key areas, etc., further highlights these key information, strengthens the fusion effect between features, and enables the model to better capture the correlation and changes between the source image and the edited image.

[0090] In this embodiment, the features output by the self-attention module are further processed by the forward network. The forward network is usually composed of multiple fully connected layers, activation functions, etc., which can perform nonlinear transformation on the features adjusted by the self-attention, further explore the complex relationships between the features, and refine and enrich the representation of the fused features. The final fused features can fully contain valuable information generated during the fusion process of the source image and the edited image, such as similarity and differences between them, which is crucial for accurately measuring the dynamic modeling situation subsequently.

[0091] In step S130, the first fusion features are linearly combined and weighted based on the first preset weight parameter, and the image contact score is obtained.

[0092] In this embodiment, the fused features obtained through the previous steps are input into the last linear layer for linear transformation operation to map the fused features to a specific numerical value, which is the final dynamic modeling score. The linear layer calculates the input fused features according to the weight parameters learned in the training process and compresses them into a scalar value. For example, the score can be set within a certain interval, such as between 0 and 1. 0 may indicate that there is little expected dynamic correlation or consistency between the source image and the edited image, and 1 indicates that there is a very high dynamic correlation between them, that is, the edited image is highly consistent with the source image in dynamic changes, etc. The intermediate values correspond to different degrees of dynamic modeling, reflecting the consistency between the source image and the edited image in dynamic features.

[0093] In this embodiment, the ViT is used to extract image features, the image features are spliced and fused, and a dynamic modeling score is obtained through linear regression, thereby realizing quantitative analysis and evaluation of the correlation characteristics between the source image and the edited image.

[0094] Based on the first embodiment of the present application, the same or similar contents as the above embodiment one can be referred to the above introduction, and will not be described in detail hereinafter. On this basis, please refer to Figure 3 , step S200 can include steps S210-S220:

[0095] Step S210, a multi-layer forward network composed of linear layers is constructed.

[0096] Step S220, inputting the second image feature into the multi-layer forward network for regression calculation to obtain the editing quality score.

[0097] In this embodiment, a multi-layer forward network composed of linear layers is constructed, the input of the network is the second image feature extracted by the CLIP model, and the output is the predicted native quality evaluation. The number of layers and the number of neurons of each layer of the network can be designed according to the actual situation, for example, 2-3 layers of linear layers can be used, and ReLU or other activation functions can be used in the middle layer to increase the nonlinear expression ability of the network.

[0098] In this embodiment, a suitable loss function is selected to measure the difference between the predicted native quality and the true quality label. For example, a mean square error (MSE) loss function is selected, which is used to calculate the average square error between the predicted value and the true value in the regression task, to guide the training process of the model, so that the model can learn accurate quality evaluation prediction ability. The editing image dataset is divided into a training set and a validation set. In the training process, for each batch of images in the training set, the feature is extracted by inputting the CLIP model, and then the feature is input into the multi-layer forward network to obtain the predicted quality. The value of the loss function is calculated according to the predicted quality and the true quality label. The parameters of the multi-layer forward network are updated according to the gradient, such as stochastic gradient descent, Adam, etc., and the learning rate and other hyperparameters are adjusted to optimize the model training process, so that the model gradually converges to a better performance state. After each training period, the performance of the model is evaluated using the validation set, for example, the loss value or other related indicators (such as mean absolute error, etc.) on the validation set are calculated, to monitor whether the model is overfitting or underfitting, and the training strategy is adjusted according to the evaluation results, such as adjusting the learning rate, increasing or decreasing the number of training rounds, etc. After the training is completed, the model is comprehensively evaluated using an independent test set, various evaluation indicators (such as mean square error, determination coefficient, etc.) are calculated, to measure the accuracy and generalization ability of the model in predicting the native quality.

[0099] Based on the first embodiment of the present application, in the fourth embodiment of the present application, the same or similar contents as the above embodiment one can be referred to the above introduction, and the subsequent will not be described. On this basis, please refer to Figure 3 , step S300 can include steps S310-S330:

[0100] Step S310, the third image feature is flattened in the spatial dimension, and the text feature is flattened in the length dimension.

[0101] In this embodiment, the text feature and the visual feature are further converted and strengthened by the Transformer block. The Transformer block has a multi-head attention mechanism, a feedforward neural network, etc. inside. The multi-head attention mechanism can capture the relationship between features from multiple angles, further refining and optimizing the feature representation. After the Transformer block, the text feature and the visual feature have been more effectively organized and refined in terms of semantic expression. Secondly, the pooling layer operation is performed. The features after the Transformer block are flattened in the spatial dimension (the length dimension for the text feature). The multi-dimensional feature matrix is converted into a one-dimensional vector form, and then input into the pooling layer. The pooling layer usually uses methods such as maximum pooling or average pooling. For example, maximum pooling selects the maximum value in each local region of the feature vector as the output, which can further compress the features, extract the most representative feature information, reduce the data volume while retaining the key semantic features, and facilitate subsequent splicing and calculation.

[0102] Step S320, the third image feature and the text feature after the flattening operation are spliced in the channel dimension to obtain a second spliced feature.

[0103] In this embodiment, the text feature and the visual feature are spliced in the channel dimension. The text feature and the visual feature compressed by the pooling layer are spliced in the channel dimension. For example, if the text feature is a vector of shape [1, n] after the previous processing (n is the feature length), and the visual feature is a vector of shape [1, m] (m is the feature length), they are spliced into a new vector of shape [1, n+m] along the channel dimension. In this way, the key semantic features of text and image are fused together to form a comprehensive feature representation, which contains the core information of text description and image presentation, facilitating subsequent unified regression analysis to measure the semantic consistency of the two.

[0104] Step S330, based on the second preset weight parameter, the second spliced feature is linearly combined and weighted to obtain the text consistency score.

[0105] In this embodiment, the spliced comprehensive feature vector is input into a linear layer. The linear layer is actually a simple fully connected neural network layer, which performs linear transformation on the input feature vector according to the pre-learned weight parameters, maps it to a specific numerical range, and the numerical value is the final score representing the consistency of the text-edit image semantics. For example, if the linear layer outputs a score between 0 and 1 after training, 0 may represent that the text and image semantics are completely inconsistent, 1 represents complete consistency, and the intermediate values correspond to different degrees of semantic consistency. The model learns appropriate weights by training on a large number of annotated text-image pair data to accurately give a reasonable consistency score according to the input features. Using the mapping ability of the linear layer to output a specific measurement score can intuitively reflect the degree of semantic consistency between the text and the edited image.

[0106] Based on the first embodiment of the present application, in the fifth embodiment of the present application, the same or similar contents as the above embodiment one can refer to the above introduction, and the subsequent will not be described in detail. On this basis, please refer to Figure 4 Before step S100, steps S01-S03 can be included:

[0107] Step S01, constructing a source image dataset;

[0108] In this embodiment, during the source image construction process, the content of the source image dataset includes but is not limited to real-world scenes, computer graphics rendering images, text-driven generated images, and artistic works. Considering the current wide range of application scenarios, real-world scenes account for a large proportion. When collecting source images, considering copyright, watermarking, resolution and other issues, this embodiment selects several images from four datasets: ADE20k dataset, WIKIArt (WIKIArt Dataset), COCO (Common Objects in Context), ReasonEdit, and other network resources.

[0109] In this embodiment, when collecting source images from the above four data sets, first confirm the data source (real world data set, computer graphics, AIGC (Artificial Intelligence Generated Content), and artistic works), then check each sample one by one, and label attributes such as "[landscape / object / animal / human]" and "[action type]". Samples of the same type (for example, landscape / object / animal or human action) or insufficient resolution are skipped until the data set is fully checked or the number of samples of the relevant category is sufficient. For smaller data sets, prefer to check, and more samples are extracted from larger data sets. Finally, suitable images are selected from the Internet for supplementation. For example, although many data sets contain landscapes such as grasslands and snow-capped mountains, scenes such as auroras, lava flows, and lightning are less common. Similarly, although the current data set provides a rich variety of action categories, there is still a certain lack in terms of significantly varying action patterns.

[0110] Step S02, according to the preset editing text instruction set and image editing method, the source image in the source image data set is edited, and the first edited image is obtained;

[0111] In this embodiment, in order to ensure the specificity and diversity of the instructions, corresponding text instructions are generated for each source image. The editing text instructions can be divided into three categories: (1) style editing, including color, texture or overall atmosphere editing; (2) semantic editing, including background editing and local editing, such as adding, replacing or removing specific objects; (3) structure editing, including changes in object size, posture, action, etc.

[0112] In this embodiment, in order to ensure the distribution of the quality of the edited images, different image editing methods are selected to edit the source images in the source image data set. The image editing method can be constructed by the latest deep learning technology and a large amount of training data, aiming to provide a performance model with high-quality image editing effect.

[0113] Exemplarily, in this embodiment, the editing methods of different base models are selected, covering models from SD (Stable Diffusion) 1-4 to SD 2-1, to improve the diversity of the editing results. In addition, in order to ensure the diversity of the edited content, 0-shot methods and methods requiring fine-tuning are selected. Models are also selected according to different editing paradigms, including effective editing strategies such as Instruct-P2P (Instruct Point-to-Point), Prompt-to-prompt, MasaCtrl (Tuning-Free Mutual Self-Attention Control), etc. Instruct-P2P is a technology that precisely controls specific regions or elements in an image through instructions for point-to-point editing. The Prompt-to-prompt strategy allows users to gradually adjust the content of generated images by modifying prompts, providing a flexible editing method. MasaCtrl is a method for image synthesis and editing, which converts the self-attention in diffusion models to mutual self-attention, achieving consistent image generation without fine-tuning and complex non-rigid image editing. It can query relevant local content and texture from the source image to ensure the consistency of the generated image.

[0114] Step S03, constructing an edited image dataset according to each of the first edited images and the evaluation scores of the first edited images by the observer.

[0115] In this embodiment, a subjective experiment is conducted to collect human feedback to construct an edited image dataset with human feedback. According to the ITU standard (International Telecommunication Union Standards), the number of participants in the subjective experiment should be at least 15 to ensure that the variance of the results is within a controllable range. Exemplarily, 25 experiment participants with diverse backgrounds are recruited. During the experiment, participants are required to evaluate the consistency of text and image, the fidelity of source image and edited image, and the quality of edited image according to subjective impressions. All participants are over 18 years old and have at least a bachelor's degree, covering multiple fields such as business, engineering, science, law, etc., and have the ability to make independent judgments. Before the experiment begins, all participants undergo face-to-face training, during which excellent and poor editing examples not included in the dataset are shown. In the experiment, each participant evaluates all video samples, and is forced to take a 5-minute break every 15 minutes of work to avoid fatigue.

[0116] In this embodiment, human feedback is collected by performing subjective experiments to construct an edited image dataset with human feedback, so that the subjective evaluation of humans can be aligned with machine evaluation, and the reliability and practicality of the evaluation results can be improved.

[0117] The present application provides a text-driven image editing quality evaluation device, which comprises at least one processor and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the text-driven image editing quality evaluation method in Embodiment I.

[0118] Reference will be made to the following Figure 5 which shows a structural schematic diagram of a text-driven image editing quality evaluation device suitable for implementing the embodiments of the present application. The text-driven image editing quality evaluation device in the embodiments of the present application can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistant), PADs (portable android devices), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. Figure 5 The text-driven image editing quality evaluation device shown is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present application.

[0119] As Figure 5As shown, the text-driven image editing quality evaluation device can include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. In the RAM 1004, various programs and data required for the operation of the text-driven image editing quality evaluation device are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the text-driven image editing quality evaluation device to communicate with other devices wirelessly or by wire to exchange data. Although the text-driven image editing quality evaluation device with various systems is shown in the figure, it should be understood that all the shown systems are not required to be implemented or possessed. More or less systems can be alternatively implemented or possessed.

[0120] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are performed.

[0121] The text-driven image editing quality evaluation device provided in the application adopts the text-driven image editing quality evaluation method in the above embodiment, and can solve the technical problem of how to improve the comprehensive and accurate evaluation of the actual effect of image editing on long texts. Compared with the prior art, the text-driven image editing quality evaluation device provided in the application has the same beneficial effects as the text-driven image editing quality evaluation method provided in the above embodiment, and other technical features in the text-driven image editing quality evaluation device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0122] It should be understood that parts of the present application can be realized by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0123] The above is merely specific implementation of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0124] The present application provides a computer readable storage medium having computer readable program instructions (i.e. computer programs) stored thereon, the computer readable program instructions being used to execute the text-driven image editing quality evaluation method in the above embodiment.

[0125] The computer readable storage medium provided in the application may be, for example, a U disk, but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination of the above. More specific examples of the computer readable storage medium may include, but are not limited to, an electrical connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the computer readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to electrical wires, optical cables, RF (Radio Frequency), and the like, or any suitable combination of the above.

[0126] The above computer readable storage medium may be included in the text-driven image editing quality evaluation device, or may exist separately without being assembled into the text-driven image editing quality evaluation device.

[0127] The above computer readable storage medium carries one or more programs, when the one or more programs are executed by the text-driven image editing quality evaluation device, the text-driven image editing quality evaluation device is caused to: input first image features of a source image and second image features of an edited image to a source image and edited image interaction module to confirm the association of the source image and the edited image, and obtain an image association score; input the second image features to a native quality evaluation module to evaluate the quality of the edited image, and obtain an edited quality score; input text features of an edited text and third image features of the edited image to a multi-modal attention interaction module to confirm the consistency of the text and the image, and obtain a text consistency score; and obtain an image editing quality score based on the image association score, the edited quality score, and the text consistency score.

[0128] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0129] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0130] The modules involved in the embodiments of the present application can be implemented in the form of software or in the form of hardware. In some cases, the name of the module does not constitute a limitation on the module itself.

[0131] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e., computer programs) for executing the text-driven image editing quality evaluation method described above, and can solve the technical problem of how to improve the comprehensive and accurate evaluation of the actual effect of image editing on long text. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the text-driven image editing quality evaluation method provided by the above-mentioned embodiments, and will not be described here.

[0132] The above merely provides part of embodiments of the present application, and does not limit the patent scope of the present application. Any equivalent structure transformation, direct / indirect application in other related technical fields based on the technical concept of the present application, and contents of the present application specification and drawings are included in the patent protection scope of the present application.

Claims

1. A text-driven image editing quality evaluation method, characterized by, The method comprises: The first image feature of the source image and the second image feature of the edited image are input into a source image and edited image interaction module to confirm the connection of the source image and the edited image, and an image connection score is obtained, comprising: the first image feature and the second image feature are spliced in the channel dimension to obtain a first spliced feature; the first spliced feature is fused through a self-attention mechanism and a forward network to obtain a first fused feature; the first fused feature is linearly combined and weighted based on a preset first weight parameter to obtain the image connection score; The second image feature is input into a native quality evaluation module to evaluate the quality of the edited image, and an edited quality score is obtained, comprising: a multi-layer forward network composed of linear layers is constructed; the second image feature is input into the multi-layer forward network for regression calculation to obtain the edited quality score; The text feature of the edited text and the third image feature of the edited image are input into a multi-modal attention interaction module to confirm the consistency of the text and the image, and a text consistency score is obtained, comprising: the third image feature is flattened in the spatial dimension, and the text feature is flattened in the length dimension; the third image feature and the text feature after the flattening operation are spliced in the channel dimension to obtain a second spliced feature; the second spliced feature is linearly combined and weighted based on a preset second weight parameter to obtain the text consistency score; Based on the image connection score, the edited quality score and the text consistency score, an image editing quality score is obtained.

2. The text-driven image editing quality assessment method of claim 1, wherein, Before the step of inputting the first image feature of the source image and the second image feature of the edited image into the source image and edited image interaction module to confirm the connection of the source image and the edited image, and obtaining an image connection score, it further comprises: The source image is input into a pre-trained first visual backbone network model to obtain the first image feature; The edited image is input into a pre-trained second visual backbone network model to obtain the second image feature.

3. The text-driven image editing quality assessment method of claim 1, wherein, Before the step of inputting the text feature of the edited text and the third image feature of the edited image into the multi-modal attention interaction module to confirm the consistency of the text and the image, and obtaining a text consistency score, it further comprises: The edited text is input into a pre-trained text feature extractor for feature extraction to obtain the text feature; The edited image is input into a pre-trained convolutional neural network model for feature extraction to obtain the third image feature.

4. The text-driven image editing quality assessment method of claim 1, wherein, The step of obtaining an image editing quality score based on the image connection score, the edited quality score and the text consistency score comprises: The image connection score, the edited quality score and the text consistency score are spliced to obtain a third spliced feature; The third spliced feature is linearly combined and weighted based on a preset third weight parameter to obtain the image editing quality score.

5. The text-driven image editing quality assessment method of claim 1, wherein, Before the step of inputting the first image feature of the source image and the second image feature of the edited image into the source image and edited image interaction module to confirm the association of the source image and the edited image and obtain an image association score, the method further comprises: constructing a source image dataset; editing the source images in the source image dataset according to a preset edited text instruction set and an image editing method to obtain first edited images; constructing an edited image dataset according to each of the first edited images and an evaluation score of the first edited images by an observer.

6. A text-driven image-editing quality evaluation device characterized by comprising: The device comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the text-driven image editing quality evaluation method according to any one of claims 1 to 5.

7. A storage medium, characterized by The storage medium is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the text-driven image editing quality evaluation method according to any one of claims 1 to 5.

Images