Quality evaluation method and device based on text map and related product

Abstract

The invention provides a text-based graph quality evaluation method and device and a related product, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring a target image generated by a text graph network model according to text description of a target text; determining a generation efficiency qualitative score, a logic consistency qualitative score, a visual consistency qualitative score, a semantic consistency qualitative score, a content consistency qualitative score and other text generation graph qualitative scores when the test object generates a target image according to the target text for the text generation graph network model; obtaining a first image quality quantitative score, a second image quality quantitative score, an image-text consistency quantitative score and other text graph quantitative scores of the target image; and performing weighted summation processing on the qualitative score of the text graph and the quantitative score of the text graph, and determining the image quality of a target image generated by the text graph network model according to the text description of the target text. According to the embodiment of the invention, the quality of the image generated by the text graph can be accurately evaluated.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a method, device, electronic device, computer-readable storage medium and computer program product for evaluating the quality of a text-based image. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the disclosure that are recited in the claims. No description herein is admitted to be prior art by inclusion in this section.

[0003] Generating images from text is a key technology in the framework of artificial intelligence (AIGC). Generating images from text uses a large multimodal model to automatically generate corresponding image content based on the input text description. It can be used in many fields such as art creation, education, game development, virtual reality, smart home, etc., bringing inspiration and convenience to designers and users.

[0004] However, the complexity of Chinese has brought challenges to the development of text-based image technology. Currently, there is no unified evaluation system to accurately evaluate and position a large number of text-based image effects. Summary of the invention

[0005] The purpose of the present disclosure is to provide a method, device, electronic device, computer-readable storage medium and computer program product for evaluating the quality of a target image generated by the Vincent image technology, which can accurately evaluate the image quality of the target image generated by the Vincent image technology.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] The disclosed embodiment provides a method for evaluating the quality of text-based graphs, including: obtaining a target text and a target image, wherein the target image is an image generated by a text-based graph network model according to a text description of the target text; obtaining a qualitative score of at least one of the generation efficiency, logical consistency, visual consistency, semantic consistency, and content consistency when the text-based graph network model generates the target image according to a target rule by an evaluation subject, and obtaining at least one of the qualitative score of generation efficiency, the qualitative score of logical consistency, the qualitative score of visual consistency, the qualitative score of semantic consistency, and the qualitative score of content consistency; and scoring the naturalness of the target image by a first image quality scoring method. A first image quality quantitative score is obtained; the authenticity and diversity of the target image are scored by a second image quality scoring method to obtain a second image quality quantitative score; the consistency of the target text and the target image is quantitatively scored by a third image quality scoring method to obtain an image-text consistency quantitative score; the first image quality quantitative score, the second image quality quantitative score and the image-text consistency quantitative score are weighted summed to obtain a Vincent graph quantitative score; the Vincent graph qualitative score and the Vincent graph quantitative score are weighted summed to determine the image quality of the target image generated by the Vincent graph network model according to the text description of the target text.

[0008] In some embodiments, the naturalness of the target image is scored by a first image quality scoring method to obtain a first image quality quantitative score, including: extracting multivariate Gaussian model parameters of natural statistical features from the target image as first model parameters; extracting multivariate Gaussian model parameters of natural statistical features from the natural image as second model parameters; determining a target distance between the first model parameters and the second model parameters; if the target distance is less than a distance threshold, the first image quality quantitative score is higher than the first score; if the target distance is greater than or equal to the distance threshold, the first image quality quantitative score is lower than or equal to the first score.

[0009] In some embodiments, extracting multivariate Gaussian model parameters of natural statistical features from the target image as first model parameters includes: performing block processing on the target image to obtain multiple image blocks; determining a target image block whose sharpness is greater than a preset threshold among the multiple image blocks; acquiring Gaussian features of the target image block; and determining the first model parameters of the target image based on the Gaussian features of the target image block.

[0010] In some embodiments, the multiple image blocks include a first image block; wherein, determining a target image block having a sharpness greater than a preset threshold among the multiple image blocks includes: obtaining a local variance corresponding to each pixel in the first image block; adding the local variance of each pixel in the first image block to determine the sharpness of the first image block; if the sharpness of the first image block is greater than the preset threshold, taking the first image block as the target image block.

[0011] In some embodiments, obtaining the Gaussian features of the target image block includes: determining the local mean and local variance corresponding to each pixel in the target image block; subtracting the local mean of the pixel from the pixel in the target image block, and then dividing the subtraction by the local variance of the pixel to obtain the spatial natural statistical features of the pixel; modeling the spatial natural statistical features of each pixel in the target image block by a generalized Gaussian distribution or by a symmetric generalized Gaussian distribution to obtain the Gaussian features of the target image block.

[0012] In some embodiments, the authenticity and diversity of the target image are scored by a second image quality scoring method to obtain a second image quality quantitative score, including: classifying the target image by an image classification model to determine the probability distribution of the target image in each category; determining the KL divergence of each probability distribution; and determining the exponential average of the KL divergence of each probability distribution as the second image quality quantitative score.

[0013] In some embodiments, the consistency between the target text and the target image is quantitatively scored by a third image quality scoring method to obtain an image-text consistency quantitative score, including: determining a text feature vector of the target text; determining an image feature vector of the target image; and determining the image-text consistency quantitative score based on the distance between the text feature vector and the image feature vector.

[0014] In some embodiments, a third image quality scoring method is used to quantitatively score the consistency between the target text and the target image to obtain an image-text consistency quantitative score, including: generating an attribute question based on the attribute information in the target text; determining the answer to the attribute question based on the target image; and determining the image-text consistency quantitative score based on the answer.

[0015] In some embodiments, the consistency between the target text and the target image is quantitatively scored by a third image quality scoring method to obtain an image-text consistency quantitative score, including: determining the number of objects in the target text and the relative positions of each object; determining the number of objects in the target image and the relative positions of each object; determining the image-text consistency quantitative score based on the number of objects in the target text and the relative positions of each object, and the number of objects in the target image and the relative positions of each object.

[0016] The embodiment of the present disclosure provides a device for evaluating the quality of a Vincent image, comprising: an image acquisition module, a qualitative score determination module, a first image quality quantitative score determination module, a second image quality quantitative score determination module, a consistency quantitative score determination module, a Vincent image quantitative score determination module, and an image quality determination module.

[0017] Among them, the image acquisition module is used to acquire the target text and the target image, and the target image is an image generated by the text graph network model according to the text description of the target text; the qualitative score determination module can be used to obtain the evaluation object to perform a qualitative score on at least one of the generation efficiency, logical consistency, visual consistency, semantic consistency and content consistency when the text graph network model generates the target image according to the target text according to the target rule, and obtain at least one of the qualitative score of generation efficiency, the qualitative score of logical consistency, the qualitative score of visual consistency, the qualitative score of semantic consistency and the qualitative score of content consistency; the first image quality quantitative score determination module can be used to score the naturalness of the target image by a first image quality scoring method to obtain a first image quality quantitative score; the second image quality quantitative score The scoring determination module can be used to score the authenticity and diversity of the target image by a second image quality scoring method to obtain a second image quality quantitative score; the consistency quantitative score determination module can be used to quantitatively score the consistency between the target text and the target image by a third image quality scoring method to obtain an image-text consistency quantitative score; the Vincent graph quantitative score determination module can be used to perform weighted sum processing on the first image quality quantitative score, the second image quality quantitative score and the image-text consistency quantitative score to obtain a Vincent graph quantitative score; the image quality determination module can be used to perform weighted sum processing on the Vincent graph qualitative score and the Vincent graph quantitative score to determine the image quality of the target image generated by the Vincent graph network model based on the textual description of the target text.

[0018] An embodiment of the present disclosure provides an electronic device, which includes: a memory and a processor; the memory is used to store computer program instructions; the processor calls the computer program instructions stored in the memory to implement any of the above-mentioned methods for evaluating the quality of text-generated images.

[0019] The embodiment of the present disclosure provides a computer-readable storage medium having computer program instructions stored thereon for implementing any of the above-mentioned methods for evaluating the quality of text-generated images.

[0020] The embodiment of the present disclosure provides a computer program product or a computer program, which includes computer program instructions stored in a computer-readable storage medium. The computer program instructions are read from the computer-readable storage medium, and a processor executes the computer program instructions to implement the above-mentioned text-generated image quality assessment method.

[0021] The text-generated image quality evaluation method, device, electronic device, computer-readable storage medium and computer program product provided by the embodiments of the present disclosure firstly use the qualitative evaluation dimension to score by professional evaluators according to the scoring criteria, covering the five dimensions of content consistency, semantic consistency, visual consistency, logical consistency and generation efficiency, and according to the different emphasis in different situations, the above dimensions are comprehensively evaluated using adaptive weight coefficients, and finally the total score of the qualitative dimension is formed, so as to ensure that the generated image is scored more reasonably in each dimension; secondly, the image quality and text-image consistency are integrated through the quantitative evaluation dimension, further reducing the deviation caused by a single indicator, and similarly, according to the different emphasis in different situations, the above dimensions are comprehensively evaluated using adaptive weight coefficients, and finally the total score of the quantitative dimension is formed, so as to ensure that the generated image is scored more reasonably in each dimension. Through the above method, the target image generated by the target model through the text-generated image technology can be accurately and fairly evaluated from quantitative and qualitative dimensions.

[0022] It is to be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0024] Figure 1A schematic diagram of a scenario that can be applied to a method or device for evaluating quality of a text-generated image in an embodiment of the present disclosure is shown.

[0025] Figure 2 The present invention is a flowchart of a method for evaluating the quality of a text-generated image according to an exemplary embodiment.

[0026] Figure 3 The figure is a flowchart of a method for determining a first image quality quantitative score according to an exemplary embodiment.

[0027] Figure 4 is a flowchart of a method for determining a first model parameter according to an exemplary embodiment.

[0028] Figure 5 The figure is a flowchart of a method for determining a target image block according to an exemplary embodiment.

[0029] Figure 6 The figure is a flow chart of a method for determining Gaussian features according to an exemplary embodiment.

[0030] Figure 7 The figure is a flowchart of a method for determining a second image quality quantitative score according to an exemplary embodiment.

[0031] Figure 8 The present invention is a flowchart of a method for determining a quantitative score of image-text consistency according to an exemplary embodiment.

[0032] Fig. 9 The present invention is a flowchart of a method for determining a quantitative score of image-text consistency according to an exemplary embodiment.

[0033] Fig.10 The present invention is a flowchart of a method for determining a quantitative score of image-text consistency according to an exemplary embodiment.

[0034] Fig.11 The present invention is a flowchart of a method for evaluating the quality of a text-generated image according to an exemplary embodiment.

[0035] Fig.12 It is a schematic diagram of a multimodal Chinese literary image evaluation dimension according to an exemplary embodiment.

[0036] Fig.13 The invention is a block diagram of a device for evaluating the quality of a text-generated image according to an exemplary embodiment.

[0037] Fig.14 A schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0038] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0039] Those skilled in the art will appreciate that the embodiments of the present disclosure may be a system, device, apparatus, method or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0040] The features, structures or characteristics described in the present disclosure may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0041] In the embodiments of the present disclosure, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0042] The accompanying drawings are only schematic diagrams of the present disclosure, and the same reference numerals in the drawings represent the same or similar parts, so their repeated description will be omitted. Some block diagrams shown in the accompanying drawings do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0043] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and steps, nor must they be executed in the order described. For example, some steps can be decomposed, and some steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0044] In the description of the present disclosure, unless otherwise specified, " / " means "or", for example, A / B can mean A or B. "And / or" in this article is only a kind of association relationship describing the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, "at least one" means one or more, and "plurality" means two or more. The words "first", "second", etc. do not limit the quantity and execution order, and the words "first", "second", etc. do not limit them to be different; the terms "comprising", "including" and "having" are used to express the meaning of open inclusion and mean that in addition to the listed elements / components / etc., there may be other elements / components / etc.

[0045] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0046] First, some terms involved in the embodiments of the present disclosure are explained below to facilitate understanding by those skilled in the art.

[0047] Multimodal big model: Big model is a machine learning technology based on deep learning, with a large number of parameters and complex structure, which can handle more complex tasks and data. The emergence of multimodal big model is based on big model. It can process multiple types of information such as text, pictures, audio and video at the same time, has the ability to understand rich pictures, texts and sounds, has a high degree of integration with the real world, is more in line with the way humans receive, process and express information, interacts with humans more flexibly, behaves more intelligently, and can perform a wider range of tasks.

[0048] Multimodal large model evaluation: Multimodal large model evaluation is a complex process that aims to comprehensively evaluate the performance, generalization ability, robustness and performance of the model in actual application scenarios when processing multiple types of data (such as text, images and videos). It is a process involving a wide range of dimensions to ensure that the model is not only technically advanced, but also reliable, efficient and ethical in practical applications.

[0049] AIGC (Generative Artificial Intelligence): refers to the technology based on artificial intelligence technical methods such as generative adversarial networks and large pre-trained models, which generates relevant content with appropriate generalization capabilities through learning and recognition of existing data. The core idea of ​​AIGC technology is to use artificial intelligence algorithms to generate content with certain creativity and quality.

[0050] Generating images from text: Generating images from text is a key technology in the AIGC framework. It can automatically generate and display corresponding image content based on the input text description. This technology combines the progress of natural language processing, computer vision and generative models, and aims to achieve automatic conversion of text to image.

[0051] MVG (Multivariate Gaussian) Model: Multivariate (variable) Gaussian model.

[0052] NSS (Natural Scene Statistic): Natural statistical characteristics.

[0053] The foregoing introduces some terms and concepts involved in the embodiments of the present disclosure. The following introduces the technical features involved in the embodiments of the present disclosure.

[0054] With the emergence of more and more Chinese multimodal large-scale model text-based image capabilities, there is currently no corresponding evaluation standard and a sound evaluation system. Therefore, this paper proposes a multimodal Chinese text-based image evaluation system.

[0055] The exemplary embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0056] Figure 1 A schematic diagram of a scenario that can be applied to a method or device for evaluating quality of a text-generated image in an embodiment of the present disclosure is shown.

[0057] Please refer to Figure 1 , which shows a schematic diagram of an implementation environment provided by an exemplary embodiment of the present disclosure.

[0058] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0059] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, wearable devices, virtual reality devices, smart homes, etc.

[0060] The server 105 may be a server that provides various services, such as a background management server that provides support for devices operated by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received request data, and feed back the processing results to the terminal device.

[0061] The server can be an independent physical server or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as basic cloud computing services such as big data and artificial intelligence platforms, etc. The present disclosure does not impose any restrictions on this.

[0062] The server 105 may, for example, obtain a target text and a target image, where the target image is an image generated by the text-based graph network model according to the text description of the target text; the server 105 may, for example, obtain a qualitative score of at least one of the generation efficiency, logical consistency, visual consistency, semantic consistency, and content consistency of the target image generated by the text-based graph network model according to the target text by the evaluation subject according to the target rule, and obtain at least one of the qualitative score of generation efficiency, the qualitative score of logical consistency, the qualitative score of visual consistency, the qualitative score of semantic consistency, and the qualitative score of content consistency; the server 105 may, for example, score the naturalness of the target image by a first image quality scoring method, and obtain a first image quality quantitative score ; The server 105 may, for example, score the authenticity and diversity of the target image by a second image quality scoring method to obtain a second image quality quantitative score; The server 105 may, for example, quantitatively score the consistency between the target text and the target image by a third image quality scoring method to obtain an image-text consistency quantitative score; The server 105 may, for example, perform a weighted summation process on the first image quality quantitative score, the second image quality quantitative score, and the image-text consistency quantitative score to obtain a Vincent graph quantitative score; The server 105 may, for example, perform a weighted summation process on the Vincent graph qualitative score and the Vincent graph quantitative score to determine the image quality of the target image generated by the Vincent graph network model according to the text description of the target text.

[0063] It should be understood that Figure 1 The number of terminal devices, networks and servers in the figure is only for illustration. The server 105 may be a physical server or may be composed of multiple servers. According to actual needs, it may have any number of terminal devices, networks and servers.

[0064] Figure 2is a flow chart of a method for evaluating the quality of a text-generated image according to an exemplary embodiment. The method provided by the embodiment of the present disclosure can be executed by any electronic device with computing processing capabilities. For example, the method can be executed by the above Figure 1 The server or terminal device in the embodiment may be executed, or the server and the terminal device may be executed together. In the following embodiments, the server is used as an example for illustration, but the present disclosure is not limited to this.

[0065] Reference Figure 2 The method for evaluating the quality of a text-generated image provided by the embodiment of the present disclosure may include the following steps.

[0066] Step S202, obtaining a target text and a target image, where the target image is an image generated by the text-graph network model according to the text description of the target text.

[0067] In some embodiments, the above-mentioned text graph network model can be a multimodal large model, a common large model, or a small model with fewer parameters, etc., and this application does not impose any restrictions on this.

[0068] Step S204, obtaining a qualitative score of at least one of the generation efficiency, logical consistency, visual consistency, semantic consistency and content consistency of the text-graph network model when generating a target image from the target text according to the target rule, and obtaining at least one of the qualitative score of generation efficiency, the qualitative score of logical consistency, the qualitative score of visual consistency, the qualitative score of semantic consistency and the qualitative score of content consistency.

[0069] In some embodiments, the evaluation subject may be asked to perform a qualitative evaluation on at least one of the generation efficiency, logical consistency, visual consistency, semantic consistency, and content consistency when generating a target image from a target text according to certain evaluation rules.

[0070] Step S206: Score the naturalness of the target image using a first image quality scoring method to obtain a first image quality quantitative score.

[0071] In some embodiments, the naturalness of the target image may be scored by an image processing method to determine whether the target image is natural.

[0072] For example, the naturalness of the target image can be evaluated by natural statistical features.

[0073] Step S208: Score the authenticity and diversity of the target image using a second image quality scoring method to obtain a second image quality quantitative score.

[0074] In some embodiments, the authenticity and diversity of the target image may be evaluated by an image processing method.

[0075] Step S210: quantitatively score the consistency between the target text and the target image using a third image quality scoring method to obtain an image-text consistency quantitative score.

[0076] In some embodiments, the consistency between the target text and the target image may be quantitatively scored by an image processing method to obtain an image-text consistency quantitative score.

[0077] Step S212: performing weighted sum processing on the first image quality quantitative score, the second image quality quantitative score, and the image-text consistency quantitative score to obtain a Vincent image quantitative score.

[0078] Step S214 , performing weighted sum processing on the qualitative score of the text graph and the quantitative score of the text graph to determine the image quality of the target image generated by the text graph network model according to the text description of the target text.

[0079] The above-mentioned embodiment integrates qualitative and quantitative multi-dimensional evaluation indicators to provide a more comprehensive and fair evaluation framework. The qualitative evaluation dimension can be scored by professional evaluators according to the scoring criteria, covering the five dimensions of content consistency, semantic consistency, visual consistency, logical consistency, and generation efficiency. According to the different focuses in different situations, the above dimensions are comprehensively evaluated using adaptive weight coefficients to finally form a total score of the qualitative dimension, thereby ensuring that the generated image receives a more reasonable score in each dimension. In addition, the quantitative evaluation dimension integrates the two aspects of image quality and text-image consistency, further reducing the deviation caused by a single indicator. Similarly, according to the different focuses in different situations, the above dimensions are comprehensively evaluated using adaptive weight coefficients to finally form a total score of the quantitative dimension, thereby ensuring that the generated image receives a more reasonable score in each dimension.

[0080] In addition, since the scoring ranges of qualitative and quantitative dimensions are not uniform, the above embodiment also needs to first normalize and calculate the two separately, and then combine the above qualitative and quantitative dimensions in an adaptive weighted manner to form a final score based on the different emphases in different situations. This will ultimately promote the standardization and internationalization of technology and accelerate its application in more practical scenarios.

[0081] Figure 3 The figure is a flowchart of a method for determining a first image quality quantitative score according to an exemplary embodiment.

[0082] refer to Figure 3 , the above-mentioned first image quality quantitative score determination method may include the following steps.

[0083] Step S302: extracting multivariate Gaussian model parameters of natural statistical features from the target image as first model parameters.

[0084] In some embodiments, the natural statistical features (such as NSS features) in the target image may be calculated first, and then the mean and covariance of the natural statistical features may be calculated as the first model parameters.

[0085] The above natural statistical features mainly refer to certain statistical characteristics that high-quality natural images obey. These statistical characteristics can be used in fields such as image quality evaluation, image processing, and computer vision.

[0086] In some embodiments, multivariate Gaussian model parameters of natural statistical features may be extracted from the target image to serve as first model parameters.

[0087] Step S304: extracting multivariate Gaussian model parameters of natural statistical features from the natural image as second model parameters.

[0088] The natural images mentioned above may be images directly from the real world that have not been artificially modified or have only been slightly processed. These images capture scenes, objects, people, etc. in nature and retain their original visual features as much as possible.

[0089] In some embodiments, natural statistical features may be extracted from natural images, and then multivariate Gaussian model parameters of the natural statistical features may be calculated as second model parameters.

[0090] Step S306, determining a target distance between the first model parameter and the second model parameter.

[0091] Step S308: If the target distance is less than the distance threshold, the first image quality quantitative score is higher than the first score.

[0092] Step S310: If the target distance is greater than or equal to the distance threshold, the first image quality quantitative score is lower than or equal to the first score.

[0093] In some embodiments, the quality of the target image can be represented by the distance between the MVG model of the natural statistical features extracted from the target image and the MVG of the natural statistical features extracted from the natural image. For details, refer to formula (1):

[0094]

[0095] Where v1, v2, ∑1, ∑2 represent the mean vector and covariance matrix of the natural statistical feature MVG model of the natural image and the natural statistical feature MVG model of the target image, respectively, and T represents the transpose.

[0096] Figure 4 is a flowchart of a method for determining a first model parameter according to an exemplary embodiment.

[0097] refer to Figure 4 , the above-mentioned first model parameter determination method may include the following steps.

[0098] Step S402, performing block processing on the target image to obtain multiple image blocks.

[0099] Step S404: determining a target image block whose sharpness is greater than a preset threshold among the multiple image blocks.

[0100] Step S406, obtaining Gaussian features of the target image block.

[0101] Step S408: determining first model parameters of the target image according to the Gaussian features of the target image block.

[0102] The above method can accurately determine the first model parameters of the target image, so that the naturalness of the target image can be accurately evaluated by the first model parameters.

[0103] Figure 5 The figure is a flowchart of a method for determining a target image block according to an exemplary embodiment.

[0104] In some embodiments, the plurality of image blocks may include a first image block.

[0105] refer to Figure 5 The above target image block determination method may include the following steps:

[0106] Step S502: Obtain the local variance corresponding to each pixel in the first image block.

[0107] In some embodiments, the local variance σ(i, j) corresponding to each pixel I(i, j) in the first image block may be obtained, as shown in formula (2):

[0108]

[0109] Among them, μ(i,j) can be determined by formula (3).

[0110]

[0111] Where i and j are the spatial coordinates of the image, i∈{1,2,…,M},j∈{1,2,…,N}, M and N are the spatial dimensions of the image, and satisfy: where {ω k,l |k=-K,…,K;l=-L,…,L} is a two-dimensional cyclic symmetric Gaussian function sampled from 3 standard deviations and rescaled to unit volume, and i or j is an integer greater than or equal to 1.

[0112] Step S504: add the local variances of the pixels in the first image block to determine the sharpness of the first image block.

[0113] The local variances of the pixels in the first image block are added to determine the sharpness of the first image block, as shown in formula (4):

[0114] Sharpness δ = ∑∑ (i,j)∈pat σ(i,j) (4)

[0115] Here, patch represents an image block.

[0116] Step S506: If the sharpness of the first image block is greater than a preset threshold, the first image block is used as a target image block.

[0117] From an aesthetic point of view, the higher the sharpness of an image, the better the imaging effect. Smooth blurring indicates potential loss of visual information in the image. Therefore, image blocks with higher sharpness are selected as target image blocks. The selection step usually first calculates the local sharpness δ of the image block, and then selects the image block with sharpness δ> threshold T as the target image block.

[0118] The above method can be used to screen out image blocks with better sharpness as target image blocks.

[0119] Figure 6 The figure is a flow chart of a method for determining Gaussian features according to an exemplary embodiment.

[0120] refer to Figure 6 , the above-mentioned Gaussian feature determination method may include the following steps.

[0121] Step S602: determine the local mean and local variance corresponding to each pixel in the target image block.

[0122] In some embodiments, the local mean and local variance corresponding to each pixel in the target image block may be obtained by referring to formulas (1) and (2).

[0123] Step S604 , subtract the local mean of the pixels from the pixels in the target image block, and then divide the subtraction by the local variance of the pixels to obtain the spatial natural statistical features of the pixels.

[0124] In some embodiments, the natural statistical characteristics of a pixel may be determined by formula (5).

[0125]

[0126] Where i and j are the spatial coordinates of the image, i∈{1,2,…,M}, j∈{1,2,…,N}, and M and N are the spatial dimensions of the image.

[0127] Step S606 , modeling the natural statistical features of the spatial domain of each pixel in the target image block by using a generalized Gaussian distribution or a symmetric generalized Gaussian distribution to obtain Gaussian features of the target image block.

[0128] In some embodiments, a generalized Gaussian distribution can be used to effectively capture the relationship between the natural image and the distorted image. The generalized Gaussian distribution (GDD) with a mean of 0 can refer to formula (6):

[0129]

[0130] in:

[0131]

[0132] In some embodiments, the Gaussian feature of the target image block may also be modeled using a symmetric generalized Gaussian distribution (AGGD) with a mean of 0:

[0133]

[0134] Figure 7 The figure is a flowchart of a method for determining a second image quality quantitative score according to an exemplary embodiment.

[0135] refer to Figure 7 , the above-mentioned second image quality quantitative score determination method may include the following steps.

[0136] Step S702: classify the target image using an image classification model to determine the probability distribution of the target image in each category.

[0137] Step S704, determining the KL divergence of each probability distribution.

[0138] Step S706: determine the exponential average of the KL divergences of each probability distribution to serve as the second image quality quantitative score.

[0139] The second quantitative image quality score can be used to evaluate the quality and diversity of generated images. It combines two considerations: the authenticity of the generated images (the probability of real images, the probability of the corresponding category should be as high as possible) and the diversity of the generated images (the entropy of the category distribution, the generated images of different categories should be as uniform as possible).

[0140] The calculation method of the second image quality quantitative score includes the following two steps:

[0141] First, the probability distribution of the generated image in each category is extracted through the classification model; then, the KL divergence (Kullback-Leibler divergence, also known as information gain or relative entropy, is an asymmetric measurement method for measuring the difference between two probability distributions P and Q) of these probability distributions is calculated. The final second image quality quantitative score is the exponential average of these KL divergences. For details, please refer to formula (9):

[0142] IS = exp(E x KL(P(y|x)||P(y))) (9)

[0143] Figure 8 The present invention is a flowchart of a method for determining a quantitative score of image-text consistency according to an exemplary embodiment.

[0144] refer to Figure 8 , the above-mentioned second image-text consistency quantitative score determination method may include the following steps.

[0145] Step S802: determine the text feature vector of the target text.

[0146] Step S804: determine the image feature vector of the target image.

[0147] Step S806: determining the image-text consistency quantitative score according to the distance between the text feature vector and the image feature vector.

[0148] The above image-text consistency quantitative score can be used to evaluate the relevance of the generated image and text. The higher the value, the better the text-image consistency.

[0149] In some embodiments, two encoders may be used to extract feature vectors of text and image respectively and then calculate similarity scores. The greater the distance, the lower the similarity, that is, the less relevant the image and text are. The specific calculation formula can refer to formula (10):

[0150] S(c,v)=ω*max(cos(c,v),0) (10)

[0151] Where ω is a coefficient, which can be set according to the specific situation; c is the characteristic vector of the target; v is the characteristic vector of the target image.

[0152] Fig. 9 The present invention is a flowchart of a method for determining a quantitative score of image-text consistency according to an exemplary embodiment.

[0153] refer to Fig. 9 , the above-mentioned second image-text consistency quantitative score determination method may include the following steps.

[0154] Step S902: Generate attribute question-answering questions based on the attribute information in the target text.

[0155] Step S904: determine the answer to the attribute question based on the target image.

[0156] Step S906: determining the image-text consistency quantitative score according to the answer.

[0157] In some embodiments, the image-text consistency quantitative score of this embodiment can be used to evaluate the relevance of the generated image and text. The higher the value, the better the text-image consistency.

[0158] In some embodiments, the image-text consistency quantitative score of this embodiment can also be used as Figure 8 Supplementary evaluation indicators for the examples shown, making up for Figure 8 The quantitative scoring of image-text consistency provided in the embodiment often ignores the detailed attribute characteristics of each object. The quantitative scoring of image-text consistency provided in this embodiment can be used to evaluate the index of text-image similarity (fine granularity-subject attribute characteristics, including color, shape, texture). The higher the value, the better the text-image consistency performs in terms of object attribute characteristics. This evaluation index clearly decomposes complex text prompts into multiple independent questions, each of which contains only one object-attribute pair, and then calculates the probability of the answer being "yes" as the final score.

[0159] Fig.10 The present invention is a flowchart of a method for determining a quantitative score of image-text consistency according to an exemplary embodiment.

[0160] refer to Fig.10 , the above-mentioned method for determining the quantitative score of image-text consistency may include the following steps.

[0161] Step S1002, determining the number of objects in the target text and the relative positions of the objects.

[0162] Step S1004, determining the number of objects in the target image and the relative positions of the objects.

[0163] Step S1006 , determining a quantitative score of image-text consistency according to the number of objects in the target text and the relative positions of the objects, and the number of objects in the target image and the relative positions of the objects.

[0164] The image-text consistency quantitative score provided in this embodiment can be used to evaluate the indicators of the spatial relationship and quantitative relationship of the generated image.

[0165] Spatial relationship evaluation (or relative position): First, use a network model (such as UniDet) to detect the position of objects in the generated image, and then determine the spatial relationship between the two objects by comparing the positions of the centers of the two bounding boxes. In this way, the spatial relationship between objects can be evaluated, including up, down, left, right, next to, adjacent, and cross.

[0166] Quantity relationship evaluation: First, the names of objects and their corresponding quantities are extracted from the target text. Then, a network model (such as UniDet) is used to detect objects in the image and count them.

[0167] Below, this application will explain and illustrate the text-generated image quality assessment method proposed in this application in combination with specific application scenarios.

[0168] Fig.11 The present invention is a flowchart of a method for evaluating the quality of a text-generated image according to an exemplary embodiment.

[0169] Combination Fig.11 , the above-mentioned method for evaluating the quality of a text-based image may include the following steps.

[0170] 1. The multimodal Chinese text image model generates corresponding pictures according to the text prompts, which serve as the input parameters of this multimodal Chinese text image evaluation system.

[0171] 2. Evaluate the generated images in qualitative and quantitative dimensions.

[0172] 3. Qualitative dimension design covers five sub-dimensions (for details, please refer to Fig.12 The images are graded manually and subjectively according to the preset rules of the five sub-dimensions (generation efficiency, logical consistency, visual consistency, semantic consistency, and content consistency).

[0173] 4. The scores of each rater in the five sub-dimensions of the qualitative dimension are weighted and combined according to different situations and emphases through the attention mechanism, so as to perform adaptive weight adjustment and finally obtain the qualitative total score.

[0174] 5. The quantitative dimension design covers two sub-dimensions, and the objective quantitative scoring of pictures is performed according to the preset models of the two sub-dimensions.

[0175] 6. The scores of different models in the two sub-dimensions of the quantitative dimension are weighted and combined according to different scenarios and emphases through the attention mechanism, so as to perform adaptive weight adjustment and finally obtain the quantitative total score.

[0176] 7. Use the maximum and minimum normalization method to normalize the qualitative dimension and the quantitative dimension, so that the two dimensions are measured uniformly, and then use the attention mechanism for weighted combination, and make adaptive weight adjustments according to different application scenarios to finally get the total evaluation score.

[0177] The multimodal Chinese language image evaluation system proposed in this embodiment consists of two dimensions: qualitative evaluation dimension and quantitative evaluation dimension. Fig.12 shown.

[0178] The following embodiment introduces the qualitative evaluation dimension evaluation system and method.

[0179] The qualitative evaluation dimension consists of five evaluation dimensions: content consistency (focusing on the examination of static elements), semantic consistency (focusing on the examination of semantic information such as emotion, context, and ambiguous resolution), visual consistency (examining the quality and aesthetics of generated images), logical consistency (focusing on text-image consistency, logical coherence, and interaction), and generation efficiency (examining the efficiency of generating images). The total score of the qualitative evaluation dimension is obtained by the weighted addition of each scorer according to each sub-dimension.

[0180]

[0181] Among them, A, B, C, D, and E represent the five sub-dimensions of the qualitative dimension, which are:

[0182] A: Content consistency;

[0183] B: semantic consistency;

[0184] C: Visual consistency;

[0185] D: logical consistency;

[0186] E: Generation efficiency.

[0187] α represents the adaptive weight of each sub-dimension in different situations, and n represents the number of evaluators.

[0188] The following example introduces the quantitative evaluation dimension evaluation system and method.

[0189] The quantitative evaluation dimension focuses on two aspects: image quality and text-image consistency. The evaluation items used in image quality include: Natural Image Quality Evaluator NIQE (Natural Image Quality Evaluator) and Inception Score (IS), which examine the naturalness of the generated image quality and the authenticity and diversity of the image; the evaluation items used in text-image consistency include: CLIP Score, Disentangled BLIP-VQA, UniDet-based Metric and 3-in-1 Metric, which examine the similarity between image and text from both coarse-grained and fine-grained aspects.

[0190] Inception Score (IS) is an objective performance indicator for evaluating the quality of generated images or synthetic images generated by a generative adversarial network (GAN).

[0191] IS evaluates the quality of generated images by calculating the KL divergence (Kullback-Leibler Divergence) between the generated image distribution and the real image distribution. KL divergence is a method to measure the difference between two probability distributions. The closer the two distributions are, the smaller the KL divergence value is. IS can be regarded as the mutual information between the input variable and the real image, and its value is between 1 and ln(K), where K represents the number of categories. The upper bound of IS is K, indicating a perfect match; the lower bound is 1, indicating that there is no correlation between the generated image and the real image distribution.

[0192] The higher the S value, the better the quality of the generated image. Specifically, a high IS value means that the generated image performs well in the following aspects.

[0193] 1. Authenticity: The generated images are closer to the real images in visual features and are difficult to distinguish.

[0194] 2. Diversity: The generated images cover multiple categories, and the images within each category are diverse.

[0195] Image quality evaluation index – NIQE

[0196] NIQE uses a multivariate Gaussian model to fit a series of features used to measure image quality, which can measure the difference in the multivariate distribution of a test image. It is based on a set of "quality-aware" features and fits them to the MVG model. The quality-aware features are derived from a simple but highly regularized NSS model. The NIQE indicator for a given test image is then expressed as the distance between the MVG model of the NSS features extracted from the test image and the MVG model of the NSS features extracted from the natural image corpus.

[0197] The entire process of NIQE index calculation can refer to formulas (12) to (18) or formula (19)

[0198] It usually involves five steps.

[0199]

[0200] Where i and j are the spatial coordinates of the image, i∈{1,2,…,M},j∈{1,2,…,N}, M and N are the spatial dimensions of the image, and satisfy:

[0201]

[0202] where {ω k,l |k=-K,…,K;l=-L,…,L} is a 2D circularly symmetric Gaussian function sampled from 3 standard deviations and rescaled to unit volume.

[0203] Image block selection:

[0204] From an aesthetic point of view, the higher the sharpness of an image, the better the imaging effect. Smooth blurring indicates potential loss of visual information in the image, so image blocks with higher sharpness are selected. The selection step usually first calculates the local sharpness δ of the image block, and then selects the image block with sharpness δ> threshold T.

[0205] Sharpness δ = ∑∑ (i,j)∈patc σ(i,j) (15)

[0206] In some embodiments, a generalized Gaussian distribution can be used to effectively capture the relationship between natural images and distorted images. The generalized Gaussian distribution (GDD) with a mean of 0 is:

[0207]

[0208] Where: α, β are the estimated parameters of the Gaussian distribution, x is the parameter in formula (12) t is the dimension in time or space, and Γ is a fixed value.

[0209]

[0210] The features of the image block can also be modeled using a symmetric generalized Gaussian distribution (AGGD) with a mean of 0:

[0211]

[0212] x is I,γ,β l ,β r is the coefficient of the generalized Gaussian distribution, and f is the feature of the image block.

[0213] MVG model construction:

[0214] By fitting the natural image block with the MVG model density function, the natural statistical feature (NSS) model can be obtained. The MVG model density function is:

[0215]

[0216] where (x1,…,x k ) is the NSS feature obtained in the characterization image block step (such as the Gaussian feature obtained in formula (16) or formula (18); v and ∑ represent the mean and covariance matrix of the MVG model respectively.

[0217] NIQE indicator calculation:

[0218] NIQE uses the distance between the NSS feature model and the MVG extracted from the distorted image features to represent the quality of the distorted image:

[0219]

[0220] Where v1, v2, ∑1, ∑2 represent the mean vector and covariance matrix of the natural MVG model and the distorted image MVG model respectively.

[0221] The distance between the MVG (Multivariate Gaussian Model) model parameters of the NSS (Natural Statistical Features) features of the target image and the MVG model parameters of the NSS features of the natural image can measure the image quality, which is mainly based on the following reasons.

[0222] 1. Relationship between NSS characteristics and image quality

[0223] The NSS feature reflects the natural statistical laws of the image, which are ubiquitous in high-quality natural images. When the image is damaged (such as lossy compression, noise pollution, etc.), these statistical laws will change, causing the NSS feature to change as well. Therefore, the NSS feature can be used to describe the quality status of the image.

[0224] 2. Modeling capability of MVG model for NSS features

[0225] The MVG model is a powerful statistical model that can be used to describe the distribution characteristics of multivariate data. In image processing, the MVG model can be used to model NSS features and capture the correlation and dependence between these features. By fitting the MVG model of NSS features, we can obtain a set of parameters that describe the statistical characteristics of the image, which can be used to distinguish images of different qualities.

[0226] 3. Effectiveness of distance metrics

[0227] After obtaining the MVG model parameters of the NSS features of the target image and the natural image, we can calculate the distance between the two sets of parameters (such as Euclidean distance, Mahalanobis distance, etc.). This distance reflects the degree of difference in statistical characteristics between the target image and the natural image. If the distance is small, it means that the statistical characteristics between the target image and the natural image are highly similar, so the image quality is also high; conversely, if the distance is large, it means that the statistical characteristics between the target image and the natural image are greatly different, so the image quality is also low.

[0228] Image quality rating indicator - Inception Score (IS): is an indicator used to evaluate the quality and diversity of generated images. It combines two considerations: the authenticity of the generated images (the probability of real images, the probability of the corresponding category should be as high as possible) and the diversity of the generated images (the entropy of the category distribution, the generated images of different categories should be as uniform as possible).

[0229] The calculation method of IS includes two steps: first, the probability distribution of each category of the generated image is extracted through the classification model; how to understand this sentence, training a classification model, one of the categories is the category corresponding to the text? But we don’t know what category the text describes?

[0230] Then, the KL divergence of these probability distributions is calculated, and the final IS is the exponential average of these KL divergences.

[0231] Image-text consistency scoring indicator--CLIP Score.

[0232] CLIP Score is an indicator used to evaluate the relevance between the generated image and the text prompt. The higher the value, the better the text-image consistency. CLIP uses two encoders to extract the embedding of the text and image respectively, and then calculates the similarity score. The larger the distance, the lower the similarity, that is, the less relevant the image and text are. The specific calculation formula is:

[0233] S(c,v)=ω*max(cos(c,v),0) (21)

[0234] Where ω is the scaling coefficient, which can be set according to the specific situation; c is the embedding of the text prompt; v is the embedding of the image.

[0235] Image-text consistency scoring metric - Disentangled BLIP-VQA.

[0236] It is used as a supplementary evaluation indicator for CLIP Score to make up for the fact that the BLIP caption generation model often ignores the detailed attribute features of each object, and to give full play to the visual question answering (VQA) capability of BLIP. An indicator used to evaluate text-image similarity (fine granularity-main attribute features, including color, shape, texture). The higher the value, the better the text-image consistency in terms of object attribute features. This evaluation indicator explicitly decomposes complex text prompts into multiple independent questions, each of which contains only one object-attribute pair, and then calculates the probability of the answer being "yes" as the final score.

[0237] Image-text consistency scoring index--UniDet-based Metric: An index used to evaluate the spatial and quantitative relationships of generated images.

[0238] Spatial relationship evaluation: First, UniDet is used to detect the position of objects in the generated image, and then the spatial relationship between the two objects is determined by comparing the positions of the centers of the two bounding boxes. In this way, the spatial relationship between objects can be evaluated, including up, down, left, right, next to, adjacent, and cross.

[0239] Quantity relationship evaluation: First, the name of the object and its corresponding quantity are extracted from the prompt. Then, UniDet is used to detect the objects in the image and count them.

[0240] Image-text consistency scoring metric - 3-in-1Metric.

[0241] Based on the above evaluation indicators, for the evaluation dimension of image-text consistency, Disentangled BLIP-VQA is most suitable for attribute feature evaluation, UniDet-based Metric is most suitable for spatial relationship and quantitative relationship evaluation, and CLIP Score is most suitable for non-spatial relationship evaluation. Therefore, the average score of the above three indicators is calculated as a three-in-one evaluation indicator for the evaluation indicator of text prompt words of complex attributes. The higher the score, the better the consistency between the complex text prompt words and the generated image.

[0242] The above six evaluation indicators constitute the evaluation system of quantitative dimension. Since the evaluation score magnitude of each evaluation indicator item is not unified, this paper proposes to first normalize each evaluation dimension using the maximum and minimum normalization method, unify them to the same dimension, and then perform weighted addition of each dimension to obtain the total score of the quantitative model. The specific calculation formula is:

[0243] Image quality score I = β A ×Norm(NIQE)+β B ×Norm(IS) (22)

[0244] Image-text consistency scoring

[0245] where β I and β T is the adaptive weight of each evaluation index under different situations, and satisfies β A +β B =1;β C +β D +β E +β F =1.

[0246] In addition, Norm uses the maximum and minimum normalization method, which is:

[0247]

[0248] Total score of quantitative model Y = γ I *I+γ T *T.

[0249] Among them, γ I and γ T is the adaptive weight of image quality score and image text consistency score, and satisfies γ I +γ T =1; total evaluation score = τ X *Norm(X)+τ Y *Norm(Y).

[0250] Among them, τ X and τ Y is the adaptive weight of the total score of qualitative evaluation and the total score of quantitative evaluation, and satisfies τ X +τ Y =1.

[0251] In some embodiments, the images generated by the large model of Chinese literary images are evaluated and scored according to the qualitative + quantitative dimensions. The specific implementation method is: the algorithm personnel input the images generated by the large model into the various evaluation indicator algorithms of the above-mentioned quantitative evaluation dimensions, and the model uniformly scores them, thereby obtaining an objective quantitative score, and then using the above formula to perform maximum and minimum normalization on each evaluation indicator, and then using the weighted addition method to obtain the final result of the quantitative score. The scoring personnel score the images generated by the large model item by item according to the "Chinese Literary Image Large Model Scoring Standard", which is a qualitative score. In order to avoid deviations due to subjective reasons of the personnel, there should be no less than 20 scorers, and the average of the scores of the total 20 people is taken as the final result of the qualitative score. The above-mentioned final result of the qualitative score and the final result of the quantitative score are unified using the maximum and minimum normalization method, and then weighted addition is performed to obtain the final score, which is the final score of the evaluation of the picture.

[0252] The above embodiment provides an evaluation system architecture that integrates quantitative evaluation dimensions and qualitative evaluation dimensions, which not only reduces the bias caused by personnel differences in subjective qualitative evaluation, but also improves the human-like scoring of the objective quantitative evaluation model, effectively avoiding the incompleteness and bias in the existing large model evaluation system.

[0253] The above embodiments, for qualitative and quantitative evaluation dimensions, comprehensively consider aspects such as image content, image quality, model generation efficiency, and large model understanding of text prompts. In particular, this article designs an investigation of Chinese-related understanding and generation capabilities for Chinese literary images, which more effectively provides a powerful evaluation method for the Chinese literary image large model. In addition, for evaluation indicators between different dimensions, it is proposed to use the maximum and minimum normalization method, unify the dimensions, and then perform weighted addition to make the evaluation results more comprehensive and three-dimensional.

[0254] In summary, this embodiment provides a complete evaluation system for the Chinese text-generating image capability of a multimodal large model, filling the gap of the lack of a systematic evaluation system for the Chinese text-generating image large model. The design covers the evaluation of the ability of the images generated by the large model to understand the text content, semantics, and context; the evaluation of the quality and aesthetics of the image generation; the evaluation of the spatial logic, time logic, and other logic displayed by the image; and the evaluation of the efficiency of the large model to generate images. And it covers the evaluation of both qualitative and quantitative dimensions, so that the evaluation results are consistent with the human aesthetic results, and avoid the deviation caused by the subjective evaluation of personnel, so it is more fair and reasonable. In addition, the method provided in this embodiment adjusts the weights adaptively according to the actual situation through the evaluation of each dimension, so that the final score is more in line with the actual situation. For example, if the picture needs to be beautiful and cute, the weight of the visual consistency dimension can be adaptively increased.

[0255] It should be particularly pointed out that the various steps in the various embodiments of the above-mentioned method for evaluating the quality of text-generated images can be mutually crossed, replaced, added, or deleted. Therefore, these reasonable permutations, combinations, and transformations of the method for evaluating the quality of text-generated images should also fall within the scope of protection of the present disclosure, and the scope of protection of the present disclosure should not be limited to the embodiments.

[0256] Based on the same inventive concept, the present disclosure also provides a device for evaluating the quality of a text image, such as the following embodiment. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0257] Fig.13 is a block diagram of a device for evaluating the quality of a text-based image according to an exemplary embodiment. Fig.13The apparatus 1300 for evaluating the quality of a Vincent image provided by an embodiment of the present disclosure may include: an image acquisition module 1301, a qualitative score determination module 1302, a first image quality quantitative score determination module 1303, a second image quality quantitative score determination module 1304, a consistency quantitative score determination module 1305, a Vincent image quantitative score determination module 1306, and an image quality determination module 1307.

[0258] Among them, the image acquisition module 1301 can be used to acquire the target text and the target image, and the target image is an image generated by the text-graph network model according to the text description of the target text; the qualitative score determination module 1302 can be used to obtain the evaluation object to perform a qualitative score on at least one of the generation efficiency, logical consistency, visual consistency, semantic consistency and content consistency when the text-graph network model generates the target image according to the target text according to the target rule, and obtain at least one of the generation efficiency qualitative score, logical consistency qualitative score, visual consistency qualitative score, semantic consistency qualitative score and content consistency qualitative score; the first image quality quantitative score determination module 1303 can be used to score the naturalness of the target image by a first image quality scoring method, and obtain a first image quality quantitative score; the second image quality The quantitative scoring determination module 1304 can be used to score the authenticity and diversity of the target image by a second image quality scoring method to obtain a second image quality quantitative score; the consistency quantitative scoring determination module 1305 can be used to quantitatively score the consistency between the target text and the target image by a third image quality scoring method to obtain an image-text consistency quantitative score; the Vincent graph quantitative score determination module 1306 can be used to perform weighted sum processing on the first image quality quantitative score, the second image quality quantitative score and the image-text consistency quantitative score to obtain a Vincent graph quantitative score; the image quality determination module 1307 can be used to perform weighted sum processing on the Vincent graph qualitative score and the Vincent graph quantitative score to determine the image quality of the target image generated by the Vincent graph network model according to the text description of the target text.

[0259] It should be noted that the above-mentioned image acquisition module 1301, qualitative score determination module 1302, first image quality quantitative score determination module 1303, second image quality quantitative score determination module 1304, consistency quantitative score determination module 1305, Vincent graph quantitative score determination module 1306 and image quality determination module 1307 correspond to S402 to S406 in the method embodiment, and the examples and application scenarios implemented by the above-mentioned modules and corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned method embodiment. It should be noted that the above-mentioned modules as part of the device can be executed in a computer system such as a set of computer executable instructions.

[0260] In some embodiments, the first image quality quantitative score determination module 1303 may include: a first parameter model determination submodule, a second parameter model determination submodule, a target distance determination submodule, a first distance judgment submodule, and a second distance judgment submodule.

[0261] Among them, the first parameter model determination submodule can be used to extract multivariate Gaussian model parameters of natural statistical features from the target image as the first model parameters; the second parameter model determination submodule can be used to extract multivariate Gaussian model parameters of natural statistical features from the natural image as the second model parameters; the target distance determination submodule can be used to determine the target distance between the first model parameter and the second model parameter; the first distance judgment submodule can be used to determine that if the target distance is less than the distance threshold, the first image quality quantitative score is higher than the first score; the second distance judgment submodule can be used to determine that if the target distance is greater than or equal to the distance threshold, the first image quality quantitative score is lower than or equal to the first score.

[0262] In some embodiments, the first parameter model determination submodule may include: an image block determination unit, a target image block determination unit, a Gaussian feature acquisition unit, and a first model parameter determination unit.

[0263] Among them, the image block determination unit can be used to perform block processing on the target image to obtain multiple image blocks; the target image block determination unit can be used to determine the target image block whose sharpness is greater than a preset threshold among the multiple image blocks; the Gaussian feature acquisition unit can be used to acquire the Gaussian feature of the target image block; the first model parameter determination unit can be used to determine the first model parameter of the target image according to the Gaussian feature of the target image block.

[0264] In some embodiments, the plurality of image blocks include a first image block; wherein the target image block determination unit may include: a local variance acquisition subunit, a sharpness determination subunit and a target image block determination subunit.

[0265] Among them, the local variance acquisition subunit can be used to obtain the local variance corresponding to each pixel in the first image block; the sharpness determination subunit can be used to add the local variance of each pixel in the first image block to determine the sharpness of the first image block; the target image block determination subunit can be used to use the first image block as the target image block if the sharpness of the first image block is greater than a preset threshold.

[0266] In some embodiments, the Gaussian feature acquisition unit may include: a local mean value determination subunit, a spatial domain natural statistical feature determination subunit, and a Gaussian feature acquisition subunit.

[0267] Among them, the local mean determination subunit can be used to determine the local mean and local variance corresponding to each pixel in the target image block; the spatial natural statistical feature determination subunit can be used to subtract the local mean of the pixels in the target image block and then divide it by the local variance of the pixels to obtain the spatial natural statistical features of the pixels; the Gaussian feature acquisition subunit can be used to model the spatial natural statistical features of each pixel in the target image block through a generalized Gaussian distribution or a symmetric generalized Gaussian distribution to obtain the Gaussian features of the target image block.

[0268] In some embodiments, the second image quality quantitative score determination module 1304 may include: a probability distribution determination submodule, a divergence determination submodule, and an exponential average submodule.

[0269] Among them, the probability distribution determination submodule can be used to classify the target image through the image classification model to determine the probability distribution of the target image in each category; the divergence determination submodule can be used to determine the KL divergence of each probability distribution; the exponential average submodule can be used to determine the exponential average of the KL divergence of each probability distribution as the second image quality quantitative score.

[0270] In some embodiments, the consistency quantitative score determination module 1305 may include: a text feature vector determination submodule, an image feature vector determination submodule, and a first quantitative score submodule.

[0271] Among them, the text feature vector determination submodule can be used to determine the text feature vector of the target text; the image feature vector determination submodule can be used to determine the image feature vector of the target image; the first quantitative scoring submodule can be used to determine the image-text consistency quantitative score based on the distance between the text feature vector and the image feature vector.

[0272] In some embodiments, the consistency quantitative score determination module 1305 may include: a question generation submodule, an answer determination submodule, and a second quantitative score submodule.

[0273] Among them, the question generation submodule can be used to generate attribute question-answering questions based on the attribute information in the target text; the answer determination submodule can be used to determine the answer to the attribute question-answering question based on the target image; and the second quantitative scoring submodule can be used to determine the image-text consistency quantitative score based on the answer.

[0274] In some embodiments, the consistency quantitative score determination module 1305 may include: a first position determination submodule, a second position determination submodule, and a third quantitative score submodule.

[0275] Among them, the first position determination submodule can be used to determine the number of objects in the target text and the relative positions of each object; the second position determination submodule can be used to determine the number of objects in the target image and the relative positions of each object; the third quantitative scoring submodule can be used to determine the image-text consistency quantitative score based on the number of objects in the target text and the relative positions of each object, and the number of objects in the target image and the relative positions of each object.

[0276] Since the functions of the device 1300 have been described in detail in the corresponding method embodiments, the present disclosure will not elaborate on them here.

[0277] The modules and / or submodules and / or units and / or subunits involved in the embodiments described in the present disclosure may be implemented by software or by hardware. The modules and / or submodules and / or units and / or subunits described may also be arranged in a processor. The names of these modules and / or submodules and / or units and / or subunits do not, in certain circumstances, constitute limitations on the modules and / or submodules and / or units and / or subunits themselves.

[0278] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a part of a module or program segment, and a part of the above-mentioned module or program segment contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer program instructions.

[0279] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0280] Fig.14 FIG. 1 shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present disclosure. It should be noted that: Fig.14The electronic device 1400 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0281] like Fig.14 As shown, the electronic device 1400 includes a central processing unit (CPU) 1401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1402 or a program loaded from a storage part 1408 into a random access memory (RAM) 1403. In the RAM 1403, various programs and data required for the operation of the electronic device 1400 are also stored. The CPU 1401, the ROM 1402, and the RAM 1403 are connected to each other via a bus 1404. An input / output (I / O) interface 1405 is also connected to the bus 1404.

[0282] The following components are connected to the I / O interface 1405: an input section 1406 including a keyboard, a mouse, etc.; an output section 1407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1408 including a hard disk, etc.; and a communication section 1409 including a network interface card such as a LAN card, a modem, etc. The communication section 1409 performs communication processing via a network such as the Internet. A drive 1410 is also connected to the I / O interface 1405 as needed. A removable medium 1411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1410 as needed, so that a computer program read therefrom is installed into the storage section 1408 as needed.

[0283] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains computer program instructions for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part 1409, and / or installed from a removable medium 1411. When the computer program is executed by a central processing unit (CPU) 1401, the above-mentioned functions defined in the system of the present disclosure are executed.

[0284] It should be noted that the computer-readable storage medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable computer program instructions. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium that can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The computer program instructions contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0285] As another aspect, the present disclosure also provides a computer-readable storage medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed by a device, the device can implement functions including: obtaining a target text and a target image, the target image is an image generated by the text graph network model according to the text description of the target text; obtaining a qualitative score of at least one of the generation efficiency, logical consistency, visual consistency, semantic consistency and content consistency when the text graph network model generates the target image according to the target text according to the target rules by the evaluation subject, and obtaining at least one of the generation efficiency qualitative score, logical consistency qualitative score, visual consistency qualitative score, semantic consistency qualitative score and content consistency qualitative score; through the first figure The naturalness of the target image is scored by the image quality scoring method to obtain a first image quality quantitative score; the authenticity and diversity of the target image are scored by the second image quality scoring method to obtain a second image quality quantitative score; the consistency of the target text and the target image is quantitatively scored by the third image quality scoring method to obtain an image-text consistency quantitative score; the first image quality quantitative score, the second image quality quantitative score and the image-text consistency quantitative score are weighted and summed to obtain a Vincent graph quantitative score; the Vincent graph qualitative score and the Vincent graph quantitative score are weighted and summed to determine the image quality of the target image generated by the Vincent graph network model according to the text description of the target text.

[0286] According to one aspect of the present disclosure, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer program instructions, the computer program instructions being stored in a computer-readable storage medium. The computer program instructions are read from the computer-readable storage medium, and a processor executes the computer program instructions to implement the methods provided in various optional implementations of the above embodiments.

[0287] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software, or by combining software with necessary hardware. Therefore, the technical solution of the embodiment of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including a number of computer program instructions for enabling an electronic device (which can be a server or a terminal device, etc.) to execute the method according to the embodiment of the present disclosure.

[0288] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and embodiments are to be regarded as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0289] It should be understood that the present disclosure is not limited to the detailed structures, drawings or implementations shown herein, but rather the present disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A method for evaluating the quality of a text image, characterized in that: include: Obtaining a target text and a target image, wherein the target image is an image generated by a text-graph network model according to a text description of the target text; Obtaining a qualitative score from the evaluation subject on at least one of generation efficiency, logical consistency, visual consistency, semantic consistency, and content consistency when the text-to-graph network model generates the target image from the target text according to the target rule, thereby obtaining at least one of the qualitative score of generation efficiency, the qualitative score of logical consistency, the qualitative score of visual consistency, the qualitative score of semantic consistency, and the qualitative score of content consistency; Performing weighted sum processing on the generation efficiency qualitative score, the logical consistency qualitative score, the visual consistency qualitative score, the semantic consistency qualitative score, and the content consistency qualitative score to determine a Wenshengtu qualitative score; Scoring the naturalness of the target image using a first image quality scoring method to obtain a first image quality quantitative score; Scoring the authenticity and diversity of the target image using a second image quality scoring method to obtain a second image quality quantitative score; Performing a quantitative score on the consistency between the target text and the target image using a third image quality scoring method to obtain an image-text consistency quantitative score; Performing weighted sum processing on the first image quality quantitative score, the second image quality quantitative score, and the image-text consistency quantitative score to obtain a Vincent image quantitative score; A weighted summation process is performed on the qualitative score of the Venjiogram and the quantitative score of the Venjiogram to determine the image quality of the target image generated by the Venjiogram network model according to the text description of the target text.

2. The method according to claim 1, characterized in that Scoring the naturalness of the target image using a first image quality scoring method to obtain a first image quality quantitative score includes: Extracting multivariate Gaussian model parameters of natural statistical features from the target image as first model parameters; Extracting multivariate Gaussian model parameters of natural statistical features from natural images as second model parameters; determining a target distance between the first model parameter and the second model parameter; If the target distance is less than the distance threshold, the first image quality quantitative score is higher than the first score; If the target distance is greater than or equal to the distance threshold, the first image quality quantitative score is lower than or equal to the first score.

3. The method according to claim 2, characterized in that Extracting multivariate Gaussian model parameters of natural statistical features from the target image as first model parameters includes: Performing block processing on the target image to obtain multiple image blocks; Determining a target image block having a sharpness greater than a preset threshold among the multiple image blocks; Obtaining Gaussian features of the target image block; The first model parameters of the target image are determined according to the Gaussian features of the target image block.

4. The method according to claim 3, characterized in that The plurality of image blocks include a first image block; wherein determining a target image block having a sharpness greater than a preset threshold among the plurality of image blocks comprises: Obtaining a local variance corresponding to each pixel in the first image block; adding the local variances of the pixels in the first image block to determine the sharpness of the first image block; If the sharpness of the first image block is greater than the preset threshold, the first image block is used as the target image block.

5. The method according to claim 3, characterized in that: Obtaining the Gaussian features of the target image block includes: Determining the local mean and local variance corresponding to each pixel in the target image block; Subtracting the local mean of the pixels in the target image block from the local mean of the pixels and then dividing the result by the local variance of the pixels to obtain spatial natural statistical features of the pixels; The spatial natural statistical features of each pixel in the target image block are modeled by generalized Gaussian distribution or symmetric generalized Gaussian distribution to obtain Gaussian features of the target image block.

6. The method according to claim 1, characterized in that Scoring the authenticity and diversity of the target image using a second image quality scoring method to obtain a second image quality quantitative score, including: Classify the target image using an image classification model to determine the probability distribution of the target image in each category; Determine the KL divergence of each probability distribution; An exponential average of the KL divergences of the respective probability distributions is determined as the second image quality quantitative score.

7. A device for evaluating the quality of a text image, characterized in that: include: An image acquisition module is used to acquire a target text and a target image, wherein the target image is an image generated by a text-graph network model according to a text description of the target text; a qualitative scoring determination module, configured to obtain a qualitative score of at least one of the generation efficiency, logical consistency, visual consistency, semantic consistency, and content consistency of the text-graph network model when generating the target image from the target text according to the target rule, and obtain at least one of the qualitative score of generation efficiency, the qualitative score of logical consistency, the qualitative score of visual consistency, the qualitative score of semantic consistency, and the qualitative score of content consistency; a first image quality quantitative score determination module, configured to score the naturalness of the target image using a first image quality scoring method to obtain a first image quality quantitative score; a second image quality quantitative score determination module, configured to score the authenticity and diversity of the target image using a second image quality scoring method to obtain a second image quality quantitative score; a consistency quantitative score determination module, configured to perform a quantitative score on the consistency between the target text and the target image using a third image quality scoring method to obtain an image-text consistency quantitative score; a Vincent image quantitative score determination module, configured to perform weighted sum processing on the first image quality quantitative score, the second image quality quantitative score, and the image-text consistency quantitative score to obtain a Vincent image quantitative score; The image quality determination module is used to perform weighted summation processing on the qualitative score of the culture graph and the quantitative score of the culture graph to determine the image quality of the target image generated by the culture graph network model according to the text description of the target text.

8. An electronic device, characterized in that: include: memory and processor; The memory is used to store computer program instructions; the processor calls the computer program instructions stored in the memory to implement the method for evaluating the quality of text-generated images according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method for evaluating the quality of text-generated images according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising computer program instructions stored in a computer-readable storage medium, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images