AI evaluation method and system based on AI-generated content matching evaluation

Abstract

The present invention discloses an AI evaluation method and system based on AI generated content matching evaluation, belonging to the field of artificial intelligence technology. The method includes: generating input data and output data of an AI model, wherein the output data is the content generated by the AI ​​model based on the input data; receiving input data and output data, and calculating the matching degree of the input data and the output data; outputting the matching degree of the input data and the output data to a user or other system component for display and further optimization calculation. A corresponding system, electronic device, and computer-readable storage medium are also disclosed. The present invention can effectively evaluate the performance of the generation model of text to image, text to video, text to audio, image to text, and image to audio, and gives a high score to the model with a high matching degree of input and output, and vice versa. It also plays a guiding role in the optimization of the generation model, and automatically optimizes the calculation method to adapt to different types of AI generation models and tasks.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence (AI) and evaluation technology under the background of artificial intelligence, and in particular to an AI evaluation method and system based on AI-generated content matching evaluation. Background Art

[0002] AIGC (AI-generated content) has gradually moved from concept to reality. As its name suggests, AIGC is content created and produced by AI. In more and more scenarios such as AI writing, AI drawing, and AI composition, AIGC has given many amazing results. Take AI drawing as an example. You only need to enter text, and the AI ​​drawing tool can generate pictures based on the text. There are currently several companies that provide such services, and there are many ways in academia to complete AI drawing.

[0003] Compared with the rapid development of AI creation methods, there has been no significant progress in the evaluation of AI-generated content. Comparing AI-generated content with existing real-world content is the earliest practice. Taking AI drawing as an example, comparing the real-world photo corresponding to the text with the AI-generated picture using the text, and calculating the gap is such an evaluation method. This method was very useful in the early days when the quality of AI-generated pictures was not high, because the closer the AI-generated pictures were to the real-world photos, the better the quality of the generation. However, when the quality of AI-generated pictures reaches a certain level, the method of calculating the gap with real-world pictures often limits creativity, because a certain picture in reality is definitely not the only solution to the text. Stunning AI-generated pictures are often very different from existing photos, so this method is not very suitable for evaluating creative pictures.

[0004] Recent AI drawing papers have begun to use manual annotation methods to evaluate the quality of generated images. Due to the difficulty of looking at a picture and annotating the picture quality score in the case of manual annotation, and the annotation consistency is low, a relative evaluation scheme is adopted, that is, looking at two pictures and letting the annotator judge which picture is better. Although this relative evaluation scheme solves the problems of annotation difficulty and data consistency, the annotation efficiency decreases due to the increase in the number of annotations. Especially when a large number of comparisons are required, manual annotation requires a lot of time to perform repetitive annotations. In addition, the annotation consistency of relative evaluation is already higher than that of absolute evaluation, but the annotation consistency of different annotators is still an important obstacle to the credibility of the evaluation results, and more and more detailed annotations are often required to alleviate the problem of poor consistency.

[0005] Therefore, there is an urgent need to find more fair, effective and universal AI evaluation methods and systems based on AI-generated content matching evaluation, and to solve the problems of explainability and controllability of AI-generated content. At the same time, we must also face and solve the ethical issues of artificial intelligence. Summary of the invention

[0006] In order to solve the problems existing in the prior art, the present invention provides the following technical solutions: an AI evaluation method and system based on AI-generated content matching evaluation, which evaluates the quality of AI-generated content by calculating the matching between the input and output of the model. For example: for the text-to-image generation model, the system will extract the key information of the input text and the output image, and calculate their similarity to obtain the matching degree; for the text-to-text generation model, the system will compare the matching degree of the input and output texts through semantic similarity analysis. In order to improve the accuracy of the matching degree, the system will combine multiple calculation methods, such as pixel-level matching, semantic-level matching, etc., and automatically optimize these calculation methods to adapt to different types of AI models and tasks.

[0007] On one hand, the present invention provides an AI evaluation method based on AI-generated content matching evaluation, and the AI ​​evaluation is performed on a single AI model, including:

[0008] S1, generating input data and output data of the AI ​​model, wherein the output data is content generated by the AI ​​model according to the input data;

[0009] S2, receiving the input data and the output data, and calculating the matching degree between the input data and the output data;

[0010] S3, outputting the matching degree between the input data and the output data to the user or other system components for display and further optimization calculation.

[0011] Preferably, the S1 comprises:

[0012] S11, generating input data for an AI model, wherein the input data is text, image, audio or video;

[0013] S12, inputting the input data into the AI ​​model to obtain the output data of the AI ​​model, wherein the output data is text, image, audio or video; the types of the input data and the output data do not need to be consistent.

[0014] Preferably, the S1 further includes:

[0015] S13, preprocessing the input data and the output data; the preprocessing includes:

[0016] If the input data and / or output data is text, perform word segmentation, stop word removal and word embedding operations;

[0017] If the input data and / or output data is an image, performing pixel normalization and feature extraction;

[0018] If the input data and / or output data is audio, performing speech framing and windowing operations;

[0019] If the input data and / or output data is a video, video shot segmentation or shot boundary detection, key frame extraction and video feature extraction are performed.

[0020] Preferably, S2 includes:

[0021] S21, receiving the pre-processed input data and output data;

[0022] S22, calculating the matching degree between the pre-processed input data and the output data;

[0023] The matching degree calculation method of S22 depends on the types of input data and output data;

[0024] For AI models whose input data is text and whose output data is image, the matching degree calculation method includes:

[0025] (1) Extract key information of input data and output data;

[0026] (2) Matching evaluation of input data and output data based on semantic similarity and pixel-level matching;

[0027] For AI models whose input data is text and whose output data is text, the matching degree is calculated as follows:

[0028] The matching degree between input data and output data is compared using word embedding and semantic similarity calculation; wherein: word embedding is a distributed representation of a word obtained using a word feature matrix;

[0029] The calculation formula of the semantic similarity is formula (1):

[0030] S_semantic=Sum(cosine_similarity(T_i, I_i)) / N (1);

[0031] Where S_semantic represents semantic similarity; T_i represents the word vector of the i-th keyword in the input text; I_i represents the feature vector of the i-th key feature in the output image; N represents the number of keywords or key features; cosine_similarity represents the cosine similarity function;

[0032] The calculation formula of pixel-level matching is as follows:

[0033] S_pixel=1-1 / (1+exp(-Sum(abs(P_i-Q_i)) / M)) (2);

[0034] Among them, S_pixel represents the matching degree at the pixel level; P_i represents the i-th pixel value of the output image; Q_i represents the i-th pixel value of the standard image; M represents the number of pixel values; abs represents the absolute value function, and exp represents the exponential function.

[0035] Preferably, the matching evaluation based on semantic similarity and pixel-level matching as input data and output data includes: obtaining a total matching score by weighted averaging the semantic similarity and pixel-level matching, and using the total matching score as input and output matching evaluation, wherein the calculation formula of the total matching score is as shown in formula (3):

[0036] S_total=w1*S_semantic+w2*S_pixel (3);

[0037] Wherein, S_total represents the total matching score; w1 and w2 represent the weights of semantic similarity and pixel-level matching, respectively, and satisfy w1+w2=1.

[0038] Preferably, the method further comprises:

[0039] S4, optimizing a calculation method for the matching degree of input data and output data based on the matching degree of input data and output data.

[0040] A second aspect of the present invention provides an AI evaluation system for AI-generated content matching evaluation, comprising:

[0041] An input layer, used to generate input and output data of the AI ​​model;

[0042] A matching degree calculation layer, used for receiving the input data and the output data, and calculating the matching degree of the input data and the output data;

[0043] The output layer is used to output the matching degree between the input data and the output data to the user or other system components for display and further optimization calculation.

[0044] Preferably, the system further comprises:

[0045] An optimizer is used to optimize the matching degree calculation method of the input data and output data of the matching degree calculation layer based on the matching degree, input data and output data.

[0046] A third aspect of the present invention provides an electronic device, comprising a processor and a memory, wherein the memory stores a plurality of instructions, and the processor is configured to read the instructions and execute the method described in the first aspect.

[0047] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a plurality of instructions, and the plurality of instructions can be read by a processor and execute the method described in the first aspect.

[0048] The AI ​​evaluation method and system based on AI generated content matching evaluation provided by the present invention have the following beneficial effects:

[0049] It can effectively evaluate the performance of AI models for text-to-image, text-to-video, text-to-audio, image-to-text, and image-to-audio. Models with high input-output matching are given high scores, and vice versa. It also guides the optimization of AI models, enables self-optimization, and continuously improves the quality of generated content. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a flow chart of an AI evaluation method based on AI-generated content matching evaluation according to a preferred embodiment of the present invention.

[0051] Figure 2 This is a schematic diagram of the principle architecture of an AI evaluation system for AI-generated content matching evaluation according to a preferred embodiment of the present invention.

[0052] Figure 3 A schematic structural diagram of an embodiment of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0053] like Figure 1 As shown, this embodiment provides an AI evaluation method based on AI-generated content matching evaluation, and the AI ​​evaluation is performed on a single AI model, including: S1, generating input data and output data of the AI ​​model, wherein the output data is content generated by the AI ​​model based on the input data; including: S11, generating input data of the AI ​​model, wherein the input data is text, image, audio, video or other types of data; S12, inputting the input data into the AI ​​model to obtain the output data of the AI ​​model, wherein the output data is text, image, audio, video or other types of data, and the types of the input data and the output data do not need to be consistent.

[0054] As a preferred implementation, the S1 further includes: S13, preprocessing the input data and the output data.

[0055] In this embodiment, the preprocessing includes:

[0056] 1. If the input data and / or output data is text, then perform word segmentation, stop word removal and word embedding operations; (1) Word Segmentation: Segment the sentence to form multiple phrases; (2) Stop word removal: When using jieba for Chinese word segmentation, all words appearing in the sentence will be divided, and some words have no practical meaning, which adds unnecessary workload to the subsequent keyword extraction and causes the extracted keywords to be invalid. Therefore, stop words will be introduced after the word segmentation process to optimize the word segmentation results. For common words, users can manually add a txt file and import the file when needed; or use a stop word list that has been sorted out; based on the existing stop word list, you can also delete or add it to improve it. Stop words include but are not limited to punctuation marks, function words, etc. (3) Word embedding: Word embedding is a general term for language models and representation learning technologies in natural language processing (NLP). Conceptually, it means embedding a high-dimensional space with the number of all words into a continuous vector space with much lower dimension, where each word or phrase is mapped to a vector in the real field. Word embedding methods include artificial neural networks, dimensionality reduction of word co-occurrence matrices, probability models, and explicit representation of the context in which the word is located. In the underlying input, the method of using word embedding to represent phrases has greatly improved the performance of grammatical analyzers and text sentiment analysis in NLP.

[0057] 2. If the input data and / or output data is an image, pixel normalization and feature extraction are performed.

[0058] (1) Pixel standardization: Pixel data is centralized by removing the mean. According to convex optimization theory and data probability distribution, data centralization conforms to the law of data distribution and is more likely to achieve generalization effect after training. The mathematical formula is (input data - data mean) / data standard deviation. The ultimate goal of this method is to adjust the pixel values ​​in the image so that they obey the standard normal distribution or have a specific mean and standard deviation.

[0059] (2) Feature extraction: Features in an image refer to distinctive, representative, and different attributes. Different features can be used for different classification tasks. In image processing, features are divided into structural features and statistical features. Among them: Structural features: describe the characteristics of an image based on some attributes of the pixels themselves, including:

[0060] A. Edge features: The edge is the dividing line between two areas of different grayscale on the image. The edge features can be extracted by edge detection algorithms;

[0061] B. Corner point features: Corner points are turning points in local areas of an image and can be used for tracking and target detection;

[0062] C. Texture features: Texture is a regular, repetitive pattern in space or color on an image, which is used for texture recognition;

[0063] D. Shape features: Shape can describe the geometric shape of an object, such as circle, oval or rectangle.

[0064] Statistical features: The characteristics of an image are described by the statistical distribution of the grayscale values ​​of each pixel in the image, including:

[0065] A. Histogram: describes the number of occurrences of the gray value of each pixel in the image;

[0066] B. Mean and variance: The mean represents the average value of the grayscale values ​​of pixels in the image area, and the variance represents the degree of variation of the grayscale values ​​of pixels in the image area;

[0067] C. Energy and Entropy: Energy represents the degree of good distribution of pixels in an image area, and entropy represents the amount of information of pixels in an image area.

[0068] 3. If the input data and / or output data is audio, perform speech framing and windowing operations.

[0069] (1) Speech framing: The number of frames that the signal can be divided into is calculated based on the signal length, frame shift, and frame length. Number of frames = (signal length - frame length) / frame shift + 1. During the framing operation, the remaining signal may be less than one frame. In this case, it is necessary to add frames to this section of the signal.

[0070] (2) Windowing: Each frame of the divided signal is multiplied by the window function in turn; after the division, there will be discontinuities at the beginning and end of each frame, so the more frames are divided, the greater the error with the original signal. Windowing is used to solve this problem, making the signal continuous after framing, and each frame will show the characteristics of a periodic function. Common window functions include: rectangular window, Hamming window, Hanning window, etc.

[0071] 4. If the input data and / or output data is a video, video shot segmentation or shot boundary detection, key frame extraction and video feature extraction are performed as preprocessing.

[0072] (1) Video shot segmentation or shot boundary detection: Video segmentation is the basis of video processing and analysis. The changes in video features within the same shot are caused by the movement of the object / camera and the change of light. The transformation methods of shot quality inspection include sudden changes and gradual changes. Video shot segmentation methods include pixel difference method, histogram-based method, block matching method, motion-based method and contour-based method.

[0073] (2) Key frame extraction: A key frame refers to one or more images that represent the most important and representative parts of a shot. Depending on the complexity of the shot content, one or more key frames can be extracted from a shot. Typical extraction methods include the first and last frame method, the middle frame method, the color, texture, and shape feature method, the motion analysis method, and the clustering method.

[0074] (3) Video feature extraction: Video features are divided into static features and dynamic features. Static features are image features of key frames, including color features, texture features, and shape features. The feature extraction method for key frames is the same as that for general static images. Dynamic features are unique to video data, including global motion and local motion. It is difficult to describe the motion changes of video sequences using only the image features of representative frames. Therefore, feature extraction and optimization can be performed based on the motion vector extraction technology of the optical flow field.

[0075] It also includes S2, receiving the input data and output data, and calculating the matching degree of the input data and output data. In this embodiment, S2 includes: S21, receiving the input data and output data after the preprocessing; S22, calculating the matching degree of the input data and output data after the preprocessing.

[0076] As a preferred implementation, the matching degree calculation method in S22 depends on the types of input and output data.

[0077] 1. For a text-to-image generation model, for example, in this embodiment, the input data is a text describing a scene, and the output data is a generated image. The matching degree calculation method includes: (1) extracting key information of the input text and the output image; (2) using semantic similarity and pixel-level matching as the matching degree evaluation of the input data and the output data.

[0078] In this embodiment, the calculation formula of semantic similarity is formula (1):

[0079] S_semantic=Sum(cosine_similarity(T_i, I_i)) / N (1).

[0080] Among them, S_semantic represents semantic similarity; T_i represents the word vector of the i-th keyword in the input text; I_i represents the feature vector of the i-th key feature in the output image; N represents the number of keywords / key features; cosine_similarity represents the cosine similarity function; Sum represents the summation function.

[0081] In this embodiment, the calculation formula of the pixel-level matching degree is formula (2):

[0082] S_pixel=1-1 / (1+exp(-Sum(abs(P_i-Q_i)) / M)) (2);

[0083] Among them, S_pixel represents the matching degree at the pixel level; P_i represents the i-th pixel value of the output image; Q_i represents the i-th pixel value of the standard image; M represents the number of pixel values; abs represents the absolute value function, and exp represents the exponential function.

[0084] As a preferred implementation, the matching evaluation based on semantic similarity and pixel-level matching as input data and output data includes: obtaining a total matching score by weighted averaging the semantic similarity and pixel-level matching, and using the total matching score as input and output matching evaluation, wherein the calculation formula of the total matching score is as shown in formula (3):

[0085] S_total=w1*S_semantic+w2*S_pixel (3).

[0086] Wherein, S_total represents the total matching score; w1 and w2 represent the weights of semantic similarity and pixel-level matching, respectively, and satisfy w1+w2=1.

[0087] Experimental results show that the method and system of the present invention can effectively evaluate the performance of the text-to-image generation model. A high score is given to the model with a high degree of input and output matching, and a low score is given otherwise, thereby providing guidance for the optimization of the AI ​​model.

[0088] 2. For text-to-text AI models, the matching degree is calculated by using word embedding and semantic similarity calculation to compare the matching degree of input and output texts; where word embedding is the distributed representation of words obtained using the word feature matrix.

[0089] The method further includes S3, outputting the matching degree between the input data and the output data to a user or other system components for display and further optimization calculation.

[0090] As a preferred implementation, the method further includes S4, optimizing a calculation method for the matching degree between input data and output data based on the matching degree, input data and output data.

[0091] As a preferred implementation, the optimization includes: selecting a more suitable image feature extraction method, improving the semantic similarity calculation formula, the pixel-level matching calculation formula, weight adjustment, etc.

[0092] like Figure 2As shown, this embodiment provides an AI evaluation system for AI-generated content matching evaluation, including: an input layer 101, used to generate input and output data of the AI ​​model; a matching calculation layer 102, used to receive the input data and output data, and calculate the matching degree of the input data and output data; an output layer 103, used to output the matching degree of the input data and output data to a user or other system components for display and further optimization calculation.

[0093] As a preferred implementation, the system further includes: an optimizer 104, which is used to optimize the matching degree calculation method of the input data and output data of the matching degree calculation layer based on the matching degree, input data and output data.

[0094] The present invention also provides a memory storing a plurality of instructions, wherein the instructions are used to implement the method as in the first embodiment.

[0095] like Figure 3 As shown, the present invention also provides an electronic device, including a processor 301 and a memory 302 connected to the processor 301, the memory 302 stores a plurality of instructions, and the instructions can be loaded and executed by the processor so that the processor can execute the method as in the first embodiment.

[0096] Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent substitutions for some of the technical features therein; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An AI evaluation method based on AI generated content matching evaluation, wherein the AI ​​evaluation is performed on a single AI model, characterized in that: include: S1, generating input data and output data of the AI ​​model, wherein the output data is content generated by the AI ​​model according to the input data; S2, receiving the input data and the output data, and calculating the matching degree between the input data and the output data; S3, outputting the matching degree between the input data and the output data to a user or other system components for display and further optimization calculation; The S1 includes: S11, generating input data for an AI model, wherein the input data is text, image, audio or video; S12, inputting the input data into the AI ​​model to obtain output data of the AI ​​model, where the output data is text, image, audio or video; the types of the input data and the output data do not need to be consistent; The S1 further comprises: S13, preprocessing the input data and the output data; the preprocessing includes: If the input data and / or output data is text, perform word segmentation, stop word removal and word embedding operations; If the input data and / or output data is an image, performing pixel normalization and feature extraction; If the input data and / or output data is audio, performing speech framing and windowing operations; If the input data and / or output data is a video, performing video shot segmentation or shot boundary detection, key frame extraction, and video feature extraction; The S2 includes: S21, receiving the pre-processed input data and output data; S22, calculating the matching degree between the pre-processed input data and the output data; The matching degree calculation method of S22 depends on the types of input data and output data; For AI models whose input data is text and whose output data is image, the matching evaluation of input data and output data includes: (1) Extract key information of input data and output data; (2) Semantic similarity and pixel-level matching are used as the matching evaluation of input data and output data; The calculation formula of the semantic similarity is formula (1): S_semantic=Sum(cosine_similarity(T_i,I_i)) / N (1); Among them, S_semantic represents semantic similarity; T_i represents the word vector of the i-th keyword in the input text; I_i represents the feature vector of the i-th key feature in the output image; N represents the number of keywords or key features; cosine_similarity represents the cosine similarity function; Sum represents the summation function; The calculation formula of pixel-level matching is as follows: S_pixel=1-1 / (1+exp(-Sum(abs(P_i-Q_i)) / M)) (2); Among them, S_pixel represents the matching degree at the pixel level; P_i represents the i-th pixel value of the output image; Q_i represents the i-th pixel value of the standard image; M represents the number of pixel values; abs represents the absolute value function, and exp represents the exponential function; The matching evaluation based on semantic similarity and pixel-level matching as input data and output data includes: obtaining a total matching score by weighted average of semantic similarity and pixel-level matching, and using the total matching score as the matching evaluation of input data and output data, wherein the calculation formula of the total matching score is as shown in formula (3): S_total= w1* S_semantic+w2*S_pixel (3); Among them, S_total represents the total matching score; w1 and w2 represent the weights of semantic similarity and pixel-level matching, respectively, and satisfy w1+w2=1.

2. According to claim 1, an AI evaluation method based on AI generated content matching evaluation is characterized in that: The method further comprises: S4, optimizing a calculation method for the matching degree of input data and output data based on the matching degree of input data and output data.

3. An AI evaluation system based on AI generated content matching evaluation, used to implement the AI ​​evaluation method based on AI generated content matching evaluation according to any one of claims 1-2, characterized in that: include: An input layer (101), used to generate input data and output data of the AI ​​model; A matching degree calculation layer (102), used for receiving the input data and the output data, and calculating the matching degree between the input data and the output data; The output layer (103) is used to output the matching degree between the input data and the output data to the user or other system components for display and further optimization calculation.

4. The AI ​​evaluation system based on AI generated content matching evaluation according to claim 3 is characterized in that: The system further comprises: The optimizer (104) is used for optimizing the matching degree calculation method of the input data and output data of the matching degree calculation layer based on the matching degree of the input data and the output data and the input data and the output data.

5. An electronic device, characterized in that: It includes a processor and a memory, the memory stores multiple instructions, and the processor is used to read the instructions and execute the AI ​​evaluation method based on AI-generated content matching evaluation as described in any one of claims 1-2.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the plurality of instructions can be read by a processor and executed as described in any one of claims 1-2 as an AI evaluation method for AI-generated content matching evaluation.

Images