Image data element value evaluation method and device, terminal and medium

Abstract

The invention discloses an image data element value evaluation method and device, a terminal and a medium, and the method comprises the steps: obtaining to-be-evaluated image data and a description text corresponding to the image data, carrying out the feature extraction of the image data and the description text, carrying out the recognition of an entity in an image, and carrying out the recognition of a key target, according to the method, feature vectors of image data and a description text, an entity object recognition result of the image data and a key entity recognition result are obtained, and a comprehensive evaluation result is obtained by integrating the feature similarity of the image and the description text, the richness of the number of entity object classification labels and a key target detection result. Therefore, the evaluation result is more comprehensive and more accurate, the automatic evaluation of the value of the image data elements is realized, and the evaluation efficiency is remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of AI technology, and in particular to a method, device, terminal and medium for evaluating the value of image data elements. Background Art

[0002] With the advent of the era of big models, the value of data has become increasingly prominent, and with the popularization of mobile phones and cameras, the scale of image data is becoming increasingly large. However, how to effectively evaluate image data elements and their value has become an urgent problem to be solved.

[0003] Traditional methods for assessing the value of image data elements often rely on manual annotation or setting simple rules such as the number of images, average image resolution, and average image size. This is inefficient and difficult to ensure accuracy when faced with large-scale and diverse image data. Summary of the invention

[0004] The present application provides a method, device, terminal and medium for evaluating the value of image data elements, which are used to solve the technical problems that the existing image data element value evaluation methods are inefficient and difficult to ensure accuracy when faced with large-scale and diverse image data.

[0005] In order to solve the above technical problems, the first aspect of the present application provides a method for evaluating the value of image data elements, comprising:

[0006] Acquire image data and description text corresponding to the image data;

[0007] By using a preset feature extraction model, feature extraction is performed on the image data and the description text respectively to obtain an image feature vector and a text feature vector;

[0008] Calculating feature vector similarity between the image feature vector and the text feature vector to determine a similarity score based on the feature vector similarity;

[0009] Identify the entity objects contained in the image data through a preset visual recognition model, and determine the entity category richness score according to the number of categories of the entity objects;

[0010] Extract keywords from the description text to obtain an entity keyword set, match the entity keyword set with the entity object to obtain a key entity category set of the image data, and determine a key entity score based on the key entity category set;

[0011] A comprehensive score is determined according to the similarity score, the entity category richness score and the key entity score, and an element value assessment result of the image data is determined by the comprehensive score.

[0012] Preferably, when the image data is an image data set containing multiple images, the image data is subjected to feature extraction by a preset feature extraction model to obtain an image feature vector including:

[0013] By using a preset feature extraction model, feature extraction is performed on each image data in the image data set to obtain a feature vector corresponding to each image data;

[0014] The mean of each feature vector is calculated to obtain an average feature vector, and the average image feature vector is used as the image feature vector.

[0015] Preferably, the preset feature extraction model is: Chinese-CLIP visual large model.

[0016] Preferably, the calculating the feature vector similarity between the image feature vector and the text feature vector comprises:

[0017] The feature vector similarity between the image feature vector and the text feature vector is calculated by using a cosine similarity calculation method.

[0018] Preferably, identifying the entity objects contained in the image data by a preset visual recognition model, and determining the entity category richness score according to the number of categories of the entity objects comprises:

[0019] Identify the entity objects contained in the image data through a preset RAM visual recognition model, and classify the entity objects according to the categories of the entity objects to obtain a set of entity object categories;

[0020] Performing mean calculation based on the entity object category set and the number of entity object categories to obtain entity category richness;

[0021] According to the entity category richness, in combination with a preset association between the entity category richness and the entity category richness score, an entity category richness score is determined.

[0022] Preferably, determining the key entity score according to the key entity category set includes:

[0023] The key entity category richness is obtained by performing mean calculation based on the key entity category set and the number of key entity categories, and then the key entity score is determined based on the ratio of the key entity category richness to the number of keywords in the entity keyword set.

[0024] Preferably, determining a comprehensive score according to the similarity score, the entity category richness score and the key entity score comprises:

[0025] A weighted sum of the similarity score, the entity category richness score and the key entity score is calculated, and the weighted sum is used as a comprehensive score.

[0026] At the same time, the second aspect of the present application provides an image data element value assessment device, including:

[0027] A data acquisition unit, used to acquire image data and description text corresponding to the image data;

[0028] A feature vector extraction unit, used to extract features from the image data and the description text respectively through a preset feature extraction model to obtain an image feature vector and a text feature vector;

[0029] An element similarity scoring unit, used to calculate the feature vector similarity between the image feature vector and the text feature vector, so as to determine a similarity score according to the feature vector similarity;

[0030] An entity element richness scoring unit, used to identify the entity objects contained in the image data through a preset visual recognition model, and determine the entity category richness score according to the number of categories of the entity objects;

[0031] A key entity element scoring unit is used to extract keywords from the description text to obtain an entity keyword set, match the entity keyword set with the entity object to obtain a key entity category set of the image data, and determine a key entity score based on the key entity category set;

[0032] The element value result determination unit is used to determine a comprehensive score according to the similarity score, the entity category richness score and the key entity score, and determine the element value evaluation result of the image data with the comprehensive score.

[0033] A third aspect of the present application provides an image data element value assessment terminal, comprising: a memory and a processor;

[0034] The memory is used to store program codes, and the program codes correspond to an image data element value assessment method provided in the first aspect of the present application;

[0035] The processor is used for reading and executing the program code.

[0036] The fourth aspect of the present application provides a computer-readable storage medium, in which program code is stored. The program code is used to be read and executed by a processor to implement an image data element value assessment method as provided in the first aspect of the present application.

[0037] It can be seen from the above technical solutions that this application has the following advantages:

[0038] The solution provided in the present application first obtains the image data to be evaluated and the descriptive text corresponding to the image data, and then performs feature extraction, image-contained entity recognition, and key target recognition on the image data and the descriptive text, respectively, to obtain feature vectors of the image data and the descriptive text, entity object recognition results of the image data, and key entity recognition results, and then obtains a comprehensive evaluation result by comprehensively considering the feature similarity between the image and the descriptive text, the number and richness of entity object classification labels, and the key target detection results, so that the evaluation result is more comprehensive and accurate, and at the same time realizes the automated evaluation of the value of image data elements, which significantly improves the efficiency of the evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0040] Figure 1 A flowchart of an embodiment of a method for evaluating the value of image data elements provided in this application.

[0041] Figure 2 A schematic diagram of the structure of an embodiment of an image data element value assessment device provided in the present application.

[0042] Figure 3 A schematic diagram of the structure of an embodiment of an image data element value assessment terminal provided in the present application. DETAILED DESCRIPTION

[0043] The embodiments of the present application provide a method, device, terminal and medium for evaluating the value of image data elements, which are used to solve the technical problems that the existing methods for evaluating the value of image data elements are inefficient and difficult to ensure accuracy when faced with large-scale and diverse image data.

[0044] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described below are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0045] First, a detailed description of an embodiment of an image data element value assessment method provided in the first aspect of the present application is as follows:

[0046] See also Figure 1 , the present application provides an embodiment of an image data element value assessment method, comprising:

[0047] Step 101: Obtain image data and description text corresponding to the image data.

[0048] It should be noted that when implementing the solution provided by this application, first, it is necessary to obtain image data and corresponding description text, wherein the image data is an image that needs to be evaluated for element value, and the description text is used to record the content summary of the image data.

[0049] Step 102: Using a preset feature extraction model, feature extraction is performed on the image data and the description text respectively to obtain an image feature vector and a text feature vector.

[0050] Step 103: Calculate the feature vector similarity between the image feature vector and the text feature vector to determine a similarity score based on the feature vector similarity.

[0051] It should be noted that, based on the image data and description text obtained in step 101, in accordance with the process of steps 102 to 103, feature extraction is performed on the image data and description text respectively through a pre-trained feature extraction model to obtain an image feature vector and a text feature vector, and then the feature vector similarity between the image feature vector and the text feature vector is calculated. Then, based on the calculated feature vector similarity, the corresponding similarity score is determined, which can be used as one of the bases for evaluating the value of the image data element.

[0052] More specifically, the similarity calculation method of this embodiment can adopt the cosine similarity calculation method, such as , where Sim represents the feature vector similarity between the image feature vector and the text feature vector, represents the image feature vector, represents the text feature vector. Then, based on the feature vector similarity, the similarity score S1 is calculated. .

[0053] The feature extraction model mentioned in this embodiment can preferably adopt the Chinese-CLIP visual model. The core concept of this model is pre-training of image-text comparison learning. It is a multimodal learning model that aims to associate images and texts to help users quickly achieve image-text feature & similarity calculation, cross-modal retrieval, zero-sample image classification and other tasks in the Chinese field. In some embodiments, in addition to Chinese-CLIP, image-text representation models such as Wukong, Taiyi, and R2D2 can also be used as substitutes for Chinese-CLIP.

[0054] In some embodiments, when the image data is an image data set including multiple images, extracting features from the image data in step 101 of this embodiment to obtain an image feature vector may further include:

[0055] 1) Through the preset feature extraction model, feature extraction is performed on each image data in the image data set to obtain the feature vector corresponding to each image data;

[0056] 2) Calculate the mean of each feature vector to obtain the average feature vector, and use the average image feature vector as the image feature vector.

[0057] It should be noted that the Chinese-CLIP model is used to obtain the feature vectors of all images in the image dataset D , and then the average feature vector of the data set is obtained by performing mean calculation, which is used as the image feature vector for calculating the similarity score , and its calculation formula is as follows: , where n1 is the number of images in the image dataset.

[0058] Step 104: Identify entity objects contained in the image data using a preset visual recognition model, and determine entity category richness scores based on the number of categories of the entity objects.

[0059] It should be noted that based on the image data obtained in step 101, the entity objects contained in the image data are identified through a pre-trained visual recognition model, and then the identified entity objects are classified according to different categories of the entity objects, so that a set of entity object categories can be obtained. , and based on this set, determine the entity category richness score S2, which can be used as the second basis for evaluating the value of image data elements.

[0060] More specifically, the entity category richness score is determined as shown in the following example:

[0061] 1) Identify the entity objects contained in the image data through the preset RAM visual recognition model, classify the entity objects according to the categories of the entity objects, and obtain a set of entity object categories;

[0062] 2) Calculate the mean based on the entity object category set and the number of entity object categories to obtain the entity category richness;

[0063] 3) According to the entity category richness, combined with the preset correlation between the entity category richness and the entity category richness score, the entity category richness score is determined.

[0064] It should be noted that RAM (Recognize Anything Model): a large visual model for recognizing everything, can automatically recognize more than 6,400 types of object labels with a high accuracy. This embodiment uses the RAM visual recognition model to perform entity object recognition on image data and obtain the number of recognized object categories. Then, the mean is calculated to get the average number of object categories in the data set as the entity category richness , and its calculation formula is: , where n2 is the number of elements in the entity object category set.

[0065] Regarding the relationship between the preset entity category richness and the entity category richness score, the preset threshold value can be used. When it is less than the threshold a, ;when When it is greater than or equal to the threshold a, .

[0066] Step 105: extract keywords from the description text to obtain an entity keyword set, match the entity keyword set with the entity object to obtain a key entity category set of the image data, and determine the key entity score based on the key entity category set.

[0067] It should be noted that based on the description text obtained in step 101, keyword extraction is performed on the description text, and keywords related to the entity object, such as pedestrian, motor vehicle, non-motor vehicle, etc., are screened out to obtain an entity keyword set, which can be recorded as , m represents the number of object keywords, and then the keywords As the text prompt of the Grounding DINO model, the number of key object detection categories of all images in the image dataset D is obtained to obtain the key entity category set of the image data, which can be recorded as ,Finally, according to the set of key entity categories, the key ,entity score S3 is determined, which can be used as the third basis for ,evaluation of the value of image data elements.

[0068] More specifically, the above-mentioned key entity score calculation method can refer to the following example:

[0069] The key entity category richness is obtained by performing mean calculation based on the key entity category set and the number of key entity categories. The key entity score is then determined based on the ratio of the key entity category richness to the number of keywords in the entity keyword set.

[0070] By calculating the mean value of the key entity category set and the number of key entity categories, we can get the key entity category richness representing the average number of key object categories in the data set, which is denoted as , and its calculation formula is , where n3 is the number of elements in the key entity category set.

[0071] Finally, according to the ratio of the key entity category richness to the number of keywords in the entity keyword set, the key entity score S3 can be obtained, and its calculation formula can be expressed as: .

[0072] Step 106: Determine a comprehensive score based on the similarity score, entity category richness score and key entity score, and use the comprehensive score to determine the element value assessment result of the image data.

[0073] Finally, calculate the comprehensive score Among them, a, b, and c are weight coefficients of the three scores, with a value range of [0, 1], which can be dynamically adjusted according to the actual situation; if a, b, and c are all 1, then the total score range is [-100, 300]. The higher the score, the more consistent the image dataset is with the description, and the higher the value.

[0074] In addition to the above-mentioned method of calculating the weighted sum as the comprehensive score, in some embodiments, ordinary summation, ordinary product or weighted product can also be used as the method of calculating the comprehensive score, which will not be described in detail here.

[0075] The above is a detailed description of an embodiment of an image data element value assessment method provided by the present application. The following is a detailed description of an embodiment of an image data element value assessment device provided by the present application.

[0076] See also Figure 2 , the present application provides an embodiment of an image data element value assessment device, comprising:

[0077] The data acquisition unit 201 is used to acquire image data and description text corresponding to the image data;

[0078] A feature vector extraction unit 202 is used to extract features from the image data and the description text respectively through a preset feature extraction model to obtain an image feature vector and a text feature vector;

[0079] The element similarity scoring unit 203 is used to calculate the feature vector similarity between the image feature vector and the text feature vector, so as to determine the similarity score according to the feature vector similarity;

[0080] The entity element richness scoring unit 204 is used to identify the entity objects contained in the image data through a preset visual recognition model, and determine the entity category richness score according to the number of categories of the entity objects;

[0081] The key entity element scoring unit 205 is used to extract keywords from the description text to obtain an entity keyword set, match the entity keyword set with the entity object to obtain a key entity category set of the image data, and determine the key entity score according to the key entity category set;

[0082] The element value result determination unit 206 is used to determine a comprehensive score according to the similarity score, the entity category richness score and the key entity score, and determine the element value evaluation result of the image data based on the comprehensive score.

[0083] In addition to the above-mentioned image data element value assessment device embodiment, the present application provides a detailed description of an image data element value assessment terminal embodiment and a computer-readable storage medium embodiment.

[0084] See also Figure 3 , the third aspect of the present application provides an image data element value assessment terminal, the implementation types of the terminal include: personal computers, industrial computers, servers and embedded intelligent devices, the main components of the terminal include: a memory 33 and a processor 31, wherein the memory 33 and the processor 31 can be connected via a communication bus 34;

[0085] The memory is used to store program codes, and the program codes correspond to a method for evaluating the value of image data elements provided in the above embodiment;

[0086] The processor is used to read and execute program code.

[0087] The present application provides a computer-readable storage medium, in which program code is stored. The program code is used to be read and executed by a processor to implement an image data element value assessment method as provided in the above embodiment.

[0088] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the terminals, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0089] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0090] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0091] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0092] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0093] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0094] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0095] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for evaluating the value of image data elements, characterized in that: include: Acquire image data and description text corresponding to the image data; Using a preset feature extraction model, feature extraction is performed on the image data and the description text respectively to obtain an image feature vector and a text feature vector; Calculating feature vector similarity between the image feature vector and the text feature vector to determine a similarity score based on the feature vector similarity; Identify the entity objects contained in the image data through a preset visual recognition model, and determine the entity category richness score according to the number of categories of the entity objects; Extract keywords from the description text to obtain an entity keyword set, match the entity keyword set with the entity object to obtain a key entity category set of the image data, and determine a key entity score based on the key entity category set; A comprehensive score is determined according to the similarity score, the entity category richness score and the key entity score, and an element value assessment result of the image data is determined by the comprehensive score.

2. The method for evaluating the value of image data elements according to claim 1, characterized in that: When the image data is an image data set containing multiple images, the image data is subjected to feature extraction by a preset feature extraction model to obtain an image feature vector including: By using a preset feature extraction model, feature extraction is performed on each image data in the image data set to obtain a feature vector corresponding to each image data; The mean of each feature vector is calculated to obtain an average feature vector, and the average image feature vector is used as the image feature vector.

3. The method for evaluating the value of image data elements according to claim 1, characterized in that: The preset feature extraction model is: Chinese-CLIP visual large model.

4. The method for evaluating the value of image data elements according to claim 1, characterized in that: The calculating the feature vector similarity between the image feature vector and the text feature vector comprises: The feature vector similarity between the image feature vector and the text feature vector is calculated by using a cosine similarity calculation method.

5. The method for evaluating the value of image data elements according to claim 1, characterized in that: The identifying the entity objects contained in the image data by using a preset visual recognition model, and determining the entity category richness score according to the number of categories of the entity objects includes: Identify the entity objects contained in the image data through a preset RAM visual recognition model, and classify the entity objects according to the categories of the entity objects to obtain a set of entity object categories; Performing mean calculation based on the entity object category set and the number of entity object categories to obtain entity category richness; According to the entity category richness, in combination with a preset association between the entity category richness and the entity category richness score, an entity category richness score is determined.

6. The method for evaluating the value of image data elements according to claim 1, characterized in that: Determining the key entity score according to the key entity category set includes: The key entity category richness is obtained by performing mean calculation based on the key entity category set and the number of key entity categories, and then the key entity score is determined based on the ratio of the key entity category richness to the number of keywords in the entity keyword set.

7. The method for evaluating the value of image data elements according to claim 1, characterized in that: Determining a comprehensive score according to the similarity score, the entity category richness score and the key entity score comprises: A weighted sum of the similarity score, the entity category richness score and the key entity score is calculated, and the weighted sum is used as a comprehensive score.

8. An image data element value assessment device, characterized in that: include: A data acquisition unit, used to acquire image data and description text corresponding to the image data; A feature vector extraction unit, used to extract features from the image data and the description text respectively through a preset feature extraction model to obtain an image feature vector and a text feature vector; An element similarity scoring unit, used to calculate the feature vector similarity between the image feature vector and the text feature vector, so as to determine a similarity score according to the feature vector similarity; An entity element richness scoring unit, used to identify the entity objects contained in the image data through a preset visual recognition model, and determine the entity category richness score according to the number of categories of the entity objects; A key entity element scoring unit is used to extract keywords from the description text to obtain an entity keyword set, match the entity keyword set with the entity object to obtain a key entity category set of the image data, and determine a key entity score based on the key entity category set; The element value result determination unit is used to determine a comprehensive score according to the similarity score, the entity category richness score and the key entity score, and determine the element value evaluation result of the image data with the comprehensive score.

9. An image data element value assessment terminal, characterized in that: include: Memory and processor; The memory is used to store program codes, and the program codes correspond to an image data element value assessment method according to any one of claims 1 to 7; The processor is used for reading and executing the program code.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, and the program code is used to be read and executed by a processor to implement an image data element value assessment method as described in any one of claims 1 to 7.

Images