An image data element value evaluation method and device, a terminal and a medium

Abstract

The application discloses an image data element value evaluation method and device, a terminal and a medium. The scheme provided by the application first acquires image data to be evaluated and description text corresponding to the image data, then performs feature extraction, image-included entity identification and key target identification on the image data and the description text respectively to obtain feature vectors of the image data and the description text, entity object identification results of the image data and key entity identification results, and finally obtains a comprehensive evaluation result by comprehensively considering feature similarity of the image and the description text, entity object classification label quantity richness and key target detection results, so that the evaluation result is more comprehensive and more accurate, and the automatic evaluation of the image data element value is realized, and the evaluation efficiency is significantly improved.

Description

Technical Field

[0001] This application relates to the field of AI technology, and in particular to a method, apparatus, terminal and medium for evaluating the value of image data elements. Background Technology

[0002] With the advent of the era of large-scale models, the value of data is becoming increasingly prominent, and with the widespread use of mobile phones and cameras, the scale of image data is growing ever larger. However, how to effectively evaluate the elements of image data and their value has become an urgent problem to be solved.

[0003] Traditional methods for evaluating the value of image data elements often rely on manual annotation or setting simple rules such as the number of images, average image resolution, and average image size. This is inefficient and difficult to guarantee accuracy when dealing with large-scale and diverse image data. Summary of the Invention

[0004] This application provides a method, apparatus, terminal, and medium for evaluating the value of image data elements, which addresses the technical problem that existing methods for evaluating the value of image data elements are inefficient and inaccurate when dealing with large-scale and diverse image data.

[0005] To address the aforementioned technical problems, the first aspect of this application provides a method for evaluating the value of image data elements, comprising:

[0006] Acquire image data and the corresponding descriptive text for the image data;

[0007] Using a preset feature extraction model, features are extracted from the image data and the descriptive text respectively to obtain image feature vectors and text feature vectors.

[0008] Calculate the feature vector similarity between the image feature vector and the text feature vector, and determine the similarity score based on the feature vector similarity;

[0009] The system identifies entity objects contained in the image data using a preset visual recognition model, and determines the entity category richness score based on the number of categories of the entity objects.

[0010] Keyword extraction is performed on the description text to obtain a set of entity keywords. The set of entity keywords is matched with the entity objects to obtain a set of key entity categories of the image data. The key entity scores are determined based on the set of key entity categories.

[0011] Based on the similarity score, the entity category richness score, and the key entity score, a comprehensive score is determined, and the element value assessment result of the image data is determined using the comprehensive score.

[0012] Preferably, when the image data is an image dataset containing multiple images, feature extraction is performed on the image data using a preset feature extraction model to obtain image feature vectors, including:

[0013] By using a preset feature extraction model, features are extracted from each image data in the image dataset to obtain the feature vector corresponding to each image data.

[0014] The mean of each feature vector is calculated to obtain the average feature vector, and the average feature vector is used as the image feature vector.

[0015] Preferably, the preset feature extraction model is: Chinese-CLIP visual large model.

[0016] Preferably, calculating the feature vector similarity between the image feature vector and the text feature vector includes:

[0017] The similarity between the image feature vector and the text feature vector is calculated using cosine similarity.

[0018] Preferably, the step of identifying entity objects contained in the image data using a preset visual recognition model, and determining the entity category richness score based on the number of categories of the entity objects, includes:

[0019] Using a preset RAM visual recognition model, the entity objects contained in the image data are identified, and the entity objects are classified according to their categories to obtain a set of entity object categories.

[0020] The entity category richness is obtained by averaging the set of entity object categories and the number of entity object categories.

[0021] Based on the entity category richness, and combined with the preset correlation between entity category richness and entity category richness score, the entity category richness score is determined.

[0022] Preferably, determining the key entity score based on the set of key entity categories includes:

[0023] The richness of key entity categories is obtained by averaging the set of key entity categories and the number of key entity categories. Then, the key entity score is determined by the ratio of the richness of key entity categories to the number of keywords in the set of entity keywords.

[0024] Preferably, determining the comprehensive score based on the similarity score, the entity category richness score, and the key entity score includes:

[0025] Calculate the weighted sum of the similarity score, the entity category richness score, and the key entity score, and use the weighted sum as the comprehensive score.

[0026] Meanwhile, a second aspect of this application provides an image data element value assessment device, comprising:

[0027] A data acquisition unit is used to acquire image data and the corresponding descriptive text of the image data;

[0028] The feature vector extraction unit is used to extract features from the image data and the descriptive text respectively using a preset feature extraction model to obtain image feature vectors and text feature vectors;

[0029] The element similarity scoring unit is used to calculate the feature vector similarity between the image feature vector and the text feature vector, so as to determine the similarity score based on the feature vector similarity.

[0030] The entity element richness scoring unit is used to identify entity objects contained in the image data through a preset visual recognition model, and to determine the entity category richness score based on the number of categories of the entity objects.

[0031] The key entity element scoring unit is used to extract keywords from the descriptive text to obtain a set of entity keywords, match the set of entity keywords with the entity objects to obtain a set of key entity categories of the image data, and determine the key entity score based on the set of key entity categories.

[0032] The element value result determination unit is used to determine a comprehensive score based on the similarity score, the entity category richness score, and the key entity score, and to determine the element value evaluation result of the image data based on the comprehensive score.

[0033] A third aspect of this application provides an image data element value assessment terminal, comprising: a memory and a processor;

[0034] The memory is used to store program code, which corresponds to an image data element value assessment method provided in the first aspect of this application;

[0035] The processor is used to read and execute the program code.

[0036] The fourth aspect of this application provides a computer-readable storage medium storing program code, which is read and executed by a processor to implement an image data element value assessment method as provided in the first aspect of this application.

[0037] As can be seen from the above technical solutions, this application has the following advantages:

[0038] The solution provided in this application first acquires the image data to be evaluated and the corresponding descriptive text. Then, it performs feature extraction, image-embedded entity recognition, and key target recognition on the image data and descriptive text, respectively, to obtain feature vectors of the image data and descriptive text, entity object recognition results of the image data, and key entity recognition results. Finally, by combining the feature similarity between the image and descriptive text, the richness of the number of entity object classification labels, and key target detection results, a comprehensive evaluation result is obtained, making the evaluation result more comprehensive and accurate. At the same time, it realizes the automated evaluation of the value of image data elements, significantly improving the evaluation efficiency. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart illustrating an embodiment of an image data element value assessment method provided in this application.

[0041] Figure 2 This is a schematic diagram of an embodiment of an image data element value assessment device provided in this application.

[0042] Figure 3 This is a schematic diagram of the structure of an embodiment of an image data element value assessment terminal provided in this application. Detailed Implementation

[0043] This application provides a method, apparatus, terminal, and medium for evaluating the value of image data elements, which addresses the technical problem that existing image data element value evaluation methods are inefficient and inaccurate when dealing with large-scale and diverse image data.

[0044] To make the inventive objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0045] First, a detailed description of an embodiment of an image data element value assessment method provided in the first aspect of this application is as follows:

[0046] Please see Figure 1 This application provides an embodiment of an image data element value assessment method, comprising:

[0047] Step 101: Obtain the image data and the corresponding descriptive text.

[0048] It should be noted that, when implementing the solution provided in this application, it is first necessary to obtain image data and corresponding descriptive text. The image data is an image for which the element value assessment is required, and the descriptive text is used to record a summary of the content of the image data.

[0049] Step 102: Using a preset feature extraction model, extract features from the image data and the descriptive text respectively to obtain image feature vectors and text feature vectors.

[0050] Step 103: Calculate the feature vector similarity between the image feature vector and the text feature vector, and determine the similarity score based on the feature vector similarity.

[0051] It should be noted that, based on the image data and descriptive text obtained in step 101, following the process of steps 102 to 103, features are extracted from the image data and descriptive text respectively using a pre-trained feature extraction model, thereby obtaining image feature vectors and text feature vectors. Then, the feature vector similarity between the image feature vector and the text feature vector is calculated. Based on the calculated feature vector similarity, the corresponding similarity score is determined. This score can be used as one of the bases for evaluating the value of image data elements.

[0052] More specifically, the similarity calculation method in this embodiment can adopt the cosine similarity calculation method, such as... In the formula, Sim represents the feature vector similarity between the image feature vector and the text feature vector. Represents the image feature vector. This represents the text feature vector. Next, a similarity score S1 is calculated based on the similarity of this feature vector. .

[0053] The feature extraction model mentioned in this embodiment can preferably be the Chinese-CLIP visual large model. The core concept of this model is image-text contrast learning pre-training, which is a multimodal learning model designed to associate images and text, helping users quickly achieve tasks such as image-text feature and similarity calculation, cross-modal retrieval, and zero-shot image classification in the Chinese domain. In some embodiments, in addition to Chinese-CLIP, image-text representation models such as Wukong, Taiyi, and R2D2 can also be used as alternatives to Chinese-CLIP.

[0054] In some embodiments, when the image data is an image dataset containing multiple images, step 101 of this embodiment, which involves feature extraction from the image data to obtain an image feature vector, may further include:

[0055] 1) Using a pre-defined feature extraction model, features are extracted from each image data in the image dataset to obtain the feature vector corresponding to each image data.

[0056] 2) Calculate the mean of each feature vector to obtain the average feature vector, and use the average feature vector as the image feature vector.

[0057] It should be noted that the Chinese-CLIP model was used to obtain the feature vectors of all images in the image dataset D. Then, by calculating the mean, the average feature vector of the dataset is obtained, which is used as the image feature vector for calculating the similarity score. The specific formula for its calculation is as follows: , where n1 is the number of images in the image dataset.

[0058] Step 104: Identify entity objects contained in the image data using a preset visual recognition model, and determine the entity category richness score based on the number of entity object categories.

[0059] It should be noted that, based on the image data obtained in step 101, a pre-trained visual recognition model is used to identify entity objects contained in the image data. Then, the identified entity objects are categorized according to their different categories to obtain a set of entity object categories. Based on this set, the entity category richness score S2 is determined, which can serve as a second basis for evaluating the value of image data elements.

[0060] More specifically, the method for determining the entity category richness score can be found in the following example:

[0061] 1) Using a pre-defined RAM visual recognition model, identify entity objects contained in image data, classify entity objects according to their categories, and obtain a set of entity object categories;

[0062] 2) Calculate the entity category richness by averaging the set of entity object categories and the number of entity object categories;

[0063] 3) Determine the entity category richness score based on the entity category richness and the pre-defined correlation between the entity category richness and the entity category richness score.

[0064] It should be noted that RAM (Recognize Anything Model) is a large-scale visual model for recognizing anything, capable of automatically recognizing over 6400 object categories with high accuracy. This embodiment uses the RAM visual recognition model to perform entity object recognition on image data, obtaining the number of recognized object categories. Next, the mean is calculated to obtain the average number of object categories in the dataset, which is used as the entity category richness. The formula for its calculation is: , where n2 is the number of elements in the entity object category set.

[0065] Regarding the correlation between the preset entity category richness and the entity category richness score, it can be determined based on a preset threshold. When it is less than the threshold a, ;when When greater than or equal to the threshold a, .

[0066] Step 105: Extract keywords from the description text to obtain a set of entity keywords. Match the set of entity keywords with entity objects to obtain a set of key entity categories for the image data. Determine the key entity score based on the set of key entity categories.

[0067] It should be noted that, based on the description text obtained in step 101, keyword extraction is performed on the description text, and keywords related to entity objects, such as pedestrians, motor vehicles, and non-motor vehicles, are selected to obtain a set of entity keywords, which can be denoted as... 'm' represents the number of keywords for the object, and then the keywords are... As a textual prompt for the Grounding DINO model, the set of key entity categories for the image dataset D is obtained by counting the number of key object detection categories in all images, which can be denoted as: Finally, based on the set of key entity categories, the key entity score S3 is determined, which can serve as the third basis for evaluating the value of image data elements.

[0068] More specifically, the calculation method for the key entity scores mentioned above can be seen in the following example:

[0069] The key entity category richness is obtained by averaging the key entity category set and the number of key entity categories. Then, the key entity score is determined by the ratio of the key entity category richness to the number of keywords in the entity keyword set.

[0070] The key entity category richness, representing the average number of key object categories in the dataset, is calculated by averaging the set of key entity categories and the number of key entity categories. This richness is denoted as . Its calculation formula is , where n3 is the number of elements in the key entity category set.

[0071] Finally, based on the ratio of the richness of key entity categories to the number of keywords in the entity keyword set, the key entity score S3 can be obtained, and its calculation formula can be expressed as: .

[0072] Step 106: Determine the comprehensive score based on the similarity score, entity category richness score, and key entity score, and use the comprehensive score to determine the element value assessment result of the image data.

[0073] Finally, calculate the overall score. Here, a, b, and c are the weighting coefficients of the three scores, with values ​​ranging from [0,1], which can be dynamically adjusted according to the actual situation; if a, b, and c are all 1, then the total score ranges from [-100,300]. The higher the score, the more the image dataset matches the description, and the higher the value.

[0074] In addition to the above-mentioned method of calculating the weighted sum as the comprehensive score, some embodiments may also use ordinary summation, ordinary product or weighted product as the method of calculating the comprehensive score, which will not be elaborated here.

[0075] The above is a detailed description of an embodiment of an image data element value assessment method provided by this application. The following is a detailed description of an embodiment of an image data element value assessment device provided by this application.

[0076] Please see Figure 2 This application provides an embodiment of an image data element value assessment device, comprising:

[0077] The data acquisition unit 201 is used to acquire image data and the corresponding descriptive text of the image data;

[0078] The feature vector extraction unit 202 is used to extract features from image data and descriptive text respectively through a preset feature extraction model to obtain image feature vectors and text feature vectors;

[0079] The element similarity scoring unit 203 is used to calculate the feature vector similarity between the image feature vector and the text feature vector, so as to determine the similarity score based on the feature vector similarity.

[0080] The entity element richness scoring unit 204 is used to identify entity objects contained in image data through a preset visual recognition model, and determine the entity category richness score based on the number of entity object categories.

[0081] The key entity element scoring unit 205 is used to extract keywords from the descriptive text to obtain a set of entity keywords, match the entity keyword set with the entity object to obtain a set of key entity categories for the image data, and determine the key entity score based on the set of key entity categories.

[0082] The element value determination unit 206 is used to determine the comprehensive score based on the similarity score, entity category richness score, and key entity score, and to determine the element value assessment result of the image data based on the comprehensive score.

[0083] In addition to the image data element value assessment device embodiments provided above, this application provides a detailed description of an image data element value assessment terminal embodiment and a computer-readable storage medium embodiment.

[0084] Please see Figure 3 The third aspect of this application provides an image data element value assessment terminal. The terminal implementation types include: personal computer, industrial computer, server and embedded intelligent device. The main components of the terminal include: memory 33 and processor 31, wherein memory 33 and processor 31 can be connected through communication bus 34.

[0085] The memory is used to store program code, which corresponds to an image data element value assessment method provided in the above embodiments.

[0086] The processor is used to read and execute program code.

[0087] This application provides a computer-readable storage medium that stores program code. The program code is read and executed by a processor to implement an image data element value assessment method as provided in the above embodiments.

[0088] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the terminals, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0089] In the several embodiments provided in this application, it should be understood that the disclosed terminals, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0090] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0091] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0092] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0093] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0094] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0095] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An image data element value assessment method, characterized by, The method comprises: acquiring image data and description text corresponding to the image data; extracting features of the image data and the description text respectively by using a preset feature extraction model to obtain an image feature vector and a text feature vector; calculating a feature vector similarity between the image feature vector and the text feature vector to determine a similarity score according to the feature vector similarity; identifying an entity object contained in the image data by using a preset visual recognition model, and determining an entity category richness score according to a category number of the entity object; extracting keywords from the description text to obtain an entity keyword set, matching the entity keyword set with the entity object to obtain a key entity category set of the image data, and determining a key entity score according to the key entity category set; determining a comprehensive score according to the similarity score, the entity category richness score and the key entity score, and determining an element value evaluation result of the image data according to the comprehensive score.

2. The method of claim 1, wherein, When the image data is an image data set containing multiple images, the feature extraction of the image data by using the preset feature extraction model comprises: extracting features of each image data in the image data set by using the preset feature extraction model to obtain a feature vector corresponding to each image data; performing mean value calculation on each feature vector to obtain an average feature vector, and taking the average feature vector as the image feature vector.

3. The method of claim 1, wherein, The preset feature extraction model is a Chinese-CLIP visual large model.

4. The method of claim 1, wherein, The calculation of the feature vector similarity between the image feature vector and the text feature vector comprises: calculating the feature vector similarity between the image feature vector and the text feature vector by using a cosine similarity calculation method.

5. The method of claim 1, wherein, The identification of the entity object contained in the image data by using the preset visual recognition model and the determination of the entity category richness score according to the category number of the entity object comprise: identifying the entity object contained in the image data by using a preset RAM visual recognition model, classifying the entity object according to the category of the entity object to obtain an entity object category set; performing mean value calculation on the entity object category set and the entity object category number to obtain entity category richness; determining the entity category richness score according to the entity category richness and a preset correlation between the entity category richness and the entity category richness score.

6. The method of claim 1, wherein, The determination of the key entity score according to the key entity category set comprises: performing mean value calculation on the key entity category set and the key entity category number to obtain key entity category richness, and determining the key entity score according to a ratio of the key entity category richness to a keyword number in the entity keyword set.

7. The method of claim 1, wherein, The determination of the comprehensive score according to the similarity score, the entity category richness score and the key entity score comprises: A weighted sum value of the similarity score, the entity category richness score and the key entity score is calculated, and the weighted sum value is taken as a comprehensive score.

8. An image data element value evaluation apparatus characterized by comprising: The method comprises the following steps: A data acquisition unit is configured to acquire image data and description text corresponding to the image data; A feature vector extraction unit is configured to extract features of the image data and the description text respectively by using a preset feature extraction model to obtain image feature vectors and text feature vectors; An element similarity scoring unit is configured to calculate a feature vector similarity between the image feature vectors and the text feature vectors, and determine a similarity score according to the feature vector similarity; An entity element richness scoring unit is configured to identify entity objects contained in the image data by using a preset visual recognition model, and determine an entity category richness score according to a number of categories of the entity objects; A key entity element scoring unit is configured to extract entity keywords from the description text to obtain a set of entity keywords, match the set of entity keywords with the entity objects to obtain a set of key entity categories of the image data, and determine a key entity score according to the set of key entity categories; An element value result determination unit is configured to determine a comprehensive score according to the similarity score, the entity category richness score and the key entity score, and determine an element value evaluation result of the image data according to the comprehensive score.

9. An image data element value evaluation terminal, characterized by, The method comprises the following steps: A memory and a processor are provided; The memory is configured to store program code corresponding to the image data element value evaluation method according to any one of claims 1 to 7; The processor is configured to read and execute the program code.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores program code, which is configured to be read and executed by the processor to implement the image data element value evaluation method according to any one of claims 1 to 7.

Images