A target detection algorithm evaluation method, device, equipment and storage medium

Abstract

The application discloses a target detection algorithm evaluation method and device, equipment and a storage medium, and relates to the technical field of computers. The method comprises the following steps: constructing an evaluation data set composed of detection index items and pre-constructed description keyword items; performing synonym merging on all the description keyword items in the evaluation data set; reconstructing a new evaluation data set based on the encoded keyword items obtained by encoding the merged keyword items and the encoded index items obtained by encoding the discretized detection index items; reconstructing a corresponding target keyword item set based on the currently determined target detection index item and the effective keyword items in the target evaluation data set obtained by eliminating the description keyword items that are not sensitive to the detection index items in the new evaluation data set, and evaluating the algorithm in the target detection scene corresponding to the target keyword item set, so that the false detection rate and the detection rate of the algorithm in the subsequent use scene can be improved in a targeted manner, and blind optimization is avoided.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, device, and storage medium for evaluating target detection algorithms. Background Technology

[0002] Currently, object detection is often the first step in many intelligent algorithms. Testing the accuracy and recall of detection algorithms has a significant impact on subsequent tasks. However, most mainstream methods for testing detection algorithms focus on classifying detected targets, lacking testing and evaluation methods specific to intelligent tasks and actual deployment environments. This leads to a significant discrepancy between test results and actual performance in real-world scenarios, resulting in distorted algorithm metrics. Furthermore, neglecting specific deployment environment scenarios increases the difficulty of refining algorithm metrics within those scenarios, often requiring substantial supplementation and improvement of training and testing data, incurring significant costs. In other words, current implementations of detection algorithms suffer from a considerable mismatch between algorithm metrics and actual needs, urgently requiring an effective algorithm evaluation method to accurately assess the accuracy of detection algorithms in real-world scenarios and guide subsequent training optimization. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide a method, apparatus, device, and storage medium for evaluating target detection algorithms, which can accurately evaluate the detection accuracy of target detection algorithms in different scenarios. This facilitates targeted improvements to the false detection rate and detection rate of the target detection algorithm in subsequent application scenarios, thereby avoiding blind optimization. The specific solution is as follows:

[0004] Firstly, this application discloses a method for evaluating object detection algorithms, including:

[0005] Construct an evaluation dataset; the evaluation data items in the evaluation dataset consist of detection index items and pre-constructed descriptive keyword items;

[0006] Synonyms are merged for all the descriptive keyword items in the evaluation dataset, and the merged keyword items are encoded to obtain encoded keyword items;

[0007] The detection index items in the evaluation dataset are discretized, and the discretized detection index items are encoded to obtain the encoded index items;

[0008] A new evaluation dataset is reconstructed based on the encoded keyword items and encoded indicator items, and the descriptive keyword items that are not sensitive to the detection indicator items are removed from the new evaluation dataset to obtain the target evaluation dataset.

[0009] Based on the currently determined target detection indicators and the effective keywords in the target evaluation dataset, a corresponding target keyword set is reconstructed, and the target detection algorithm is analyzed and evaluated in the target detection scenario corresponding to the target keyword set to obtain the corresponding evaluation results.

[0010] Optionally, before constructing the evaluation dataset, the following steps are also included:

[0011] Construct descriptive keyword items that describe categories, scenarios, and the mutual states between the categories and scenarios; wherein, the descriptive keyword items consist of structured primary keywords and unstructured secondary keywords.

[0012] Optionally, the step of merging all the descriptive keyword items in the evaluation dataset using synonyms and encoding the merged keyword items to obtain encoded keyword items includes:

[0013] All secondary keywords extracted from the evaluation dataset are deduplicated to obtain a set of deduplicated secondary keywords. Then, a word vector file is used to encode all the deduplicated secondary keywords in the set of deduplicated secondary keywords to obtain a corresponding set of secondary keyword vectors.

[0014] Calculate the cosine similarity between the subkeyword vectors in the subkeyword vector set;

[0015] Based on the cosine similarity, the deduplicated secondary keywords in the deduplicated secondary keyword set are merged, and the secondary keywords in the merged secondary keyword set are encoded using the OneHot method to obtain the encoded secondary keyword set.

[0016] Extract the main keywords from the descriptive keyword items in the evaluation dataset to obtain a set of main keywords, and merge the main keywords in the set of main keywords to obtain a merged set of main keywords;

[0017] The main keywords in the merged main keyword set are encoded using a discretization encoding method to obtain the encoded main keyword set.

[0018] Optionally, merging the deduplicated secondary keywords in the deduplicated secondary keyword set based on the cosine similarity includes:

[0019] Determine whether the cosine similarity is not less than a first preset threshold;

[0020] If the cosine similarity is not less than the first preset threshold, then the deduplicated secondary keywords in the deduplicated secondary keyword set are merged.

[0021] Optionally, the step of removing descriptive keywords that are insensitive to the detection metrics from the new evaluation dataset to obtain the target evaluation dataset includes:

[0022] Calculate the information gain of all the encoded keyword items in the new evaluation dataset to the detection index item to obtain the gain value corresponding to each encoded keyword item;

[0023] Determine whether the gain value corresponding to the encoded keyword is not less than a preset threshold;

[0024] If the gain value corresponding to the encoded keyword item is not less than the preset threshold, then the encoded keyword item is retained;

[0025] If the gain value corresponding to the encoded keyword is less than the preset threshold, the encoded keyword is removed from the new evaluation dataset to obtain the target evaluation dataset.

[0026] Optionally, the step of reconstructing the corresponding target keyword set based on the currently determined target detection index items and the effective keyword items in the target evaluation dataset includes:

[0027] Determine whether the set of main keywords corresponding to the valid keyword items in the target evaluation dataset is an empty set;

[0028] If the set of main keywords corresponding to the effective keyword items in the target evaluation dataset is an empty set, then the target main keyword with the highest conditional probability is queried based on the secondary keywords in the effective keyword items, and the corresponding target keyword item set is reconstructed based on the predetermined set of target secondary keywords and the target main keyword set composed of the target main keywords.

[0029] If the set of primary keywords corresponding to the effective keyword items in the target evaluation dataset is not an empty set, then the target primary keyword with the highest conditional probability in the target evaluation dataset is determined, and the corresponding target keyword item set is reconstructed based on the pre-determined set of target secondary keywords and the target primary keyword set composed of the target primary keyword.

[0030] Optionally, after reconstructing the corresponding target keyword set based on the currently determined target detection index items and the effective keyword items in the target evaluation dataset, the method further includes:

[0031] Calculate the scene adaptability of all secondary keywords in the target keyword set;

[0032] The secondary keywords in the target keyword set are sorted according to the scene adaptability.

[0033] Secondly, this application discloses a target detection algorithm evaluation device, comprising:

[0034] The dataset construction module is used to construct the evaluation dataset; the evaluation data items in the evaluation dataset consist of detection index items and pre-constructed descriptive keyword items;

[0035] The keyword merging module is used to merge synonyms for all the descriptive keyword items in the evaluation dataset.

[0036] The keyword encoding module is used to encode the merged keyword items to obtain the encoded keyword items;

[0037] The indicator discretization module is used to discretize the detection indicator items in the evaluation dataset;

[0038] The index encoding module is used to encode the discretized detection index items to obtain the encoded index items;

[0039] The dataset reconstruction module is used to reconstruct a new evaluation dataset based on the encoded keyword items and encoded indicator items;

[0040] The keyword removal module is used to remove descriptive keywords that are insensitive to the detection index from the new evaluation dataset to obtain the target evaluation dataset;

[0041] The keyword reconstruction module is used to reconstruct the corresponding target keyword set based on the currently determined target detection index items and the effective keyword items in the target evaluation dataset;

[0042] The detection algorithm evaluation module is used to analyze and evaluate the target detection algorithm in the target detection scenario corresponding to the target keyword set to obtain the corresponding evaluation results.

[0043] Thirdly, this application discloses an electronic device, including:

[0044] Memory, used to store computer programs;

[0045] A processor is configured to execute the computer program to implement the steps of the aforementioned disclosed object detection algorithm evaluation method.

[0046] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the aforementioned disclosed target detection algorithm evaluation method.

[0047] As can be seen, this application provides a method for evaluating object detection algorithms, comprising: constructing an evaluation dataset; the evaluation data items in the evaluation dataset consist of detection index items and pre-constructed descriptive keyword items; merging all the descriptive keyword items in the evaluation dataset using synonyms, and encoding the merged keyword items to obtain encoded keyword items; discretizing the detection index items in the evaluation dataset, and encoding the discretized detection index items to obtain encoded index items; reconstructing a new evaluation dataset based on the encoded keyword items and encoded index items, and removing descriptive keyword items insensitive to the detection index items from the new evaluation dataset to obtain a target evaluation dataset; reconstructing a corresponding target keyword item set based on the currently determined target detection index items and the effective keyword items in the target evaluation dataset, and analyzing and evaluating the object detection algorithm in the target detection scenario corresponding to the target keyword item set to obtain the corresponding evaluation results. Therefore, this application constructs an evaluation dataset by fully describing the test data, merges synonyms in the keyword items of the evaluation dataset, encodes the merged keyword items and the detection index items in the evaluation dataset, and then reconstructs a new evaluation dataset. The descriptive keyword items that are not sensitive to the detection index items in the new evaluation dataset are removed to obtain the target evaluation dataset. Then, a target keyword item set is reconstructed based on the target evaluation dataset, and the target detection algorithm is evaluated based on the target keyword item set. This facilitates the subsequent targeted improvement of the false detection rate and detection rate of the target detection algorithm in the application scenario, thereby avoiding blind optimization. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0049] Figure 1 This is a flowchart of an object detection algorithm evaluation method disclosed in this application;

[0050] Figure 2 This is a schematic diagram of the keyword composition structure in test data disclosed in this application;

[0051] Figure 3 This is a flowchart of a specific object detection algorithm evaluation method disclosed in this application;

[0052] Figure 4 This is a schematic diagram of the structure of an object detection algorithm evaluation device disclosed in this application;

[0053] Figure 5 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] Currently, object detection is often the first step in many intelligent algorithms. The accuracy and recall of these algorithms have a significant impact on subsequent tasks. However, most mainstream methods for testing detection algorithms focus on classifying the detected targets, lacking specific testing and evaluation methods for particular intelligent tasks and actual deployment environments. This leads to a significant discrepancy between the test results and the actual performance in real-world scenarios, resulting in distorted algorithm metrics. Furthermore, neglecting specific deployment environment scenarios increases the difficulty of refining algorithm metrics within those scenarios, often requiring substantial supplementation and improvement of training and testing data, incurring significant costs. In other words, current implementations of detection algorithms suffer from a considerable mismatch between algorithm metrics and actual needs. Therefore, this application provides an object detection algorithm evaluation scheme that facilitates targeted improvements to the false detection rate and detection rate in specific application scenarios, thereby avoiding blind optimization.

[0056] This invention discloses a method for evaluating target detection algorithms, see [link to relevant documentation]. Figure 1 As shown, the method includes:

[0057] Step S11: Construct an evaluation dataset; the evaluation data items in the evaluation dataset consist of detection index items and pre-constructed descriptive keyword items.

[0058] In this embodiment, the test data is first fully described to construct an evaluation dataset. This evaluation dataset consists of evaluation data items, and each evaluation data item in the dataset comprises detection index items and pre-constructed descriptive keyword items. It is understood that each element in the evaluation dataset consists of detection index items and descriptive keyword items. For example, the evaluation dataset can be described as follows:

[0059] D=((kw1,in1),(kw2,in2),...,(kw m ,in m ));

[0060] Here, kw represents the keyword item, and in represents the two-dimensional indicator item.

[0061] In this embodiment, for single-target detection, the detection metrics are whether the image and video to be detected are detected and whether there are false detections. For multi-target detection, the metrics are precision and recall. Furthermore, for the descriptive keywords, descriptive keyword items describing categories, scenes, and the mutual states between the categories and scenes are constructed. Each descriptive keyword item consists of structured main keywords and unstructured sub-keywords. In other words, each descriptive keyword item has two parts: the first part is the structured main keyword, selected from a defined keyword set, primarily representing the category and the scene in which the category resides. Alternatively, it can vary according to the actual needs of the target detection algorithm. The more main keywords there are, the more categories there are, and the higher the level of detail in the corresponding scenes, thus requiring greater data balance and completeness. The second part is the unstructured sub-keyword, which describes key points in the scene. The main function of the sub-keyword is to enrich the category and scene information. It is determined from multiple dimensions, such as category state and scene characteristics, based on the applicable scenes for the target detection algorithm. It should be noted that each keyword describes different dimensions and is therefore independent, allowing them to coexist. For example, in an electric bicycle detection algorithm, different targets, including false positives, are distinguished. Categories could include electric bicycles, bicycles, motorcycles, etc., and scenarios could include corridors, plazas, elevator interiors, etc. A complete description could be: electric bicycle - elevator - riding in, entering forward, with a windshield, black body, with headlights, elevator reflection, low light, etc. Figure 2 As shown, in the keyword composition structure of the test data description, the keywords "Type" and "Scenes" represent the structured description of the detection category and detection scene, i.e., the structured main keywords. The terms within the dashed boxes are the options for these main keywords. Data balancing can be performed based on the number of terms and options in the structured main keywords to ensure that the number of data for each term and option is basically consistent. "Key" represents descriptive keywords for the category and scene, i.e., unstructured secondary keywords. In other words, this embodiment, from the perspective of practical application scenarios, constructs a semi-structured evaluation dataset to unify the target factors that are usually ignored by object detection algorithms. It uses text mining methods to encode and evaluate many effective environmental features that are observable to the naked eye, and extracts keywords for the worst or best scenes under determined indicators. This improves the effectiveness of algorithm testing, effectively supplementing and refining the material scenes required for object detection algorithms, and accelerating the algorithm's practical application.

[0062] Step S12: Perform synonym merging on all the descriptive keyword items in the evaluation dataset, and encode the merged keyword items to obtain the encoded keyword items.

[0063] In this embodiment, due to individual cognitive differences, there may be many synonyms in the descriptive keyword items in the constructed evaluation dataset, which is not conducive to subsequent encoding and calculation. Therefore, synonyms are merged in all the descriptive keyword items in the evaluation dataset, and the merged keyword items are encoded to obtain encoded keyword items. Specifically, all the secondary keywords in the descriptive keyword items extracted from the evaluation dataset are deduplicated to obtain a deduplicated secondary keyword set, and a word vector file is used to encode all the deduplicated secondary keywords in the deduplicated secondary keyword set to obtain a corresponding secondary keyword vector set; the calculation is performed... The cosine similarity between the subkeyword vectors in the subkeyword vector set is calculated; based on the cosine similarity, the deduplicated subkeywords in the deduplicated subkeyword set are merged, and the subkeywords in the merged subkeyword set are encoded using the OneHot method to obtain an encoded subkeyword set; the main keywords in the descriptive keyword items are extracted from the evaluation dataset to obtain a main keyword set, and the main keywords in the main keyword set are merged to obtain a merged main keyword set; the main keywords in the merged main keyword set are encoded using a discretization encoding method to obtain an encoded main keyword set. In other words, for the secondary keywords in the descriptive keyword items of the evaluation dataset, all secondary keywords extracted from the evaluation dataset are deduplicated to form a deduplicated secondary keyword set. Then, a secondary keyword vector set is formed by encoding all deduplicated secondary keywords using word vectors. The cosine similarity between all secondary keyword vectors is calculated, and it is determined whether the cosine similarity is not less than a first preset threshold. If the cosine similarity is not less than the first preset threshold, the deduplicated secondary keywords in the deduplicated secondary keyword set are merged. If the cosine similarity is less than the first preset threshold, merging the deduplicated secondary keywords in the deduplicated secondary keyword set is prohibited. For example, keywords with a similarity greater than 0.90 are merged, and the merged secondary keyword set has P items. Then, the OneHot method is used to encode the secondary keywords in the merged secondary keyword set. For the primary keywords in the descriptive keyword items of the evaluation dataset, multiple primary keywords with synonyms are merged, and the primary keywords in the merged primary keyword set are encoded using a discretization encoding method. It should be noted that the word vector files mentioned above can be the open-source thesaurus tencent-ailab-embedding-zh, or other open-source thesauruses can be selected for word vector encoding. Similarly, a more suitable thesaurus can be obtained by training a targeted network according to actual needs.

[0064] Step S13: Discretize the detection index items in the evaluation dataset, and encode the discretized detection index items to obtain the encoded index items.

[0065] In this embodiment, the detection metrics in the evaluation dataset are discretized, and the discretized metrics are encoded to obtain the encoded metrics. It is understood that for single-target detection, there can be two detection metrics: whether a target was detected and whether a false positive was detected. The values ​​for both can be 0 or 1, representing discretized data. For multi-target detection, precision and recall are used as detection metrics. Precision and recall are not discretized data; they are discretized. For example, when precision exceeds 90%, the value is 1; otherwise, the value is 0. When recall exceeds 50%, the value is 1; otherwise, the value is 0.

[0066] Step S14: Reconstruct a new evaluation dataset based on the encoded keyword items and encoded indicator items, and remove the descriptive keyword items in the new evaluation dataset that are not sensitive to the detection indicator items to obtain the target evaluation dataset.

[0067] In this embodiment, an evaluation dataset is constructed, and the keywords in the evaluation dataset are merged using synonyms. The merged keywords are then encoded. Additionally, the detection metrics in the evaluation dataset are discretized, and the discretized detection metrics are encoded to obtain a new evaluation dataset. For example, the new evaluation dataset can be described as follows:

[0068] D={(kw_en1, in_en1), (kw_en2, in_en2),..., (kw_en m ,in_en m )};

[0069] Wherein, kw_en is the encoded P+1 dimensional keyword vector, and in_en is the discretized two-dimensional index vector.

[0070] In this embodiment, after reconstructing a new evaluation dataset based on the encoded keywords and encoded indicators, since the selected secondary keywords are unstructured data, it is necessary to remove descriptive keywords in the new evaluation dataset that are insensitive to the detection indicators to obtain the target evaluation dataset. Specifically, the information gain of all encoded keywords in the new evaluation dataset to the detection indicators is calculated to obtain the gain value corresponding to each encoded keyword; it is then determined whether the gain value corresponding to the encoded keyword is not less than a preset threshold; if the gain value corresponding to the encoded keyword is not less than the preset threshold, the encoded keyword is retained; if the gain value corresponding to the encoded keyword is less than the preset threshold, the encoded keyword is removed from the new evaluation dataset to obtain the target evaluation dataset. For example, the information gain of all encoded keywords in the new evaluation dataset to the detection indicators is calculated using the following formula:

[0071]

[0072] Where k represents the keyword, p i The I index represents the proportion of samples occupied under the i∈I index state, where I is 3 for single-target detection and 4 for multi-target detection; j indicates whether the encoded keyword exists, 0 for yes / no and 1 for yes; N represents the number of data points. j This indicates the number of data points containing the keyword. This indicates the proportion of states i under this keyword.

[0073] If the information gain value of a keyword is greater than 1 / P+1, then the keyword is retained; if the information gain value of a keyword is not greater than 1 / P+1, then the keyword is discarded. The set of retained keywords can be described as follows:

[0074] K′={K′ m , K′ s};

[0075] This keyword set consists of a main keyword set and a secondary keyword set, K′ m Let K' represent the set of primary keywords. s This represents a set of secondary keywords.

[0076] Step S15: Based on the currently determined target detection index items and the effective keyword items in the target evaluation dataset, reconstruct the corresponding target keyword item set, and analyze and evaluate the target detection algorithm under the target detection scenario corresponding to the target keyword item set to obtain the corresponding evaluation results.

[0077] In this embodiment, after removing keywords that are insensitive to detection metrics from the new evaluation dataset to obtain the target evaluation dataset, all keywords in the target evaluation dataset are valid keywords. Then, based on the currently determined target detection metrics and the valid keywords in the target evaluation dataset, the corresponding target keyword set is reconstructed. It can be understood that the target keyword set is reconstructed according to specific metric preferences. For example, for single-target detection, the two-dimensional metric of whether a detection is made or not, and whether a false detection is made, has three states: (0,0), (1,0), and (1,1), where (0,0) represents no detection, (1,0) represents a false detection, and (1,1) represents a positive detection. For multi-target detection, the two-dimensional metric of precision and recall has four states: (0,0), (0,1), (1,0), and (1,1), where (0,0) represents low detection and low precision, (0,1) represents low detection and high precision, (1,0) represents high detection and low precision, and (1,1) represents high detection and high precision. In single-object detection, to understand the conditions under which false positives occur most frequently, (1, 0) can be selected as the object detection state, and the target keyword set can be reconstructed under this state. This target keyword set consists of two main keywords and several sub-keywords. The influence of each keyword on the object detection metric is determined by assessing its conditional probability. Specifically, it is determined whether the set of main keywords corresponding to the valid keywords in the object evaluation dataset is empty. If the set of main keywords corresponding to the valid keywords in the object evaluation dataset is empty, the target main keyword with the highest conditional probability is queried based on the sub-keywords in the valid keywords, and the corresponding target keyword set is reconstructed based on a pre-determined set of target sub-keywords and the target main keyword set composed of the target main keyword. If the set of main keywords corresponding to the valid keywords in the object evaluation dataset is not empty, the target main keyword with the highest conditional probability in the object evaluation dataset is determined, and the corresponding target keyword set is reconstructed based on a pre-determined set of target sub-keywords and the target main keyword set composed of the target main keyword.

[0078] For example, if the set of primary keywords K′ m If the set is empty, then calculate the conditional probabilities of all satisfying subkeywords for the target detection index item in turn, i.e.:

[0079] Where, k∈K′ s .

[0080] Then, using the known secondary keywords, find the target primary keyword with the highest conditional probability:

[0081]

[0082] Stk∈K′ m ;

[0083] Among them, K″ s This represents the set of target secondary keywords with a conditional probability greater than 1. The set of target main keywords (composed of the target primary keywords) and the set of target secondary keywords together form the set of target keyword items {k} under this target detection metric. m , K″ s}

[0084] If the set of primary keywords is K′ m If the set is not empty, then the target primary keyword is obtained directly by obtaining the primary keyword with the highest probability.

[0085]

[0086] k∈K′ m

[0087] st

[0088] Then, determine the set of target subkeywords K″ with a conditional probability greater than 1. s The target keyword set {k} is composed of the set of target main keywords and the set of target secondary keywords obtained from the target main keywords. m , K″ s}

[0089] In this embodiment, the target keyword set is reconstructed for specific indicator preferences, and then the target detection algorithm is analyzed and evaluated in the target detection scenario corresponding to the target keyword set to determine the actual performance of the target detection algorithm in the target detection scenario.

[0090] As can be seen, in this embodiment of the application, by fully describing the test data to construct an evaluation dataset, the keywords in the evaluation dataset are merged using synonyms, and the merged keywords and the detection index items in the evaluation dataset are encoded. Then, a new evaluation dataset is reconstructed, and the descriptive keywords that are not sensitive to the detection index items in the new evaluation dataset are removed to obtain the target evaluation dataset. Then, a target keyword set is reconstructed based on the target evaluation dataset, and the target detection algorithm is evaluated based on the target keyword set. This makes it easier for the target detection algorithm to make targeted improvements to the false detection rate and detection rate in the application scenario, thereby avoiding blind optimization.

[0091] See Figure 3As shown, this embodiment of the invention discloses a specific method for evaluating target detection algorithms. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution.

[0092] Step S21: Construct an evaluation dataset; the evaluation data items in the evaluation dataset consist of detection index items and pre-constructed descriptive keyword items.

[0093] Step S22: Synonymize all the descriptive keyword items in the evaluation dataset, and encode the merged keyword items to obtain encoded keyword items.

[0094] Step S23: Discretize the detection index items in the evaluation dataset, and encode the discretized detection index items to obtain the encoded index items.

[0095] Step S24: Reconstruct a new evaluation dataset based on the encoded keyword items and encoded indicator items, and remove the descriptive keyword items in the new evaluation dataset that are not sensitive to the detection indicator items to obtain the target evaluation dataset.

[0096] Step S25: Reconstruct the corresponding target keyword set based on the currently determined target detection index items and the effective keyword items in the target evaluation dataset.

[0097] Step S26: Calculate the scene adaptability of all secondary keywords in the target keyword set.

[0098] In this embodiment, since different secondary keywords have varying degrees of influence on the metric, their scene adaptability under that metric also differs. Therefore, the scene adaptability of all secondary keywords in the target secondary keyword set corresponding to the target keyword set can be calculated. For example, under the target detection metric, the formula for calculating the scene adaptability of each secondary keyword can be:

[0099]

[0100] Among them, c i In the sample set D m In the calculation of the target subkeyword set K″ s The probability of all secondary keywords appearing, D m Represents the target main keyword set k′ m The sample set, and the probability c i Represented as:

[0101] Step S27: Sort the secondary keywords in the target keyword set according to the scene adaptability.

[0102] In this embodiment, after calculating the scene adaptability of all secondary keywords in the target keyword set, the secondary keywords in the target keyword set are sorted according to the scene adaptability to obtain the sorted target keyword set. For example, by arranging all secondary keywords from high to low according to scene adaptability, secondary keywords that have a significant impact on the corresponding detection indicators can be quickly captured. If the setting is unordered, excessive attention may be paid to secondary keywords with little impact on the indicators, thus ignoring the main components.

[0103] Step S28: Analyze and evaluate the target detection algorithm in the target detection scenario corresponding to the sorted target keyword set to obtain the corresponding evaluation results.

[0104] For details regarding steps S21 to S25 and S28, please refer to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.

[0105] As can be seen, in this embodiment of the application, by fully describing the test data to construct an evaluation dataset, merging the synonyms of the keyword items in the evaluation dataset, encoding the merged keyword items and the detection index items in the evaluation dataset, and then reconstructing a new evaluation dataset, removing the descriptive keyword items in the new evaluation dataset that are not sensitive to the detection index items to obtain the target evaluation dataset, and then reconstructing the target keyword item set based on the target evaluation dataset, and sorting the secondary keywords in the target keyword item set, the target detection algorithm is evaluated based on the sorted target keyword item set. This facilitates the target detection algorithm to make targeted improvements to the false detection rate and detection rate in the application scenario, thereby avoiding blind optimization.

[0106] Accordingly, embodiments of this application also disclose an object detection algorithm evaluation device, see [link to relevant documentation]. Figure 4 As shown, the device includes:

[0107] The dataset construction module 11 is used to construct the evaluation dataset; the evaluation data items in the evaluation dataset consist of detection index items and pre-constructed descriptive keyword items;

[0108] Keyword merging module 12 is used to merge synonyms for all the descriptive keyword items in the evaluation dataset;

[0109] Keyword encoding module 13 is used to encode the merged keyword items to obtain the encoded keyword items;

[0110] The indicator discretization module 14 is used to discretize the detection indicator items in the evaluation dataset;

[0111] The index encoding module 15 is used to encode the discretized detection index items to obtain the encoded index items;

[0112] Dataset reconstruction module 16 is used to reconstruct a new evaluation dataset based on the encoded keyword items and encoded indicator items;

[0113] Keyword removal module 17 is used to remove descriptive keywords that are insensitive to the detection index from the new evaluation dataset to obtain the target evaluation dataset;

[0114] Keyword reconstruction module 18 is used to reconstruct the corresponding target keyword item set based on the currently determined target detection index items and the effective keyword items in the target evaluation dataset;

[0115] The detection algorithm evaluation module 19 is used to analyze and evaluate the target detection algorithm in the target detection scenario corresponding to the target keyword set to obtain the corresponding evaluation results.

[0116] As can be seen from the above, in this embodiment of the application, the test data is fully described to construct an evaluation dataset, the keywords in the evaluation dataset are merged using synonyms, and the merged keywords and the detection index items in the evaluation dataset are encoded. Then, a new evaluation dataset is reconstructed, and the descriptive keywords that are not sensitive to the detection index items in the new evaluation dataset are removed to obtain the target evaluation dataset. Then, a target keyword set is reconstructed based on the target evaluation dataset, and the target detection algorithm is evaluated based on the target keyword set. This makes it easier for the target detection algorithm to make targeted improvements to the false detection rate and detection rate in the application scenario, thereby avoiding blind optimization.

[0117] In some specific embodiments, the keyword merging module 12 may specifically include:

[0118] The keyword deduplication module is used to deduplicat all the secondary keywords in the descriptive keyword items extracted from the evaluation dataset to obtain a deduplicated set of secondary keywords;

[0119] The word vector encoding module is used to encode all the deduplicated subkeywords in the deduplicated subkeyword set using the word vector file to obtain the corresponding subkeyword vector set;

[0120] The cosine similarity calculation module is used to calculate the cosine similarity between the subkeyword vectors in the subkeyword vector set.

[0121] The first merging module is used to merge the deduplicated secondary keywords in the deduplicated secondary keyword set based on the cosine similarity.

[0122] The first encoding module is used to encode the subkeywords in the merged subkeyword set using the OneHot method to obtain the encoded subkeyword set.

[0123] The keyword extraction module is used to extract the main keywords from the descriptive keyword items in the evaluation dataset to obtain a set of main keywords;

[0124] The second merging module is used to merge the main keywords in the main keyword set to obtain a merged main keyword set;

[0125] The second encoding module is used to encode the main keywords in the merged main keyword set using a discretization encoding method to obtain the encoded main keyword set.

[0126] In some specific embodiments, the first merging module may specifically include:

[0127] The judgment unit is used to determine whether the cosine similarity is not less than a first preset threshold.

[0128] The merging unit is used to merge the deduplicated secondary keywords in the deduplicated secondary keyword set when the cosine similarity is not less than the first preset threshold.

[0129] In some specific embodiments, the keyword removal module 17 may specifically include:

[0130] The information gain calculation unit is used to calculate the information gain of all the encoded keyword items in the new evaluation dataset to the detection index item, and obtain the gain value corresponding to each encoded keyword item.

[0131] A gain value determination unit is used to determine whether the gain value corresponding to the encoded keyword item is not less than a preset threshold.

[0132] The keyword retention unit is used to retain the encoded keyword when the gain value corresponding to the encoded keyword is not less than the preset threshold.

[0133] The keyword removal unit is used to remove the encoded keyword from the new evaluation dataset to obtain the target evaluation dataset when the gain value corresponding to the encoded keyword is less than the preset threshold.

[0134] In some specific embodiments, the keyword reconstruction module 18 may specifically include:

[0135] The set judgment module is used to determine whether the set of main keywords corresponding to the valid keyword items in the target evaluation dataset is an empty set.

[0136] The target primary keyword query module is used to query the target primary keyword with the highest conditional probability based on the secondary keywords in the effective keyword items when the set of primary keywords corresponding to the effective keyword items in the target evaluation dataset is the empty set, and to reconstruct the corresponding target keyword item set based on the pre-determined set of target secondary keywords and the target primary keyword set composed of the target primary keywords.

[0137] The target primary keyword determination module is used to determine the target primary keyword with the highest conditional probability in the target evaluation dataset if the set of primary keywords corresponding to the effective keyword items in the target evaluation dataset is not an empty set, and to reconstruct the corresponding target keyword item set based on the pre-determined set of target secondary keywords and the target primary keyword set composed of the target primary keyword.

[0138] In some specific embodiments, the target detection algorithm evaluation device may further include:

[0139] The fitness calculation module is used to calculate the scene fitness of all secondary keywords in the target keyword set;

[0140] The secondary keyword sorting module is used to sort the secondary keywords in the target keyword set according to the scene adaptability.

[0141] Furthermore, embodiments of this application also provide an electronic device. Figure 5 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0142] Figure 5 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the target detection algorithm evaluation method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0143] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0144] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0145] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the target detection algorithm evaluation method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0146] Furthermore, embodiments of this application also disclose a computer-readable storage medium storing a computer program. When the computer program is loaded and executed by a processor, it implements the target detection algorithm evaluation method steps disclosed in any of the foregoing embodiments.

[0147] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0148] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0149] The above provides a detailed description of the target detection algorithm evaluation method, apparatus, device, and storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for evaluating target detection algorithms, characterized in that, include: Construct an evaluation dataset; the evaluation data items in the evaluation dataset consist of detection index items and pre-constructed descriptive keyword items; Synonyms are merged for all the descriptive keyword items in the evaluation dataset, and the merged keyword items are encoded to obtain encoded keyword items; The detection index items in the evaluation dataset are discretized, and the discretized detection index items are encoded to obtain the encoded index items; A new evaluation dataset is reconstructed based on the encoded keyword items and encoded indicator items, and the descriptive keyword items that are not sensitive to the detection indicator items are removed from the new evaluation dataset to obtain the target evaluation dataset. Based on the currently determined target detection index items and the effective keyword items in the target evaluation dataset, the corresponding target keyword item set is reconstructed, and the target detection algorithm is analyzed and evaluated in the target detection scenario corresponding to the target keyword item set to obtain the corresponding evaluation results; For single-target detection, the detection metrics include whether the image and video to be detected are detected and whether there are false detections; for multi-target detection, the detection metrics include precision and recall. Furthermore, the method further includes: constructing descriptive keyword items describing categories, scenarios, and the mutual state between the categories and scenarios; the descriptive keyword items include structured main keywords and unstructured sub-keywords, and determining the degree of influence of the corresponding descriptive keyword item on the target detection index item by determining the conditional probability of each of the descriptive keyword items; The step of reconstructing the corresponding target keyword set based on the currently determined target detection index items and the effective keyword items in the target evaluation dataset includes: If the set of primary keywords corresponding to the effective keyword items in the target evaluation dataset is empty, then the target primary keyword with the highest conditional probability is queried based on the secondary keywords in the effective keyword items, and the corresponding target keyword item set is reconstructed based on the predetermined set of target secondary keywords and the target primary keyword set composed of the target primary keywords. If the set of primary keywords is not the empty set, then the primary target keyword with the highest conditional probability in the target evaluation dataset is determined, and the corresponding set of target keyword items is reconstructed based on the pre-determined set of secondary target keywords and the set of primary target keywords composed of the primary target keywords.

2. The target detection algorithm evaluation method according to claim 1, characterized in that, The step of merging all the descriptive keyword items in the evaluation dataset using synonyms and encoding the merged keyword items to obtain encoded keyword items includes: All secondary keywords extracted from the evaluation dataset are deduplicated to obtain a set of deduplicated secondary keywords. Then, a word vector file is used to encode all the deduplicated secondary keywords in the set of deduplicated secondary keywords to obtain a corresponding set of secondary keyword vectors. Calculate the cosine similarity between the subkeyword vectors in the subkeyword vector set; Based on the cosine similarity, the deduplicated secondary keywords in the deduplicated secondary keyword set are merged, and the secondary keywords in the merged secondary keyword set are encoded using the OneHot method to obtain the encoded secondary keyword set. Extract the main keywords from the descriptive keyword items in the evaluation dataset to obtain a set of main keywords, and merge the main keywords in the set of main keywords to obtain a merged set of main keywords; The main keywords in the merged main keyword set are encoded using a discretization encoding method to obtain the encoded main keyword set.

3. The target detection algorithm evaluation method according to claim 2, characterized in that, The merging of the deduplicated secondary keywords in the deduplicated secondary keyword set based on the cosine similarity includes: Determine whether the cosine similarity is not less than a first preset threshold; If the cosine similarity is not less than the first preset threshold, then the deduplicated secondary keywords in the deduplicated secondary keyword set are merged.

4. The target detection algorithm evaluation method according to claim 2, characterized in that, The target evaluation dataset is obtained by removing descriptive keywords that are insensitive to the detection metrics from the new evaluation dataset, including: Calculate the information gain of all the encoded keyword items in the new evaluation dataset to the detection index item to obtain the gain value corresponding to each encoded keyword item; Determine whether the gain value corresponding to the encoded keyword is not less than a preset threshold; If the gain value corresponding to the encoded keyword item is not less than the preset threshold, then the encoded keyword item is retained; If the gain value corresponding to the encoded keyword is less than the preset threshold, the encoded keyword is removed from the new evaluation dataset to obtain the target evaluation dataset.

5. The target detection algorithm evaluation method according to claim 1, characterized in that, After reconstructing the corresponding target keyword set based on the currently determined target detection index items and the effective keyword items in the target evaluation dataset, the method further includes: Calculate the scene adaptability of all secondary keywords in the target keyword set; The secondary keywords in the target keyword set are sorted according to the scene adaptability.

6. A target detection algorithm evaluation device, characterized in that, include: The dataset construction module is used to construct the evaluation dataset; the evaluation data items in the evaluation dataset consist of detection index items and pre-constructed descriptive keyword items; The keyword merging module is used to merge synonyms for all the descriptive keyword items in the evaluation dataset. The keyword encoding module is used to encode the merged keyword items to obtain the encoded keyword items; The indicator discretization module is used to discretize the detection indicator items in the evaluation dataset; The index encoding module is used to encode the discretized detection index items to obtain the encoded index items; The dataset reconstruction module is used to reconstruct a new evaluation dataset based on the encoded keyword items and encoded indicator items; The keyword removal module is used to remove descriptive keywords that are insensitive to the detection index from the new evaluation dataset to obtain the target evaluation dataset; The keyword reconstruction module is used to reconstruct the corresponding target keyword set based on the currently determined target detection index items and the effective keyword items in the target evaluation dataset; The detection algorithm evaluation module is used to analyze and evaluate the target detection algorithm in the target detection scenario corresponding to the target keyword set to obtain the corresponding evaluation results. The target detection algorithm evaluation device is configured such that, for single-target detection, the detection metrics include whether the image and video to be detected are detected and whether there are false detections; for multi-target detection, the detection metrics include accuracy and recall. Furthermore, the target detection algorithm evaluation device is used to construct descriptive keyword items that describe the category, the scene, and the mutual state between the category and the scene; the descriptive keyword items include structured main keywords and unstructured sub-keywords, and the influence degree of the corresponding descriptive keyword item on the target detection index item is determined by determining the conditional probability of each of the descriptive keyword items; The keyword reconstruction module includes: The target main keyword query module is used to query the target main keyword with the highest conditional probability based on the secondary keywords in the effective keyword items if the set of main keywords corresponding to the effective keyword items in the target evaluation dataset is an empty set, and to reconstruct the corresponding target keyword item set based on the predetermined set of target secondary keywords and the target main keyword set composed of the target main keywords. The target primary keyword determination module is used to determine the target primary keyword with the highest conditional probability in the target evaluation dataset if the set of primary keywords is not the empty set, and to reconstruct the corresponding target keyword item set based on the pre-determined set of target secondary keywords and the target primary keyword set composed of the target primary keywords.

7. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the target detection algorithm evaluation method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, Used to store a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the target detection algorithm evaluation method as described in any one of claims 1 to 5.

Images