Test method and device of recognition model, electronic equipment and storage medium

Abstract

The application provides a test method of a recognition model, acquires a test data set and a to-be-recognized model, the test data set includes test pictures and label data of subject targets, and the test pictures include the subject targets; for each test picture in the test data set, the subject target in the test picture is processed by the to-be-recognized model to obtain a recognition result of the test picture; based on a first array and a second array, the recognition result of the test picture is judged, and the correct number and the error number of the recognition result are recorded, the first array is used for recording first judgment data, the first judgment data is used for judging whether the recognition result is correct, the second array is used for recording second judgment data, and the second judgment data is used for judging whether the recognition result is incorrect; after the judgment and the recording of all the test pictures in the test data set are performed, the final correct number and the final error number are obtained; and based on the final correct number and the final error number, a test result of the to-be-recognized model is calculated.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to a method, apparatus, electronic device, and storage medium for testing recognition models. Background Technology

[0002] Subject recognition models are input images and output the names of the main objects in those images. This has significant educational value for identifying household items, plants, animals, and other similar subjects.

[0003] The performance test of a subject recognition model refers to whether the name of the main object output by the model is correct and matches the name of the object in the input image, given a single input image. Recognition accuracy refers to the percentage of images from which the model correctly outputs the object name, out of the total number of input images. Existing testing methods include manual result analysis, where each test image is manually reviewed for accuracy, which is inefficient, especially with numerous model iterations. Another method involves script-based automatic result analysis, where each test image is pre-labeled with the object name, and the script checks if the recognition result matches the labeled name; otherwise, it's considered an error. However, this method has low reliability; for example, if "office desk" is recognized as "table," the script-based analysis method would classify it as an error.

[0004] Therefore, existing testing methods suffer from low testing efficiency and low reliability of test results. Summary of the Invention

[0005] This invention provides a testing method for a recognition model, aiming to offer an efficient and intelligent method to improve the reliability and efficiency of test results. For each test image in the test dataset, the model to be recognized identifies the main target within the image, obtaining the recognition result. Based on a first array and a second array, the recognition results are judged, and the number of correct and incorrect recognitions is recorded. After judging and recording all test images in the test dataset, the final number of correct and incorrect results is obtained. Using these final numbers, the test result of the model to be recognized is calculated, thereby improving the reliability and efficiency of the test results.

[0006] In a first aspect, embodiments of the present invention provide a testing method for a recognition model, the method comprising the following steps:

[0007] Obtain a test dataset and a model to be identified. The test dataset includes test images and labeled data of the main target. The test images include the main target.

[0008] For each test image in the test dataset, the main target in the test image is identified using the model to be identified, and the recognition result of the test image is obtained.

[0009] Based on the first array and the second array, the recognition results of the test image are judged, and the number of correct and incorrect recognition results are recorded. The first array is used to record the first discrimination data, which is used to judge whether the recognition result is correct. The second array is used to record the second discrimination data, which is used to judge whether the recognition result is incorrect.

[0010] After performing the discrimination and recording of all the test images in the test dataset, the final number of correct and incorrect results are obtained.

[0011] Based on the final number of correct answers and the number of incorrect answers, the test results of the model to be identified are calculated.

[0012] Optionally, before judging and recording the recognition results of the test image based on the first array and the second array, the method further includes:

[0013] Initialize the configuration of the first array, take all the labeled data in the test dataset as the initial first discrimination data, and add the recognition results with correct discrimination results as new first discrimination data to the first array;

[0014] The second array is initialized and configured. The second array is initially empty. During the discrimination process, the recognition result that is incorrect is added to the second array as new second discrimination data.

[0015] Optionally, the step of judging and recording the recognition results of the test image based on the first array and the second array includes:

[0016] For each test image, based on the recognition result and the first discrimination data, it is determined whether the discrimination result of the test image is correct;

[0017] If the judgment result of the test image is correct, then the number of correct results is incremented by 1.

[0018] If the discrimination result of the test image is unsuccessful, then based on the recognition result and the second discrimination data, it is determined whether the discrimination result of the test image is incorrect;

[0019] If the judgment result of the test image is incorrect, then the number of errors is incremented by 1.

[0020] Optionally, determining whether the discrimination result of the test image is correct based on the recognition result and the first discrimination data includes:

[0021] The recognition result is matched with the first discrimination data. If the recognition result is the same as the first discrimination data, the discrimination result of the test image is determined to be correct.

[0022] If the recognition result is different from the first discrimination data, then based on the first similarity between the recognition result and the first discrimination data, it is determined whether the discrimination result of the test image is correct;

[0023] If the first similarity is greater than or equal to the first similarity threshold, then the discrimination result of the test image is determined to be correct, and the recognition result is added to the first array as new first discrimination data;

[0024] If the similarity is less than the first similarity threshold, then the discrimination result of the test image is determined to be unsuccessful.

[0025] Optionally, determining whether the discrimination result of the test image is incorrect based on the recognition result and the second discrimination data includes:

[0026] The recognition result is matched with the second discrimination data. If the recognition result is the same as the second discrimination data, the discrimination result of the test image is determined to be incorrect.

[0027] If the recognition result is different from the second discrimination data, then based on the second similarity between the recognition result and the second discrimination data, it is determined whether the discrimination result of the test image is incorrect;

[0028] If the second similarity is greater than or equal to the second similarity threshold, the discrimination result of the test image is determined to be incorrect, and the recognition result is added to the second array as new second discrimination data.

[0029] If the similarity is less than the second similarity threshold, the discrimination result of the test image is determined to be unknown.

[0030] Optionally, after determining that the discrimination result of the test image is unknown, the method further includes:

[0031] The recognition results of the test images are sent to the discrimination terminal for manual judgment;

[0032] The manual judgment result from the judgment terminal is accepted, and the manual judgment result is either correct or incorrect;

[0033] If the manual judgment result is correct, the number of correct results is incremented by 1, and the recognition result is added to the first array as new first judgment data.

[0034] If the manual judgment result is incorrect, the number of errors is incremented by 1, and the recognition result is added to the second array as new second judgment data.

[0035] Optionally, for each test image in the test dataset, the process of identifying the main target in the test image using the model to be identified, to obtain the identification result of the test image, includes:

[0036] For each test image in the test dataset, the main target in the test image is identified by the recognition model to obtain the recognition text of the test image;

[0037] The identified text is segmented to obtain the key nouns of the identified text.

[0038] The key words in the identified text are used as the identification results for the test image.

[0039] Secondly, embodiments of the present invention also provide a testing apparatus for a recognition model, the testing apparatus for the recognition model comprising:

[0040] The acquisition module is used to acquire a test dataset and a model to be identified. The test dataset includes test images and labeled data of the main target. The test images include the main target.

[0041] The first processing module is used to identify the main target in each test image in the test dataset using the model to be identified, and to obtain the recognition result of the test image.

[0042] The second processing module is used to judge the recognition results of the test image based on the first array and the second array, and record the number of correct and incorrect recognition results. The first array is used to judge whether the recognition result is correct based on the first discrimination data. The second array is used to record the second discrimination data, which is used to judge whether the recognition result is incorrect.

[0043] The third processing module is used to obtain the final number of correct and incorrect results after performing the discrimination and recording of all the test images in the test dataset.

[0044] The calculation module is used to calculate the measurement result of the model to be identified based on the final number of correct and the number of incorrect.

[0045] Thirdly, embodiments of the present invention provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the testing method for the recognition model provided in embodiments of the present invention.

[0046] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the testing method for the recognition model provided in the embodiments of the invention.

[0047] In this embodiment of the invention, a test dataset and a model to be identified are obtained. The test dataset includes test images and labeled data of the main target. The test images include the main target. For each test image in the test dataset, the model to be identified performs identification processing on the main target in the test image to obtain the identification result of the test image. Based on a first array and a second array, the identification results of the test images are judged, and the number of correct and incorrect identification results are recorded. The first array is used to record the first discrimination data, which is used to judge whether the identification result is correct. The second array is used to record the second discrimination data, which is used to judge whether the identification result is incorrect. After the discrimination and recording of all test images in the test dataset are completed, the final number of correct and incorrect results are obtained. Based on the final number of correct and incorrect results, the test result of the model to be identified is calculated. For each test image in the test dataset, the identification model identifies the main target in the test image, obtaining the identification result. Based on the first and second arrays, the identification results are judged, and the number of correct and incorrect identification results is recorded. After judging and recording all test images in the test dataset, the final number of correct and incorrect results is obtained. Using the final number of correct and incorrect results, the test result of the identification model is calculated, thereby improving the reliability and efficiency of the test results. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1This is a flowchart of a testing method for a recognition model provided in an embodiment of the present invention;

[0050] Figure 2 This is a flowchart of another testing method for an identification model provided in an embodiment of the present invention;

[0051] Figure 3 This is a schematic diagram of the structure of a testing device for recognizing a model provided in an embodiment of the present invention;

[0052] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] like Figure 1 As shown, Figure 1 This is a flowchart of a testing method for a recognition model provided in an embodiment of the present invention. The testing method for the recognition model includes the following steps:

[0055] 101. Obtain the test dataset and the model to be identified.

[0056] In this embodiment of the invention, the test dataset includes test images and labeled data of the main target. The test images include the main target. The labeled data is used to guide the model in learning how to identify the main target in different images.

[0057] The test images mentioned above can be pictures of household items, plants, or animals, taken with a camera or uploaded by users. The main target can be understood as the main item in the test image, such as a table, bench, potted plant, or cat.

[0058] The aforementioned model to be identified can be a recognition model built based on deep learning or machine learning, such as a convolutional neural network (CNN) or a recurrent neural network (RNN). This model is trained using a training dataset, which includes sample images and labeled data of the main targets within those images.

[0059] 102. For each test image in the test dataset, the main target in the test image is identified using the recognition model to obtain the recognition result of the test image.

[0060] In this embodiment of the invention, the above-mentioned recognition process can be understood as the process of analyzing and understanding the input test image to identify the main target in the image.

[0061] Specifically, the model analyzes each input test image, identifies the main target in the image, and outputs a corresponding recognition result. This recognition result is either correct or incorrect.

[0062] In one possible embodiment, if the test image is a picture of a "neck pillow" and the initial label name is "neck pillow", and the recognition result output by the model to be recognized is "pillow wrap", then "pillow wrap" is the recognition result of the test image.

[0063] In another possible embodiment, if the test image is a picture of an "office desk", the initial label name is "office desk", and the recognition result output by the recognition model is "office desk", then "office desk" is the recognition result of the test image.

[0064] 103. Based on the first array and the second array, judge the recognition results of the test images and record the number of correct and incorrect recognition results.

[0065] In this embodiment of the invention, the first array is used to record the first discrimination data, which is used to determine whether the recognition result is correct; the second array is used to record the second discrimination data, which is used to determine whether the recognition result is incorrect.

[0066] The recognition result of each test image can be compared with the first set of discriminant data to determine if the recognition result is correct. If the recognition result matches the first set of discriminant data, the recognition result of the test image is correct, and when counting correct recognition results, the recognition result of the test image will be added to the first array and the count of correct recognition results will be incremented by 1. If the recognition result of the test image is incorrect, it is compared with the second set of discriminant data to determine if the recognition result of the test image is incorrect. If the recognition result matches the second set of discriminant data, the recognition result of the test image is incorrect, and when counting incorrect recognition results, the recognition result of the test image will be added to the second array and the count of incorrect recognition results will be incremented by 1.

[0067] In one possible embodiment, the test image is an image of a "neck pillow" and the initial label name is "neck pillow". If the recognition result of the model to be recognized is "neck pillow", then the recognition result is correct and the correct count is incremented by 1.

[0068] In another possible embodiment, the test image is an image of a "neck pillow," initially labeled as "neck pillow." If the recognition model identifies it as "pillowcase," the recognition result is incorrect, and the error count is incremented by 1.

[0069] 104. After completing the discrimination and recording of all test images in the test dataset, the final number of correct and incorrect results is obtained.

[0070] In this embodiment of the invention, the number of correct recognition results that match the first discrimination data for all test images in the test dataset can be counted to obtain the number of correct recognition results. Then, the number of incorrect recognition results that match the second discrimination data for the test images can be counted to obtain the number of incorrect recognition results.

[0071] In one possible embodiment, if the model's judgment result is unknown, the recognition result can be manually judged to see if it is correct. If the manual judgment result is correct, the correct number is incremented by 1 until the correct number at the end of the manual judgment is added to the correct number of the recognition result to obtain the final correct number. If the manual judgment result is incorrect, the incorrect number is incremented by 1 until the incorrect number at the end of the manual judgment is added to the incorrect number of the recognition result to obtain the final incorrect number.

[0072] 105. Based on the final number of correct and incorrect results, calculate the test results of the model to be identified.

[0073] In this embodiment of the invention, the test result of the model to be identified can be calculated by comparing the number of correct and incorrect results identified by the model on the test data.

[0074] In this embodiment of the invention, a test dataset and a model to be identified are obtained. The test dataset includes test images and labeled data of the main target. The test images include the main target. For each test image in the test dataset, the model to be identified performs identification processing on the main target in the test image to obtain the identification result of the test image. Based on a first array and a second array, the identification results of the test images are judged, and the number of correct and incorrect identification results are recorded. The first array is used to record the first discrimination data, which is used to judge whether the identification result is correct. The second array is used to record the second discrimination data, which is used to judge whether the identification result is incorrect. After the discrimination and recording of all test images in the test dataset are completed, the final number of correct and incorrect results are obtained. Based on the final number of correct and incorrect results, the test result of the model to be identified is calculated. For each test image in the test dataset, the identification model identifies the main target in the test image, obtaining the identification result. Based on the first and second arrays, the identification results are judged, and the number of correct and incorrect identification results is recorded. After judging and recording all test images in the test dataset, the final number of correct and incorrect results is obtained. Using the final number of correct and incorrect results, the test result of the identification model is calculated, thereby improving the reliability and efficiency of the test results.

[0075] It is understood that in the specific implementation of this application, data related to robots, home appliances, and tasks are involved. When the embodiments in this application are applied to specific products or technologies, user permission or consent is required. Furthermore, the collection, use, and processing of related data, as well as the training, deployment, and invocation of large language models, must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0076] Optionally, before the step of judging and recording the recognition results of the test image based on the first array and the second array, the first array can be initialized and configured, all labeled data in the test dataset can be used as the initial first discrimination data, and the recognition results with correct discrimination results can be added as new first discrimination data to the first array; and the second array can be initialized and configured, the second array is initially empty, and during the discrimination process, the recognition results with incorrect discrimination results can be added as new second discrimination data to the second array.

[0077] In this embodiment of the invention, the aforementioned initialization configuration can be understood as the process of predefining and setting necessary parameters, variables, or resources before execution. Initialization configuration ensures that the program can start execution from the correct state, avoiding errors caused by using uninitialized data. The initialization configuration can be a value of 0, an empty string, null, or other specific default values. This is done to prevent undefined behavior before the variables are used. Initialization configuration is fundamental to ensuring the correct and stable operation of the program and can effectively avoid many common errors.

[0078] The labeled data mentioned above includes the category labels corresponding to the main target, which are used to evaluate the model's performance.

[0079] Specifically, the first array is initialized by using all labeled data from the test dataset as initial first discriminant data. During recognition, correctly identified results are added to the first array as new first discriminant data. The second array is then initialized without any data. During recognition, incorrectly identified results are added to the second array as new second discriminant data.

[0080] Optionally, in the step of judging and recording the recognition results of the test images based on the first array and the second array, for each test image, based on the recognition result and the first discrimination data, it can be determined whether the discrimination result of the test image is correct; if the discrimination result of the test image is correct, the number of correct records is incremented by 1; if the discrimination result of the test image is unsuccessful, based on the recognition result and the second discrimination data, it can be determined whether the discrimination result of the test image is incorrect; if the discrimination result of the test image is incorrect, the number of incorrect records is incremented by 1.

[0081] In this embodiment of the invention, the above recognition result is obtained by processing the input test image through a recognition model.

[0082] The first discriminant data is used to determine whether the recognition result is correct; the second discriminant data is used to determine whether the recognition result is incorrect.

[0083] Specifically, based on the recognition result of each test image, the result can be compared with the first discrimination data to determine if the recognition result is correct. If the recognition result matches the first discrimination data, the recognition result of the test image is correct, and when counting correct recognition results, the recognition result of the test image will be added to the first array and the correct count will be incremented by 1. The aforementioned first array is used to record the first discrimination data. If the recognition result does not match the first discrimination data, the recognition result of the test image is unsuccessful, meaning that the recognition result cannot be determined to be correct.

[0084] Furthermore, if the recognition result of the test image is unsuccessful, the recognition result is compared with the second discrimination data to determine if the recognition result is incorrect. If the recognition result matches the second discrimination data, the recognition result of the test image is incorrect, and when counting incorrect recognition results, the recognition result of the test image will be added to the second array and the error count will be incremented by 1. The aforementioned second array is used to record the second discrimination data. If the recognition result does not match the second discrimination data, the recognition result of the test image is unknown, meaning it cannot be determined whether the recognition result is correct or incorrect.

[0085] Optionally, in the step of determining whether the discrimination result of the test image is correct based on the recognition result and the first discrimination data, the recognition result can be matched with the first discrimination data. If the recognition result is the same as the first discrimination data, the discrimination result of the test image is determined to be correct. If the recognition result is different from the first discrimination data, the discrimination result of the test image is determined to be correct based on the first similarity between the recognition result and the first discrimination data. If the first similarity is greater than or equal to the first similarity threshold, the discrimination result of the test image is determined to be correct, and the recognition result is added to the first array as new first discrimination data. If the similarity is less than the first similarity threshold, the discrimination result of the test image is determined to be unsuccessful.

[0086] In this embodiment of the invention, the first discrimination data is used to determine whether the recognition result is correct.

[0087] The above matching can be understood as a process of comparing whether the identification result is completely consistent with the first discrimination data.

[0088] The aforementioned first similarity can be understood as an indicator measuring the degree of similarity between the recognition result of the test image and the first discriminant data. The semantic cosine similarity between the recognition result of the test image and each element marked in the first discriminant data can be calculated using the cosine similarity calculation method. The cosine similarity calculation method is an indicator measuring the degree of similarity between two vectors, measuring their similarity by calculating the cosine value of the angle between the two vectors. First, the recognition result and the first discriminant data are converted into vector form, then standardized so that the modulus (Euclidean norm) of each vector is 1. The dot product of the standardized two vectors is then calculated, and the dot product result is multiplied by the product of the moduli of the two vectors. The result is the cosine value between the two vectors. The cosine value ranges from [-1, 1]. The closer the value is to 1, the more similar the directions of the two vectors are; the closer the value is to 0, the lower the similarity. The dot product is the sum of the product of corresponding elements.

[0089] The first similarity threshold mentioned above is a similarity threshold preset by the system. It is the most suitable empirical value obtained from experimental testing, and can be 0.95.

[0090] Specifically, the recognition result of the test image is compared with the first discriminant data. If they are the same, the discrimination result of the test image is determined to be correct. If the recognition result is different from the first discriminant data, the similarity between them can be calculated using the cosine similarity calculation method to determine the correctness of the discrimination result. If the semantic cosine similarity between the recognition result and the first discriminant data is greater than or equal to 0.95, the discrimination result of the test image is correct, and the current recognition result is added to the first array as the new first discriminant data for use in subsequent comparisons.

[0091] Furthermore, if the semantic cosine similarity between the recognition result and the first discrimination data is less than 0.95, the discrimination result of the test image is unsuccessful, and it cannot be determined whether the recognition result is correct. It is necessary to determine whether the recognition result is incorrect or to make a manual judgment to determine whether the recognition result is correct or incorrect.

[0092] In one possible implementation, for example, the test image is an image of a "neck pillow" with the initial label name "neck pillow". If the recognition model identifies "neck pillow" as the recognition result, the recognition result is correct. If the recognition model identifies "pillow wrap" as the recognition result, the semantic pre-similarity calculation method can be used to calculate the semantic pre-similarity between "pillow wrap" and "neck pillow". If the semantic pre-similarity is greater than or equal to 0.95, the recognition result is determined to be correct, and the recognition result "pillow wrap" is added to the first array as new first discrimination data. When the recognition model outputs "pillow wrap" later, the recognition result can be automatically determined to be correct, and the judgment result can be obtained quickly.

[0093] Optionally, in the step of determining whether the discrimination result of the test image is incorrect based on the recognition result and the second discrimination data, the recognition result can be matched with the second discrimination data. If the recognition result and the second discrimination data are the same, the discrimination result of the test image is determined to be incorrect. If the recognition result and the second discrimination data are different, the discrimination result of the test image is determined to be incorrect based on the second similarity between the recognition result and the second discrimination data. If the second similarity is greater than or equal to the second similarity threshold, the discrimination result of the test image is determined to be incorrect, and the recognition result is added as new second discrimination data to the second array. If the similarity is less than the second similarity threshold, the discrimination result of the test image is determined to be unknown.

[0094] In this embodiment of the invention, the second discrimination data is used to determine whether the recognition result is incorrect.

[0095] The aforementioned second similarity can be understood as an indicator measuring the degree of similarity between the recognition result of the test image and the second discriminant data. The semantic cosine similarity between the recognition result of the test image and each element labeled in the second discriminant data can be calculated using the cosine similarity calculation method.

[0096] The second similarity threshold can be a pre-set similarity threshold by the system, which is the most suitable empirical value obtained from experimental testing, specifically 0.98.

[0097] Specifically, the recognition result of the test image is compared with the second discriminant data. If they are the same, the discrimination result of the test image is determined to be incorrect. If the recognition result is different from the second discriminant data, the similarity between them can be calculated using the cosine similarity calculation method to determine the correctness of the discrimination result. If the semantic cosine similarity between the recognition result and the second discriminant data is greater than or equal to 0.98, the discrimination result of the test image is determined to be incorrect, and the current recognition result is added to the second array as new second discriminant data for use in subsequent comparisons.

[0098] Furthermore, if the semantic cosine similarity between the recognition result and the second discriminant data is less than 0.98, the difference between the recognition result and the second discriminant data is too large to be judged simply by similarity. Therefore, in this case, the system will determine the discrimination result of the test image as "unknown".

[0099] Optionally, after determining that the discrimination result of the test image is unknown, the recognition result of the test image can also be sent to the discrimination terminal for manual discrimination; the manual discrimination result of the discrimination terminal is received, and the manual discrimination result is either correct or incorrect; if the manual discrimination result is correct, the correct count is incremented by 1, and the recognition result is added to the first array as new first discrimination data; if the manual discrimination result is incorrect, the incorrect count is incremented by 1, and the recognition result is added to the second array as new second discrimination data.

[0100] In this embodiment of the invention, the aforementioned discrimination terminal can be understood as a system specifically designed for manual judgment and classification.

[0101] The aforementioned manual judgment can be understood as the process of reviewing and confirming the results of the judgment system through the knowledge and experience of professionals. Specifically, the personnel at the judgment terminal will receive images for which the system cannot determine the result, analyze the image content, judge whether the automatic system's recognition result is correct, and give a final judgment.

[0102] Specifically, once the discrimination result of the test image is determined to be unknown, the recognition result of the test image can be sent to the discrimination terminal for manual discrimination, and the system waits to receive the manual discrimination result from the discrimination intermediate. If the manual discrimination result is "correct", the number of correct results is incremented by 1, and this recognition result is added to the first array as the new first discrimination data.

[0103] If the manual judgment result is "error", the error count is incremented by 1, and this recognition result is added to the second array as new second judgment data.

[0104] It should be noted that when the judgment result of the test image is unknown, manual verification can be used to improve the system's accuracy in judging images with unknown results, and the performance of the automatic judgment model can be continuously optimized by collecting this data.

[0105] In one possible implementation, for example, the test image is an image of a "neck pillow" with the initial label name "neck pillow". If the recognition result of the model to be recognized is "unknown", then a manual judgment method can be used to manually judge the test image to determine whether the recognition result is correct or incorrect.

[0106] Optionally, in the step of identifying the main target in each test image in the test dataset using the model to be identified to obtain the recognition result of the test image, the main target in each test image in the test dataset can be identified using the model to be identified to obtain the recognition text of the test image; the recognition text can be segmented to obtain the key words of the recognition text; and the key words of the recognition text can be determined as the recognition result of the test image.

[0107] In this embodiment of the invention, the model to be identified is trained using a training dataset, which includes sample images and labeled data of the main targets in the sample images. The model to be identified is capable of recognizing different objects, scenes, animals, etc.

[0108] The aforementioned recognized text can be understood as a text description of the test image. It can be keywords or a sentence that summarizes the main object in the image. For example, if there is an office desk in the test image, the text description "office desk" can be generated; if there is a backpack in the test image, the text description "backpack" can be generated.

[0109] The word segmentation process described above can be understood as the process of identifying keywords in the identified text. Keywords are the main features and core components of the test image.

[0110] Specifically, the model to be recognized is used to recognize each test image in the test dataset, identify the descriptive text of the test image, and perform keyword extraction on the descriptive text to obtain the keywords of the descriptive text. The keywords of the descriptive text can summarize the main content and theme of the test image and can be directly used as the recognition result of the test image.

[0111] like Figure 2 As shown, Figure 2 This is a flowchart of another testing method for a recognition model provided in an embodiment of the present invention. Specifically, it includes:

[0112] 200. Begin.

[0113] 201. Obtain the test dataset.

[0114] The test dataset includes test images and labeled data of the main target. The test images include the main target.

[0115] 202. Construct the first array.

[0116] The first array can be a correct array CorrectList: [], used to record the first discrimination data, which is used to determine whether the recognition result is correct.

[0117] 203. Construct the second array.

[0118] The second array can be an error array ErrorList:[].

[0119] 204. Input the test dataset into the recognition model.

[0120] 205. Recognition model output results.

[0121] 206. Determine if the recognition result is in the first array?

[0122] If the recognition result is in the first array, proceed to step 207; if the recognition result is not in the first array, proceed to step 208.

[0123] 207. The recognition result is correct.

[0124] If the identification result is correct, the number of correct results is incremented by 1, and the process proceeds to step 230.

[0125] 208. Determine if the recognition result is in the second array?

[0126] If the recognition result is in the second array, proceed to step 209; if the recognition result is not in the second array, proceed to step 210.

[0127] 209. Recognition result is incorrect.

[0128] If the identification result is incorrect, the error count is incremented by 1, and the next step is 230.

[0129] 210. Calculate the first semantic cosine similarity between the recognition result and the first discriminant data in the first array.

[0130] The semantic cosine similarity calculation mentioned above is an indicator that measures the degree of similarity between two vectors. It measures their similarity by calculating the cosine value of the angle between the two vectors.

[0131] 211. Determine whether the first similarity result exceeds the first empirical threshold.

[0132] If the first similarity result exceeds the first empirical threshold, the recognition result is correct, and the process proceeds to step 212; if the first similarity result is lower than the first empirical threshold, the process proceeds to step 214. The aforementioned first empirical threshold is an empirical value obtained from experimental testing, and may specifically be 0.95.

[0133] 212. The recognition result is correct.

[0134] If the identification result is correct, the number of correct results is incremented by 1, and the process proceeds to step 213.

[0135] 213. Add the recognition results to the first array.

[0136] The recognition results are added to the first array, and then proceed to step 230.

[0137] 214. Calculate the second semantic cosine similarity between the recognition result and the first discriminant data.

[0138] 215. Determine whether the second similarity result exceeds the second empirical threshold.

[0139] If the second similarity result exceeds the second empirical threshold, proceed to step 216; if the second similarity result does not exceed the second empirical threshold, proceed to step 218. The second similarity result refers to the similarity result calculated at the second semantic cosine similarity point between the recognition result and the first discriminant data. The second empirical threshold is an empirical value obtained from experimental testing, specifically 0.98.

[0140] 216. Recognition result is incorrect.

[0141] If the identification result is incorrect, the error count is incremented by 1, and the process proceeds to step 217.

[0142] 217. The recognition results are added to the second array.

[0143] The recognition results are then added to the second array, proceeding to step 230.

[0144] 218. Introduce a word segmentation library and extract keywords from the recognition results as the word segmentation results.

[0145] The aforementioned word segmentation library can be the jieba word segmentation library, which segments the recognition results and extracts the main keywords from the recognition results as the word segmentation results.

[0146] 219. Determine whether the word segmentation result is in the first array.

[0147] If the word segmentation result is in the first array, proceed to step 220; if the word segmentation result is not in the first array, proceed to step 222.

[0148] 220. The recognition result is correct.

[0149] If the identification result is correct, the number of correct results is incremented by 1, and the process proceeds to step 221.

[0150] 221. Add the recognition results to the first array.

[0151] The recognition results are added to the first array, and then proceed to step 230.

[0152] 222. Determine whether the word segmentation result is in the second array.

[0153] If the word segmentation result is in the second array, proceed to step 223; if the word segmentation result is not in the second array, proceed to step 225.

[0154] 223. Recognition result is incorrect.

[0155] If the identification result is incorrect, the error count is incremented by 1, and the process proceeds to step 224.

[0156] 224. The recognition results are added to the second array.

[0157] The recognition results are then added to the second array, proceeding to step 230.

[0158] 225. Manually determine whether the recognition result is correct.

[0159] If the manual identification result is correct, proceed to step 226; if the manual identification result is incorrect, proceed to step 228.

[0160] 226. The recognition result is correct.

[0161] If the identification result is correct, the number of correct results is incremented by 1, and the process proceeds to step 227.

[0162] 227. Add the recognition results to the first array.

[0163] The recognition results are added to the first array, and then proceed to step 230.

[0164] 228. Recognition result is incorrect.

[0165] If the identification result is incorrect, the error count is incremented by 1, and the process proceeds to step 229.

[0166] 229. The recognition results are added to the second array.

[0167] The recognition results are then added to the second array, proceeding to step 230.

[0168] 230. Count the number of correct identification results and calculate the accuracy rate.

[0169] 231. End.

[0170] In this embodiment, the present invention improves the reliability and efficiency of test results by combining automated script statistics with manual statistics.

[0171] like Figure 3 As shown, an embodiment of the present invention provides a testing device for a recognition model, the testing device for the recognition model comprising:

[0172] The acquisition module 301 is used to acquire a test dataset and a model to be identified. The test dataset includes test images and labeled data of the main target. The test images include the main target.

[0173] The first processing module 302 is used to identify the main target in each test image in the test dataset using the model to be identified, and to obtain the recognition result of the test image.

[0174] The second processing module 303 is used to judge the recognition result of the test image based on the first array and the second array, and record the number of correct and incorrect recognition results. The first array is used to record the first discrimination data, which is used to judge whether the recognition result is correct. The second array is used to record the second discrimination data, which is used to judge whether the recognition result is incorrect.

[0175] The third processing module 304 is used to obtain the final number of correct and incorrect results after performing the discrimination and recording of all the test images in the test dataset.

[0176] The calculation module 305 is used to calculate the test results of the model to be identified based on the final number of correct answers and the number of incorrect answers.

[0177] Optionally, the device is further configured to initialize and configure a first array, taking all the labeled data in the test dataset as initial first discrimination data, and adding the recognition results with correct discrimination results as new first discrimination data to the first array; and initialize and configure a second array, which is initially empty, and during the discrimination process, adding the recognition results with incorrect discrimination results as new second discrimination data to the second array.

[0178] Optionally, the second processing module 303 is further configured to, for each test image, determine whether the discrimination result of the test image is correct based on the recognition result and the first discrimination data; if the discrimination result of the test image is correct, then increment the number of correct images by 1; if the discrimination result of the test image is unsuccessful, then determine whether the discrimination result of the test image is incorrect based on the recognition result and the second discrimination data; if the discrimination result of the test image is incorrect, then increment the number of incorrect images by 1.

[0179] Optionally, the second processing module 303 is further configured to match the recognition result with the first discrimination data. If the recognition result is the same as the first discrimination data, the discrimination result of the test image is determined to be correct. If the recognition result is different from the first discrimination data, the discrimination result of the test image is determined to be correct based on the first similarity between the recognition result and the first discrimination data. If the first similarity is greater than or equal to the first similarity threshold, the discrimination result of the test image is determined to be correct, and the recognition result is added to the first array as new first discrimination data. If the similarity is less than the first similarity threshold, the discrimination result of the test image is determined to be unsuccessful.

[0180] Optionally, the second processing module 303 is further configured to match the recognition result with the second discrimination data. If the recognition result is the same as the second discrimination data, the discrimination result of the test image is determined to be incorrect. If the recognition result is different from the second discrimination data, the discrimination result of the test image is determined to be incorrect based on the second similarity between the recognition result and the second discrimination data. If the second similarity is greater than or equal to the second similarity threshold, the discrimination result of the test image is determined to be incorrect, and the recognition result is added to the second array as new second discrimination data. If the similarity is less than the first similarity threshold, the discrimination result of the test image is determined to be unknown.

[0181] Optionally, the device is further configured to send the recognition result of the test image to a discrimination terminal for manual discrimination; receive the manual discrimination result from the discrimination terminal, wherein the manual discrimination result is correct or incorrect; if the manual discrimination result is correct, record the correct count incremented by 1, and add the recognition result as new first discrimination data to the first array; if the manual discrimination result is incorrect, record the incorrect count incremented by 1, and add the recognition result as new second discrimination data to the second array.

[0182] Optionally, the first processing module 302 is further configured to, for each test image in the test dataset, perform recognition processing on the main target in the test image using the recognition model to obtain the recognition text of the test image; perform word segmentation processing on the recognition text to obtain the key words of the recognition text; and determine the key words of the recognition text as the recognition result of the test image.

[0183] like Figure 4 As shown, this embodiment of the invention also provides an electronic device, including a processor, which can execute any of the above-described methods for testing recognition models.

[0184] Specifically, it includes a processor 401 and a memory 402, as well as a computer program stored in the memory 402 and capable of running on the processor 401, which executes a test method for recognizing a model, wherein:

[0185] The processor 401 executes the calculator program for the test method of the recognition model stored in the memory 402, and performs the following steps:

[0186] Obtain a test dataset and a model to be identified. The test dataset includes test images and labeled data of the main target. The test images include the main target.

[0187] For each test image in the test dataset, the main target in the test image is identified using the model to be identified, and the recognition result of the test image is obtained.

[0188] Based on the first array and the second array, the recognition results of the test image are judged, and the number of correct and incorrect recognition results are recorded. The first array is used to record the first discrimination data, which is used to judge whether the recognition result is correct. The second array is used to record the second discrimination data, which is used to judge whether the recognition result is incorrect.

[0189] After performing the discrimination and recording of all the test images in the test dataset, the final number of correct and incorrect results are obtained.

[0190] Based on the final number of correct answers and the number of incorrect answers, the test results of the model to be identified are calculated.

[0191] Optionally, before the recognition results of the test image are judged and recorded based on the first array and the second array, the method executed by the processor 401 further includes:

[0192] Initialize the configuration of the first array, take all the labeled data in the test dataset as the initial first discrimination data, and add the recognition results with correct discrimination results as new first discrimination data to the first array;

[0193] The second array is initialized and configured. The second array is initially empty. During the discrimination process, the recognition result that is incorrect is added to the second array as new second discrimination data.

[0194] Optionally, the process of processor 401 judging and recording the recognition results of the test image based on the first array and the second array includes:

[0195] For each test image, based on the recognition result and the first discrimination data, it is determined whether the discrimination result of the test image is correct;

[0196] If the judgment result of the test image is correct, then the number of correct results is incremented by 1.

[0197] If the discrimination result of the test image is unsuccessful, then based on the recognition result and the second discrimination data, it is determined whether the discrimination result of the test image is incorrect;

[0198] If the judgment result of the test image is incorrect, then the number of errors is incremented by 1.

[0199] Optionally, the step of determining whether the discrimination result of the test image is correct based on the recognition result and the first discrimination data executed by the processor 401 includes:

[0200] The recognition result is matched with the first discrimination data. If the recognition result is the same as the first discrimination data, the discrimination result of the test image is determined to be correct.

[0201] If the recognition result is different from the first discrimination data, then based on the first similarity between the recognition result and the first discrimination data, it is determined whether the discrimination result of the test image is correct;

[0202] If the first similarity is greater than or equal to the first similarity threshold, then the discrimination result of the test image is determined to be correct, and the recognition result is added to the first array as new first discrimination data;

[0203] If the similarity is less than the first similarity threshold, then the discrimination result of the test image is determined to be unsuccessful.

[0204] Optionally, the step of determining whether the discrimination result of the test image is incorrect based on the recognition result and the second discrimination data, executed by the processor 401, includes:

[0205] The recognition result is matched with the second discrimination data. If the recognition result is the same as the second discrimination data, the discrimination result of the test image is determined to be incorrect.

[0206] If the recognition result is different from the second discrimination data, then based on the second similarity between the recognition result and the second discrimination data, it is determined whether the discrimination result of the test image is incorrect;

[0207] If the second similarity is greater than or equal to the second similarity threshold, the discrimination result of the test image is determined to be incorrect, and the recognition result is added to the second array as new second discrimination data.

[0208] If the similarity is less than the first similarity threshold, the discrimination result of the test image is determined to be unknown.

[0209] Optionally, after determining that the discrimination result of the test image is unknown, the method executed by the processor 401 further includes:

[0210] The recognition results of the test images are sent to the discrimination terminal for manual judgment;

[0211] The manual judgment result from the judgment terminal is accepted, and the manual judgment result is either correct or incorrect;

[0212] If the manual judgment result is correct, the number of correct results is incremented by 1, and the recognition result is added to the first array as new first judgment data.

[0213] If the manual judgment result is incorrect, the number of errors is incremented by 1, and the recognition result is added to the second array as new second judgment data.

[0214] Optionally, the processor 401 executes the following step: for each test image in the test dataset, the processor performs recognition processing on the main target in the test image using the recognition model to obtain the recognition result of the test image, including:

[0215] For each test image in the test dataset, the main target in the test image is identified by the recognition model to obtain the recognition text of the test image;

[0216] The identified text is segmented to obtain the key nouns of the identified text.

[0217] The key words in the identified text are used as the identification results for the test image.

[0218] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the testing method for the recognition model or the testing method for the application-side recognition model provided in this invention, and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0219] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0220] The above description discloses only preferred embodiments of the present invention and should not be construed as limiting the scope of the present invention. Therefore, equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A testing method for a recognition model, characterized in that, The method includes the following steps: Obtain a test dataset and a model to be identified. The test dataset includes test images and labeled data of the main target. The test images include the main target. For each test image in the test dataset, the main target in the test image is identified using the model to be identified, and the recognition result of the test image is obtained. Based on the first array and the second array, the recognition results of the test image are judged, and the number of correct and incorrect recognition results are recorded. The first array is used to record the first discrimination data, which is used to judge whether the recognition result is correct. The second array is used to record the second discrimination data, which is used to judge whether the recognition result is incorrect. After performing the discrimination and recording of all the test images in the test dataset, the final number of correct and incorrect results are obtained. Based on the final number of correct answers and the number of incorrect answers, the test results of the model to be identified are calculated.

2. The testing method for the recognition model as described in claim 1, characterized in that, Before judging and recording the recognition results of the test image based on the first array and the second array, the method further includes: Initialize the configuration of the first array, take all the labeled data in the test dataset as the initial first discrimination data, and add the recognition results with correct discrimination results as new first discrimination data to the first array; Additionally, the second array is initialized and configured. The second array is initially empty. During the discrimination process, the identification result that is incorrect is added to the second array as new second discrimination data.

3. The testing method for the recognition model as described in claim 2, characterized in that, The step of judging and recording the recognition results of the test image based on the first array and the second array includes: For each test image, based on the recognition result and the first discrimination data, it is determined whether the discrimination result of the test image is correct; If the judgment result of the test image is correct, then the number of correct results is incremented by 1. If the discrimination result of the test image is unsuccessful, then based on the recognition result and the second discrimination data, it is determined whether the discrimination result of the test image is incorrect; If the judgment result of the test image is incorrect, then the number of errors is incremented by 1.

4. The testing method for the recognition model as described in claim 3, characterized in that, The step of determining whether the discrimination result of the test image is correct based on the recognition result and the first discrimination data includes: The recognition result is matched with the first discrimination data. If the recognition result is the same as the first discrimination data, the discrimination result of the test image is determined to be correct. If the recognition result is different from the first discrimination data, then based on the first similarity between the recognition result and the first discrimination data, it is determined whether the discrimination result of the test image is correct; If the first similarity is greater than or equal to the first similarity threshold, then the discrimination result of the test image is determined to be correct, and the recognition result is added to the first array as new first discrimination data; If the similarity is less than the first similarity threshold, then the discrimination result of the test image is determined to be unsuccessful.

5. The testing method for the recognition model as described in claim 3, characterized in that, The step of determining whether the discrimination result of the test image is incorrect based on the recognition result and the second discrimination data includes: The recognition result is matched with the second discrimination data. If the recognition result is the same as the second discrimination data, the discrimination result of the test image is determined to be incorrect. If the recognition result is different from the second discrimination data, then based on the second similarity between the recognition result and the second discrimination data, it is determined whether the discrimination result of the test image is incorrect; If the second similarity is greater than or equal to the second similarity threshold, the discrimination result of the test image is determined to be incorrect, and the recognition result is added to the second array as new second discrimination data. If the similarity is less than the second similarity threshold, the discrimination result of the test image is determined to be unknown.

6. The testing method for the recognition model as described in claim 5, characterized in that, After determining that the discrimination result of the test image is unknown, the method further includes: The recognition results of the test images are sent to the discrimination terminal for manual judgment; Receive the manual judgment result from the judgment terminal, wherein the manual judgment result is correct or incorrect; If the manual judgment result is correct, the number of correct results is incremented by 1, and the recognition result is added to the first array as new first judgment data. If the manual judgment result is incorrect, the number of errors is incremented by 1, and the recognition result is added to the second array as new second judgment data.

7. The testing method for the recognition model as described in any one of claims 1 to 6, characterized in that, For each test image in the test dataset, the identification model is used to identify the main target in the test image to obtain the identification result of the test image, including: For each test image in the test dataset, the main target in the test image is identified by the recognition model to obtain the recognition text of the test image; The identified text is segmented to obtain the key nouns of the identified text; The key words in the identified text are used as the identification results for the test image.

8. A testing device for recognizing models, characterized in that, The testing device for the recognition model includes: The acquisition module is used to acquire a test dataset and a model to be identified. The test dataset includes test images and labeled data of the main target. The test images include the main target. The first processing module is used to identify the main target in each test image in the test dataset using the model to be identified, and to obtain the recognition result of the test image. The second processing module is used to judge the recognition result of the test image based on the first array and the second array, and record the number of correct and incorrect recognition results. The first array is used to record the first discrimination data, which is used to judge whether the recognition result is correct. The second array is used to record the second discrimination data, which is used to judge whether the recognition result is incorrect. The third processing module is used to obtain the final number of correct and incorrect results after performing the discrimination and recording of all the test images in the test dataset. The calculation module is used to calculate the measurement result of the model to be identified based on the final number of correct and the number of incorrect.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps in the testing method for the recognition model as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps in the testing method for the recognition model as described in any one of claims 1 to 7.

Images