Model detection method and device, electronic equipment and computer readable storage medium

Abstract

The embodiment of the invention provides a model detection method and device, electronic equipment and a computer readable storage medium, and relates to the technical field of computer vision, and the method comprises the steps: constructing a test data set for testing a model; initializing a to-be-tested model to obtain an initial model; testing the initial model based on the test data set to obtain a test result; analyzing the test result to obtain an analysis result; if the analysis result meets a preset requirement, generating a test report; and if the analysis result does not meet the preset requirement, iterating the initial model to obtain an iterated target model, taking the target model as the initial model, and skipping to test the initial model based on the test data set to obtain a test result. According to the embodiment of the invention, manual operation is reduced, and test efficiency is improved; the batch test capability is realized; the accuracy is improved; result storage and further analysis are facilitated; and a model iteration process and model comparison are simplified.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a model detection method, a model detection device, an electronic device and a computer-readable storage medium. Background Art

[0002] Large-model multi-target detection refers to the use of large models in deep learning to identify and locate all and multiple targets of different categories in an image. This is a key technology in the field of computer vision. This technology requires not only that the model can accurately identify various objects in the image, but also that they can be located in the image. Since the target objects may have different appearances, shapes, and postures, and are affected by imaging conditions such as lighting, occlusion, angle, distance, background complexity, etc., multi-target detection has always been one of the most challenging problems in the field of computer vision.

[0003] The main function of the large model target detection algorithm test tool is to test whether the large model algorithm is accurate, calculate test indicators, and save test results. The existing target detection algorithm test methods in the prior art are as follows:

[0004] 1) Methods for accessing manual subjective testing of the visual interface system:

[0005] Access the system visualization interface to upload images or video streams to the algorithm module. After processing, the system returns the recognition results or alarm information, and the manual statistics results are used to calculate some indicators. The disadvantages are: repeated operations, manual statistics results are troublesome and prone to errors, and only some indicators can be calculated, which consumes a lot of manpower and is costly.

[0006] 2) How to use interface testing tools:

[0007] The image or video stream is passed to the algorithm interface to obtain the recognition information, and then the recognition information is tested to see if it matches the result. The disadvantages are: data processing is difficult, the returned information is not easy to process, the indicators cannot be directly counted, the algorithm prediction box cannot be drawn on the image for the detection result, and the data cannot be visualized.

[0008] 3) Method of directly calling the algorithm interface using Python language:

[0009] The labeled data is passed to the algorithm interface to obtain the recognition result information, and the performance indicators are calculated using the Python language. The disadvantages are: the applicable scenario is single, suitable for traditional algorithms for single target detection, and as the number of categories increases, it is impossible to effectively evaluate the performance of large-scale category target detection algorithms and compare multiple algorithms. Summary of the invention

[0010] In view of the above problems, embodiments of the present invention are proposed to provide a model detection method, a model detection device, an electronic device and a computer-readable storage medium that overcome the above problems or at least partially solve the above problems.

[0011] The embodiment of the present invention discloses a model detection method, which includes:

[0012] Build a test dataset for testing the model;

[0013] Initialize the model to be tested to obtain an initial model;

[0014] Testing the initial model based on the test data set to obtain a test result;

[0015] Analyze the test results to obtain analysis results;

[0016] If the analysis result meets the preset requirements, a test report is generated; if the analysis result does not meet the preset requirements, the initial model is iterated to obtain the iterated target model, the target model is used as the initial model, and the process jumps to the test of the initial model based on the test data set to obtain the test results.

[0017] In an embodiment of the present invention, the step of constructing a test data set for testing a model includes:

[0018] For the acquired original image data set, preprocessing the original image data set to obtain a target image data set; the original image data set includes at least one original image data, and the target image data set includes at least one target image data;

[0019] Performing target recognition on the target image data set to obtain a candidate annotation box set and a category identification set corresponding to the target image data set; the category identification includes an identification of at least one category of the target;

[0020] Filtering and supplementing the candidate annotation frame set to obtain a target annotation frame set; the target annotation frame set includes at least one target annotation frame;

[0021] The first coordinate point of each target annotation box is determined, and a first information file of the target image data set is generated based on the first coordinate point and the category identification set.

[0022] In the embodiment of the present invention, the step of initializing the model to be tested to obtain an initial model includes:

[0023] A model to be tested is obtained from a model database based on a preset script, and the model to be tested is initialized to obtain an initial model; the model database includes at least one untested model.

[0024] In the embodiment of the present invention, the testing of the initial model based on the test data set to obtain the test result includes:

[0025] Using the initial model to perform target recognition on the test data set to obtain at least one target recognition frame;

[0026] Determine a second coordinate point of the at least one target recognition frame, and generate a second information file based on the second coordinate point;

[0027] Based on the first information file and the second information file in the test data set, the intersection ratio of the coordinate points of each target image data is calculated to obtain a test result.

[0028] In the embodiment of the present invention, the step of analyzing the test result to obtain the analysis result includes:

[0029] Calculate, according to the test results, a first average accuracy, a first precision and a first recall rate of each category of targets in the test data set at each confidence level, as well as a second average accuracy, a second precision and a second recall rate of the initial model;

[0030] Determining the detection speed and computing resource consumption of the initial model in the test;

[0031] Calculating a weighted score of the initial model using the second average correct rate, the second precision rate, the second recall rate, the detection speed, and the computing resource consumption;

[0032] The first average accuracy, the first precision, the first recall and the weighted score are taken as analysis results.

[0033] In the embodiment of the present invention, if the analysis result does not meet the preset requirement, iterating the initial model to obtain an iterated target model includes:

[0034] If the first average accuracy in the analysis result does not exceed the average accuracy threshold, and / or the first precision does not exceed the precision threshold, and / or the first recall does not exceed the recall threshold, and / or the weighted score does not exceed the weighted score threshold, then the analysis result does not meet the preset requirement;

[0035] The initial model is iterated to obtain an iterated target model.

[0036] In an embodiment of the present invention, it also includes:

[0037] Using the first information file in the test data set to mark a first marking box in the target image data set, and using the second information file in the test result to mark a second marking box in the target image data set, to obtain a marked target image data set;

[0038] The labeled target image dataset is displayed.

[0039] Accordingly, an embodiment of the present invention discloses a model detection device, the device comprising:

[0040] A building module for building a test dataset for testing the model;

[0041] An initialization module is used to initialize the model to be tested to obtain an initial model;

[0042] A testing module, used to test the initial model based on the test data set to obtain a test result;

[0043] An analysis module, used to analyze the test results to obtain analysis results;

[0044] A generating module, used for generating a test report if the analysis result meets the preset requirements;

[0045] The iteration module is used to iterate the initial model if the analysis result does not meet the preset requirement, obtain the iterated target model, use the target model as the initial model, and call the test module.

[0046] In the embodiment of the present invention, the building module is specifically used for:

[0047] For the acquired original image data set, preprocessing the original image data set to obtain a target image data set; the original image data set includes at least one original image data, and the target image data set includes at least one target image data;

[0048] Performing target recognition on the target image data set to obtain a candidate annotation box set and a category identification set corresponding to the target image data set; the category identification includes an identification of at least one category of the target;

[0049] Filtering and supplementing the candidate annotation frame set to obtain a target annotation frame set; the target annotation frame set includes at least one target annotation frame;

[0050] The first coordinate point of each target annotation box is determined, and a first information file of the target image data set is generated based on the first coordinate point and the category identification set.

[0051] In the embodiment of the present invention, the initialization module is specifically used to:

[0052] A model to be tested is obtained from a model database based on a preset script, and the model to be tested is initialized to obtain an initial model; the model database includes at least one untested model.

[0053] In the embodiment of the present invention, the test module is specifically used for:

[0054] Using the initial model to perform target recognition on the test data set to obtain at least one target recognition frame;

[0055] Determine a second coordinate point of the at least one target recognition frame, and generate a second information file based on the second coordinate point;

[0056] Based on the first information file and the second information file in the test data set, the intersection ratio of the coordinate points of each target image data is calculated to obtain a test result.

[0057] In the embodiment of the present invention, the analysis module is specifically used for:

[0058] Calculate, according to the test results, a first average accuracy, a first precision and a first recall rate of each category of targets in the test data set at each confidence level, as well as a second average accuracy, a second precision and a second recall rate of the initial model;

[0059] Determining the detection speed and computing resource consumption of the initial model in the test;

[0060] Calculating a weighted score of the initial model using the second average correct rate, the second precision rate, the second recall rate, the detection speed, and the computing resource consumption;

[0061] The first average accuracy, the first precision, the first recall and the weighted score are taken as analysis results.

[0062] In the embodiment of the present invention, the iteration module is specifically used for:

[0063] If the first average accuracy in the analysis result does not exceed the average accuracy threshold, and / or the first precision does not exceed the precision threshold, and / or the first recall does not exceed the recall threshold, and / or the weighted score does not exceed the weighted score threshold, then the analysis result does not meet the preset requirement;

[0064] The initial model is iterated to obtain an iterated target model.

[0065] In an embodiment of the present invention, it also includes:

[0066] a labeling module, configured to label a first labeling box in a target image dataset using a first information file in the test dataset, and to label a second labeling box in the target image dataset using a second information file in the test result, so as to obtain a labeled target image dataset;

[0067] A display module is used to display the labeled target image data set.

[0068] Accordingly, an embodiment of the present invention discloses an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the various steps of the above-mentioned model detection method embodiment when executed by the processor.

[0069] Accordingly, an embodiment of the present invention discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each step of the above-mentioned model detection method embodiment is implemented.

[0070] The embodiments of the present invention include the following advantages:

[0071] Construct a test data set for testing the model; initialize the model to be tested to obtain an initial model; test the initial model based on the test data set to obtain test results; analyze the test results to obtain analysis results; if the analysis results meet the preset requirements, generate a test report; if the analysis results do not meet the preset requirements, iterate the initial model to obtain the iterated target model, use the target model as the initial model, jump to the test of the initial model based on the test data set to obtain the test results. Through the above method, manual operations are reduced and test efficiency is improved; batch testing capabilities are achieved, allowing large amounts of image data to be processed simultaneously; key indicators are automatically counted, human errors are reduced, and accuracy is improved; standardized test result recording and analysis facilitates the preservation and further analysis of results; automatic parameter tuning based on test results is achieved, simplifying the model iteration process and model comparison. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 is a flow chart of steps of an embodiment of a model detection method of the present invention;

[0073] Figure 2 is an architectural diagram of the automated testing system of the present invention;

[0074] Figure 3 It is a structural block diagram of an embodiment of a model detection device of the present invention. DETAILED DESCRIPTION

[0075] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0076] Reference Figure 1 , shows a flow chart of steps of a model detection method embodiment of the present invention, which can be applied to a server for testing a model, wherein the functions of the model include but are not limited to multi-target detection. The server can be deployed in an automated testing system, referring to Figure 2 , shows the architecture diagram of the automated test system, which includes high-performance servers and database servers, and realizes high-speed data exchange through internal network connection. The software architecture consists of back-end services and database services written in Python. The back-end services are responsible for data processing, algorithm calls and result analysis, while the database services are used to store test data sets and results, which include:

[0077] 1) Data processing module: This module mainly uses Python's Numpy and Pandas libraries for efficient data processing, including data preprocessing, data loading, and data enhancement.

[0078] 2) Model calling module, which mainly integrates deep learning frameworks such as TensorFlow or PyTorch, and calls large model target detection algorithms, including loading pre-trained or customized large models, model inference and model update functions.

[0079] 3) Test execution module: This module is mainly responsible for designing automated test scripts and controlling the test process, including writing automated test scripts, defining test processes and parameters, and controlling the execution of the test process (data loading, model calling, result recording, and other steps). It can handle abnormal situations that may occur during the test and ensure the continuity and stability of the test.

[0080] 4) Result recording model: This module mainly records the test results and saves them in JSON format, verifies the correctness and consistency of the results, ensures the reliability of the test results, analyzes the saved test results, and extracts key indicators.

[0081] 5) Parameter management module: This module mainly manages the parameter configuration during the test process, supports parameter tuning, and records the parameter configuration of each test to facilitate subsequent analysis and comparison.

[0082] The method may specifically include the following steps:

[0083] Step 101: construct a test data set for testing the model.

[0084] Before testing the model, a test data set for testing the model may be constructed. The test data set may include at least one test data and an information file corresponding to each test data (referred to as a "first information file" for easy distinction).

[0085] Among them, since the function of the model includes multi-target detection, the test data can be image data. Further, the first information file can be a JSON file. Of course, the first information file can also be a file in other formats. In practical applications, the specific format of the first information file can be set according to actual needs, and the embodiment of the present invention does not limit this.

[0086] In an embodiment of the present invention, the step of constructing a test data set for testing a model includes:

[0087] For the acquired original image data set, preprocessing the original image data set to obtain a target image data set; the original image data set includes at least one original image data, and the target image data set includes at least one target image data;

[0088] Performing target recognition on the target image data set to obtain a candidate annotation box set and a category identification set corresponding to the target image data set; the category identification includes an identification of at least one category of the target;

[0089] Filtering and supplementing the candidate annotation frame set to obtain a target annotation frame set; the target annotation frame set includes at least one target annotation frame;

[0090] The first coordinate point of each target annotation box is determined, and a first information file of the target image data set is generated based on the first coordinate point and the category identification set.

[0091] Specifically, when constructing a test data set, a large-scale, diverse image data set (referred to as a "raw image data set") may be collected and organized, and the raw image data set includes but is not limited to multiple raw image data under different environments, lighting, occlusion, and other conditions. The raw image data set may be obtained manually, automatically, or in combination with the two or in other ways. In actual applications, the method for obtaining the raw image data set may be set according to actual needs, and the embodiments of the present invention do not limit this.

[0092] After obtaining the original image data set, each original image data in the original image data set can be preprocessed. The preprocessing includes but is not limited to image scaling and normalization, so as to obtain the target image data corresponding to each original image data. The set of each target image data is the target image data set.

[0093] Then, a deep learning-based target detection model is used to perform target recognition on each target image data in the target image dataset to obtain the annotation box corresponding to each target image data (denoted as "candidate annotation box") and the category identification of the target. The set of candidate annotation boxes is the candidate annotation box set, and the set of category identifications is the category identification set.

[0094] The candidate annotation box set is filtered and supplemented to ensure that the annotation box and category identifier of each target image data are correct, thereby obtaining a target annotation box set. Each target image data may correspond to at least one annotation box and category identifier. For example, if the target image data A includes a kitten and a puppy, then the target image data A corresponds to two target annotation boxes and two category identifiers.

[0095] After obtaining the target annotation frame set, the coordinate point (referred to as “first coordinate point”) of each target annotation frame in the target annotation frame set can be determined, and each first coordinate point and category identifier can be recorded in the first information file according to the identifier of the target image data.

[0096] Step 102, initializing the model to be tested to obtain an initial model.

[0097] After the test data set is constructed, the model to be tested can be called and initialized to obtain an initial model. The model to be tested can be one or more. In practical applications, the number of models to be tested can be set according to actual needs, and the embodiment of the present invention does not limit this.

[0098] In the embodiment of the present invention, the step of initializing the model to be tested to obtain an initial model includes:

[0099] A model to be tested is obtained from a model database based on a preset script, and the model to be tested is initialized to obtain an initial model; the model database includes at least one untested model.

[0100] Specifically, a pre-set script may be used to obtain a model to be tested from a model database, and then the model to be tested is initialized to obtain an initial model. The model database is used to store at least one model that needs to be tested and has not been tested.

[0101] To ensure that the script can effectively call the model for target detection and conduct subsequent performance evaluation and optimization, the script framework design can adopt a modular approach to encapsulate different functions such as data processing, model calling, result recording and parameter management into independent modules. The content and implementation methods of the specific modules are as follows:

[0102] 1) The data processing module mainly uses Python's NumPy and Pandas libraries for efficient data processing. This module mainly includes data preprocessing, data loading and data enhancement;

[0103] 2) The model calling module mainly integrates deep learning frameworks such as TensorFlow or PyTorch to call the target algorithm. This module mainly includes model loading, model reasoning, and model updating;

[0104] 3) The result recording module mainly records the test results and saves the results in JSON format. This module mainly includes result saving and automatic calculation of target detection indicators, such as recall rate, precision rate, etc.;

[0105] 4) The parameter management module is mainly responsible for managing parameter configuration during the test process and supports parameter tuning using hyperparameter optimization technology. This module is mainly responsible for parameter configuration, parameter tuning and parameter recording.

[0106] The framework script is configurable. In the command line interface, it allows you to enter parameters and start tests. You can set the behavior of the script through command line parameters, such as selecting different data sets, different models, adjusting input parameters, etc.

[0107] Furthermore, the iteration mechanism can be implemented by cyclically calling the above modules until satisfactory performance is achieved or the limit on the number of iterations is reached.

[0108] Step 103: Test the initial model based on the test data set to obtain a test result.

[0109] After determining the test data set and the initial model, the initial model can be used to perform target recognition on each target image data in the test data set to obtain a recognition result, and the test result of the model to be tested can be determined based on the recognition result.

[0110] In the embodiment of the present invention, the testing of the initial model based on the test data set to obtain the test result includes:

[0111] Using the initial model to perform target recognition on the test data set to obtain at least one target recognition frame;

[0112] Determine a second coordinate point of the at least one target recognition frame, and generate a second information file based on the second coordinate point;

[0113] Based on the first information file and the second information file in the test data set, the intersection ratio of the coordinate points of each target image data is calculated to obtain a test result.

[0114] Specifically, the initial model is used to perform target recognition on each target image data in the test data set to obtain at least one recognition box corresponding to each target image data (recorded as a "target recognition box"), and then the coordinate points of each target recognition box corresponding to each target image data are determined (recorded as "second coordinate points"), and all second coordinate points are recorded in an information file (recorded as a "second information file").

[0115] According to each target image data identifier, the intersection over union (IOU) of each first coordinate point in the first information file and each second coordinate point in the second information file is calculated to obtain a test result.

[0116] Among them, the intersection ratio of coordinate points = the intersection area of ​​the two labeled boxes / the union area of ​​the two labeled boxes. If the IOU exceeds the threshold (for example, 0.5), it means that the target recognition is a positive detection. If the IOU does not exceed the threshold, it means that the target recognition is a missed detection. If the category of the identified target is wrong, it means that the target identification is a false detection. Of course, in practical applications, the specific threshold of IOU can be adjusted according to actual needs, and the embodiment of the present invention does not limit this.

[0117] It should be noted that in addition to recording the second coordinate point, the second information file can also record the identification, category identification, confidence level, etc. of the target image data. In actual applications, the specific content of the second information file can be set according to actual needs, and the embodiment of the present invention does not limit this.

[0118] Step 104: Analyze the test results to obtain analysis results.

[0119] After obtaining the test results, the test results can be statistically analyzed to obtain the analysis results of the initial model. The analysis results can be visualized to facilitate users to observe various indicators of the analysis results.

[0120] In the embodiment of the present invention, the step of analyzing the test result to obtain the analysis result includes:

[0121] Calculate, according to the test results, a first average accuracy, a first precision and a first recall rate of each category of targets in the test data set at each confidence level, as well as a second average accuracy, a second precision and a second recall rate of the initial model;

[0122] Determining the detection speed and computing resource consumption of the initial model in the test;

[0123] Calculating a weighted score of the initial model using the second average correct rate, the second precision rate, the second recall rate, the detection speed, and the computing resource consumption;

[0124] The first average accuracy, the first precision, the first recall and the weighted score are taken as analysis results.

[0125] Specifically, for all target image data, since the categories of recognized targets can have multiple confidence levels, the average accuracy (AP, denoted as “first average accuracy”), precision (Precision, denoted as “first precision”) and recall (Recall, denoted as “first recall”) of each category of targets at each confidence level can be calculated based on the test results, as well as the average accuracy (denoted as “second average accuracy”), precision (denoted as “second precision”) and recall (denoted as “second recall”) of the initial model during the entire test process.

[0126] Furthermore, since the number of samples in each category is unbalanced, the second average accuracy, second precision and second recall of the initial model can be calculated by micro-average. The calculation method is as follows:

[0127] Average accuracy:

[0128] Recall:

[0129] Average accuracy:

[0130] Among them, TP c refers to the true positive examples of category C, FP c is a false positive, FN c is a false negative example, and N is the total number of categories.

[0131] Then, the detection speed and computing resource consumption of the initial model during the entire testing process are determined by statistical methods.

[0132] The weighted score of the initial model is calculated using the second average accuracy, second precision, second recall, detection speed, and computing resource consumption. The calculation method is as follows:

[0133]

[0134] where w 1 to w 5 is the corresponding weight coefficient, w 1 +W 2 +W 3 +w 4 +W 5=1; P refers to the Micro-average Precision of the model; R refers to the Micro-average Recallx; AP refers to the Micro-average Accuracy; Speed ​​is the average detection speed (ms); Cost is the computing resource consumption (GPU hours).

[0135] For each model in the model database, the above method can be used to comprehensively evaluate the performance of different models. 1 to w 5 When the corresponding weight coefficients are the same, the model with a higher Weighted Score value represents a better performance.

[0136] After the weighted score is obtained, the first average accuracy, the first precision, the first recall rate and the weighted score can be used as the analysis result. Of course, the analysis result can also include the second average accuracy, the second precision, the second recall rate, the detection speed and the computing resource consumption. In practical applications, the specific indicators of the analysis result can be set according to actual needs, and the embodiment of the present invention does not limit this.

[0137] Step 105, if the analysis result meets the preset requirements, a test report is generated; if the analysis result does not meet the preset requirements, the initial model is iterated to obtain the iterated target model, the target model is used as the initial model, and the process jumps to the test data set to test the initial model and obtain the test results.

[0138] After obtaining the analysis results, if the analysis results meet the preset requirements, it means that the performance of the initial model has reached the standard, so a test report can be generated based on the analysis results; if the analysis results do not meet the prediction requirements, it means that the performance of the initial model has not reached the standard and iterative optimization is required, so the initial model can be iterated to obtain the iterated target model, and then the target model is used as the initial model for the next round of testing, that is, jump to step 103.

[0139] In the embodiment of the present invention, if the analysis result does not meet the preset requirement, iterating the initial model to obtain an iterated target model includes:

[0140] If the first average accuracy in the analysis result does not exceed the average accuracy threshold, and / or the first precision does not exceed the precision threshold, and / or the first recall does not exceed the recall threshold, and / or the weighted score does not exceed the weighted score threshold, then the analysis result does not meet the preset requirement;

[0141] The initial model is iterated to obtain an iterated target model.

[0142] Specifically, the analysis results may include a first average accuracy, a first precision, a first recall rate and a weighted score. If at least one item does not reach the threshold, that is, the first average accuracy in the analysis result does not exceed the average accuracy threshold, and / or the first precision does not exceed the precision threshold, and / or the first recall rate does not exceed the recall rate threshold, and / or the weighted score does not exceed the weighted score threshold, then it can be determined that the analysis results do not meet the preset requirements, and hyperparameter optimization technology can be used to adjust the algorithm parameters of the initial model to achieve iterative optimization of the model, thereby obtaining the target model after iteration.

[0143] Among them, the hyperparameter optimization technology includes but is not limited to grid search, random search, and Bayesian optimization. Of course, in addition to using the hyperparameter optimization technology, other technologies can also be used for iterative optimization. In practical applications, the technology used for iterative optimization can be adjusted according to actual needs, and the embodiment of the present invention does not limit this.

[0144] In an embodiment of the present invention, it also includes:

[0145] Using the first information file in the test data set to mark a first marking box in the target image data set, and using the second information file in the test result to mark a second marking box in the target image data set, to obtain a marked target image data set;

[0146] The labeled target image dataset is displayed.

[0147] Specifically, after the test is completed, the first coordinate points in the first information file can be used to mark the corresponding annotation boxes (recorded as "first annotation boxes") in each target image data. At the same time, the second coordinate points in the second information file can be used to mark the corresponding annotation boxes (recorded as "second annotation boxes") in each target image data, so as to obtain each target image data after annotation. The set of target image data is the target image data set. For easy distinction, the display forms of each first annotation box and each second annotation box can be different. For example, the color of the first annotation box is red, and the color of the second annotation box is green. After marking each first annotation box and each second annotation box, the annotated target image data, that is, the annotated target image data set, can be displayed.

[0148] In an embodiment of the present invention, a test data set for testing a model is constructed; the model to be tested is initialized to obtain an initial model; the initial model is tested based on the test data set to obtain a test result; the test result is analyzed to obtain an analysis result; if the analysis result meets the preset requirements, a test report is generated; if the analysis result does not meet the preset requirements, the initial model is iterated to obtain an iterated target model, the target model is used as the initial model, and the process jumps to the test data set to test the initial model to obtain a test result. Through the above-mentioned method, manual operations are reduced and test efficiency is improved; batch testing capabilities are achieved, allowing a large amount of image data to be processed simultaneously; key indicators are automatically counted, human errors are reduced, and accuracy is improved; standardized test result recording and analysis facilitates the preservation and further analysis of results; automatic parameter tuning based on test results is achieved, simplifying the model iteration process and model comparison.

[0149] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0150] Reference Figure 3 , shows a structural block diagram of an embodiment of a model detection device of the present invention, which may specifically include the following modules:

[0151] A construction module 301 is used to construct a test data set for testing the model;

[0152] An initialization module 302 is used to initialize the model to be tested to obtain an initial model;

[0153] A testing module 303 is used to test the initial model based on the test data set to obtain a test result;

[0154] The analysis module 304 is used to analyze the test results to obtain analysis results;

[0155] A generating module 305 is used to generate a test report if the analysis result meets the preset requirements;

[0156] The iteration module 306 is used to iterate the initial model if the analysis result does not meet the preset requirement, obtain the iterated target model, use the target model as the initial model, and call the test module.

[0157] In the embodiment of the present invention, the building module is specifically used for:

[0158] For the acquired original image data set, preprocessing the original image data set to obtain a target image data set; the original image data set includes at least one original image data, and the target image data set includes at least one target image data;

[0159] Performing target recognition on the target image data set to obtain a candidate annotation box set and a category identification set corresponding to the target image data set; the category identification includes an identification of at least one category of the target;

[0160] Filtering and supplementing the candidate annotation frame set to obtain a target annotation frame set; the target annotation frame set includes at least one target annotation frame;

[0161] The first coordinate point of each target annotation box is determined, and a first information file of the target image data set is generated based on the first coordinate point and the category identification set.

[0162] In the embodiment of the present invention, the initialization module is specifically used to:

[0163] A model to be tested is obtained from a model database based on a preset script, and the model to be tested is initialized to obtain an initial model; the model database includes at least one untested model.

[0164] In the embodiment of the present invention, the test module is specifically used for:

[0165] Using the initial model to perform target recognition on the test data set to obtain at least one target recognition frame;

[0166] Determine a second coordinate point of the at least one target recognition frame, and generate a second information file based on the second coordinate point;

[0167] Based on the first information file and the second information file in the test data set, the intersection ratio of the coordinate points of each target image data is calculated to obtain a test result.

[0168] In the embodiment of the present invention, the analysis module is specifically used for:

[0169] Calculate, according to the test results, a first average accuracy, a first precision and a first recall rate of each category of targets in the test data set at each confidence level, as well as a second average accuracy, a second precision and a second recall rate of the initial model;

[0170] Determining the detection speed and computing resource consumption of the initial model in the test;

[0171] Calculating a weighted score of the initial model using the second average correct rate, the second precision rate, the second recall rate, the detection speed, and the computing resource consumption;

[0172] The first average accuracy, the first precision, the first recall and the weighted score are taken as analysis results.

[0173] In the embodiment of the present invention, the iteration module is specifically used for:

[0174] If the first average accuracy in the analysis result does not exceed the average accuracy threshold, and / or the first precision does not exceed the precision threshold, and / or the first recall does not exceed the recall threshold, and / or the weighted score does not exceed the weighted score threshold, then the analysis result does not meet the preset requirement;

[0175] The initial model is iterated to obtain an iterated target model.

[0176] In an embodiment of the present invention, it also includes:

[0177] a labeling module, configured to label a first labeling box in a target image dataset using a first information file in the test dataset, and to label a second labeling box in the target image dataset using a second information file in the test result, so as to obtain a labeled target image dataset;

[0178] A display module is used to display the labeled target image data set.

[0179] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0180] An embodiment of the present invention further provides an electronic device, including:

[0181] It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, each process of the above-mentioned model detection method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0182] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned model detection method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0183] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0184] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0185] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0186] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0187] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0188] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0189] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0190] The above is a detailed introduction to a model detection method and a model detection device provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A model detection method, characterized in that: The method comprises: Build a test dataset for testing the model; Initialize the model to be tested to obtain an initial model; Testing the initial model based on the test data set to obtain a test result; Analyze the test results to obtain analysis results; If the analysis result meets the preset requirements, a test report is generated; if the analysis result does not meet the preset requirements, the initial model is iterated to obtain the iterated target model, the target model is used as the initial model, and the process jumps to the test of the initial model based on the test data set to obtain the test results.

2. The model detection method according to claim 1, characterized in that: The step of constructing a test data set for testing the model includes: For the acquired original image data set, preprocessing the original image data set to obtain a target image data set; the original image data set includes at least one original image data, and the target image data set includes at least one target image data; Performing target recognition on the target image data set to obtain a candidate annotation box set and a category identification set corresponding to the target image data set; the category identification includes an identification of at least one category of the target; Filtering and supplementing the candidate annotation frame set to obtain a target annotation frame set; the target annotation frame set includes at least one target annotation frame; The first coordinate point of each target annotation box is determined, and a first information file of the target image data set is generated based on the first coordinate point and the category identification set.

3. The model detection method according to claim 1, characterized in that: The step of initializing the model to be tested to obtain an initial model includes: A model to be tested is obtained from a model database based on a preset script, and the model to be tested is initialized to obtain an initial model; the model database includes at least one untested model.

4. The model detection method according to claim 1, characterized in that: The testing of the initial model based on the test data set to obtain a test result includes: Using the initial model to perform target recognition on the test data set to obtain at least one target recognition frame; Determine a second coordinate point of the at least one target recognition frame, and generate a second information file based on the second coordinate point; Based on the first information file and the second information file in the test data set, the intersection ratio of the coordinate points of each target image data is calculated to obtain a test result.

5. The model detection method according to claim 1, characterized in that: The step of analyzing the test results to obtain analysis results includes: Calculate, according to the test results, a first average accuracy, a first precision and a first recall rate of each category of targets in the test data set at each confidence level, as well as a second average accuracy, a second precision and a second recall rate of the initial model; Determining the detection speed and computing resource consumption of the initial model in the test; Calculating a weighted score of the initial model using the second average correct rate, the second precision rate, the second recall rate, the detection speed, and the computing resource consumption; The first average accuracy, the first precision, the first recall and the weighted score are taken as analysis results.

6. The model detection method according to claim 1, characterized in that: If the analysis result does not meet the preset requirement, iterating the initial model to obtain an iterated target model includes: If the first average accuracy in the analysis result does not exceed the average accuracy threshold, and / or the first precision does not exceed the precision threshold, and / or the first recall does not exceed the recall threshold, and / or the weighted score does not exceed the weighted score threshold, then the analysis result does not meet the preset requirement; The initial model is iterated to obtain an iterated target model.

7. The model detection method according to claim 1, characterized in that: Also includes: Using the first information file in the test data set to mark a first marking box in the target image data set, and using the second information file in the test result to mark a second marking box in the target image data set, to obtain a marked target image data set; The labeled target image dataset is displayed.

8. A model detection device, characterized in that: The device comprises: A building module for building a test dataset for testing the model; The initialization module is used to initialize the model to be tested to obtain an initial model; A testing module, used to test the initial model based on the test data set to obtain a test result; An analysis module, used to analyze the test results to obtain analysis results; A generating module, used for generating a test report if the analysis result meets the preset requirements; The iteration module is used to iterate the initial model if the analysis result does not meet the preset requirement, obtain the iterated target model, use the target model as the initial model, and call the test module.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the steps of the model detection method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the model detection method according to any one of claims 1 to 7 are implemented.

Images