Evaluation method and device based on image target recognition algorithm, equipment and medium

Abstract

The invention provides an evaluation method and device based on an image target recognition algorithm, equipment and a medium. The method comprises the following steps: acquiring annotation data of a plurality of first target objects in an image test sample; synchronously using a plurality of image target recognition algorithms, performing target recognition on the image test sample under the same recognition threshold, and obtaining recognition data of a plurality of second target objects in the image test sample calculated by each image target recognition algorithm; and determining an evaluation result of each image target recognition algorithm based on the recognition data and the annotation data calculated by each image target recognition algorithm. According to the evaluation method based on the image target recognition algorithm, automatic evaluation of the performance of the image target recognition algorithm under the same recognition threshold value is achieved, the workload of testers can be greatly reduced, time and energy consumed by algorithm evaluation are greatly reduced, and the evaluation efficiency is improved. Therefore, the evaluation efficiency of the image target recognition algorithm is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of algorithm evaluation technology, and in particular to an evaluation method, device, equipment and medium based on an image target recognition algorithm. Background Art

[0002] Image recognition can be achieved through different recognition algorithms, such as using a neural network model to recognize images, or using a face tracking algorithm to recognize faces in images. The performance of the recognition algorithm will affect the image recognition effect, so it is necessary to evaluate the performance of the recognition algorithm. In addition, during the development of the recognition algorithm, different algorithms may have significant differences in the recognition effect of the target under the same recognition threshold, so in order to find a suitable algorithm, it is necessary to evaluate the service performance of as many algorithms as possible, and screen out the optimal algorithm by comparing the evaluation results.

[0003] At present, the algorithm evaluation process is carried out manually. A recognition threshold is set for the algorithm, and then the results of algorithm recognition are manually compared and counted. Testers can only test the performance of a certain algorithm under a certain recognition threshold at a time. However, since the performance of different algorithms under the same recognition threshold may vary greatly, each algorithm needs to be evaluated once. However, each test of an algorithm requires the same workload, which will generate a huge workload. Therefore, the algorithm evaluation method carried out manually requires a lot of time and energy to perform inefficient and repetitive work. Summary of the invention

[0004] The present invention provides an evaluation method, device, equipment and medium based on an image target recognition algorithm, so as to solve the problem of low efficiency of the evaluation method based on the image target recognition algorithm in the prior art.

[0005] In a first aspect, the present invention provides an evaluation method based on an image target recognition algorithm, comprising: Acquire labeling data of a plurality of first target objects in an image test sample; Synchronously using multiple image target recognition algorithms to perform target recognition on the image test sample under the same recognition threshold, and obtaining recognition data of multiple second target objects in the image test sample calculated by each image target recognition algorithm; Based on the recognition data calculated by each image target recognition algorithm and the labeled data, an evaluation result of each image target recognition algorithm is determined.

[0006] In one embodiment, the evaluation result includes an algorithm performance evaluation result; when determining the evaluation result of each image object recognition algorithm based on the recognition data calculated by each image object recognition algorithm and the annotation data, the following steps are performed for each image object recognition algorithm: According to the recognition data calculated by the image target recognition algorithm and the annotation data, a plurality of second target objects in the image test sample are compared with a plurality of first target objects in the image test sample to obtain a target existence comparison result of the image target recognition algorithm; According to the recognition data calculated by the image target recognition algorithm and the annotation data, the type of each second target object in the image test sample is compared with the type of each first target object in the image test sample to obtain a target type comparison result of the image target recognition algorithm; Calculating target overlapping areas between a plurality of second target objects in the image test sample and a plurality of first target objects in the image test sample according to the recognition data calculated by the image target recognition algorithm and the annotation data; An algorithm evaluation is performed based on the target existence comparison result, the target type comparison result and the target overlapping area to obtain an algorithm performance evaluation result of the image target recognition algorithm.

[0007] In one embodiment, the algorithm evaluation is performed based on the target existence comparison result, the target type comparison result and the target overlap area to obtain the algorithm performance evaluation result of the image target recognition algorithm, including: Determine the number of true positive detections, the number of false positive detections, the number of missed detections, the accuracy rate and the recall rate of the image target recognition algorithm based on the target existence comparison result, the target type comparison result and the target overlap area; Based on the number of true detections, the number of false detections, the number of missed detections, the accuracy rate and the recall rate, an algorithm performance evaluation result of the image object recognition algorithm is determined.

[0008] In one embodiment, the evaluation result includes a service performance evaluation result; when determining the evaluation result of each image object recognition algorithm based on the recognition data calculated by each image object recognition algorithm, the following steps are performed for each image object recognition algorithm: Based on the recognition data calculated by the image object recognition algorithm, determine the average service time, average inference time and number of images processed per second of the image object recognition algorithm; Based on the average service time, the average inference time and the number of images processed per second, the service performance evaluation result of the image object recognition algorithm is determined.

[0009] In one embodiment, after determining the evaluation result of each image object recognition algorithm based on the recognition data calculated by each image object recognition algorithm and the annotation data, the method includes: Determine the optimal image target recognition algorithm based on the evaluation results of each image target recognition algorithm; Based on the evaluation results of each image target recognition algorithm, provide a performance improvement strategy for each image target recognition algorithm; Based on the evaluation results of each image object recognition algorithm, a service performance improvement strategy for each image object recognition algorithm is provided.

[0010] In one embodiment, the algorithm performance improvement strategy includes at least a labeled data set optimization strategy, a data labeling optimization strategy, and a first recognition algorithm optimization strategy.

[0011] In one embodiment, the service performance improvement strategy includes at least a resource scheduling optimization strategy and a second identification algorithm optimization strategy.

[0012] In a second aspect, the present invention further provides an evaluation device based on an image target recognition algorithm, comprising: An acquisition module, used to acquire labeling data of a plurality of first target objects in an image test sample; An image target recognition module, used to synchronously use a plurality of image target recognition algorithms to perform target recognition on the image test sample under the same recognition threshold, and obtain recognition data of a plurality of second target objects in the image test sample calculated by each image target recognition algorithm; The evaluation result determination module is used to determine the evaluation result of each image target recognition algorithm based on the recognition data calculated by each image target recognition algorithm and the annotation data.

[0013] In a third aspect, the present invention provides a device, comprising an electronic device, wherein the electronic device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the evaluation method based on the image target recognition algorithm as described above are implemented.

[0014] In a fourth aspect, the present invention further provides a medium, comprising a non-transitory computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the evaluation method based on the image target recognition algorithm as described in any one of the above are implemented.

[0015] The evaluation method, device, equipment and medium based on the image target recognition algorithm provided by the present invention obtain the labeled data in the image test sample, further use multiple image target recognition algorithms simultaneously, perform target recognition on the image test sample under the same recognition threshold, and obtain the recognition data calculated by each image target recognition algorithm to test the algorithm performance under the same recognition threshold, and further determine the evaluation result of each image target recognition algorithm based on the recognition data calculated by each image target recognition algorithm and the original labeled data, so as to realize the automatic evaluation of the image target recognition algorithm performance under the same recognition threshold, which can greatly reduce the workload of testers and greatly reduce the time and energy required for algorithm evaluation, thereby effectively improving the efficiency of image target recognition algorithm evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is one of the flow charts of the evaluation method based on the image target recognition algorithm provided by the present invention.

[0018] Figure 2 This is the second flow chart of the evaluation method based on the image target recognition algorithm provided by the present invention.

[0019] Figure 3 It is a structural schematic diagram of the evaluation device based on the image target recognition algorithm provided by the present invention.

[0020] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] The terms "first", "second", etc. in the present invention are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention can be implemented in an order other than those illustrated or described herein.

[0023] Combine the following Figure 1-Figure 4 The present invention describes the evaluation method, device, equipment and medium based on the image target recognition algorithm.

[0024] Combination Figure 1 and Figure 2 , Figure 1 This is one of the flow charts of the evaluation method based on the image target recognition algorithm provided by the present invention. Figure 2 This is the second flow chart of the evaluation method based on the image target recognition algorithm provided by the present invention.

[0025] like Figure 1 As shown, the method includes the following: Step 101: Obtaining labeling data of a plurality of first target objects in an image test sample; Step 102: Synchronously use multiple image target recognition algorithms to perform target recognition on the image test sample under the same recognition threshold, and obtain recognition data of multiple second target objects in the image test sample calculated by each image target recognition algorithm; Step 103: Determine the evaluation result of each image object recognition algorithm based on the recognition data calculated by each image object recognition algorithm and the labeled data.

[0026] It should be noted that the evaluation method based on the image target recognition algorithm provided in the embodiment of the present invention is implemented by an evaluation device based on the image target recognition algorithm. Figure 2 As shown, the evaluation method based on the image target recognition algorithm provided by the embodiment of the present invention is used to test the algorithm performance under the same recognition threshold. The labeled data is generated by annotating the image test samples, and the image test samples are recognized by adopting multiple image target recognition algorithms to generate recognition data under the same recognition threshold. Finally, the labeled data and the recognition data are compared in multiple aspects such as whether the detection target exists, whether the detection target type is the same, and whether the detection target overlapping area meets the requirements. The evaluation result of the algorithm recognition under the same recognition threshold is obtained to construct an automated testing tool for the performance of the target recognition algorithm under the same recognition threshold. The workload of the testers can be greatly reduced. When the algorithm is iterated quickly, a large amount of time can be avoided from being spent on inefficient repetitive work.

[0027] The embodiment of the present invention describes the evaluation method based on the image target recognition algorithm by taking the evaluation device based on the image target recognition algorithm as an example of the execution body. The evaluation device based on the image target recognition algorithm is referred to as the evaluation device for short.

[0028] Specifically, the evaluation device obtains image test samples for the recognition task.

[0029] Furthermore, the tester uses the evaluation device to annotate multiple first target objects in the image test sample to obtain annotated data, wherein the annotated data includes but is not limited to the target frame information and type of each first target object, and the target frame information can determine the location and area occupied by the corresponding first target object.

[0030] It should be noted that the image test sample only needs to be labeled once. Even if the image target recognition algorithm is iteratively upgraded later, there is no need to label it here. The labeled data of the image test sample can be used for each round of image target recognition algorithm evaluation.

[0031] Furthermore, the evaluation device selects a variety of image target recognition algorithms, such as convolutional neural networks, real-time target detection algorithms, target tracking algorithms, etc., and combines the multiple image target recognition algorithms with the labeled data of the image test samples to create a new evaluation task.

[0032] Furthermore, the evaluation device sets the recognition thresholds of the various image object recognition algorithms to be the same, wherein the recognition threshold is an Intersection over Union (IOU) threshold.

[0033] It should be noted that IOU is the ratio of the intersection and union of the areas of the target's true annotation box and the predicted box. The IOU threshold is an indicator used to measure the accuracy of target detection or segmentation results. The IOU threshold can be used to decide when to accept or reject a predicted box. If the IOU value of the predicted box is greater than the set threshold, the prediction is considered correct. Conversely, if the IOU value is lower than the threshold, it may be considered an incorrect prediction or an unreliable detection result.

[0034] Therefore, by setting the recognition thresholds of various image target recognition algorithms to the same, it can be considered that the prediction benchmarks of various image target recognition algorithms are consistent. Then, the evaluation of various image target recognition algorithms under the same recognition threshold is comparable and fair to a certain extent.

[0035] Furthermore, after setting the same recognition threshold, the evaluation device starts the evaluation task.

[0036] Furthermore, the evaluation device uses asynchronous multi-threading technology to synchronously use multiple image target recognition algorithms to perform target recognition on image test samples under the same recognition threshold, and obtains recognition data of multiple second target objects in the image test samples calculated by each image target recognition algorithm, wherein the recognition data includes but is not limited to the target frame information and type of each second target object, and the target frame information can determine the location and area occupied by the corresponding second target object.

[0037] It should be noted that the first target object is obtained by manually annotating the image test sample, and the second target object is identified by an image object recognition algorithm.

[0038] Furthermore, the evaluation device performs data cleaning on the annotated data of the image test sample and the recognition data calculated by each image target recognition algorithm to obtain the recognition data and corresponding annotated data under the same recognition threshold in a standardized format.

[0039] Furthermore, the evaluation device determines the evaluation results of each image target recognition algorithm based on the recognition data and annotation data calculated by each image target recognition algorithm, wherein the evaluation results include algorithm performance evaluation results and service performance evaluation results.

[0040] Therefore, it can be understood that the evaluation device compares the recognition data of the image test samples calculated by each image target recognition algorithm with the labeled data of the image test samples to obtain the algorithm performance evaluation results of each image target recognition algorithm.

[0041] Furthermore, the evaluation device determines the service performance evaluation result of each image target recognition algorithm based on the recognition data of the image test samples calculated by each image target recognition algorithm.

[0042] The evaluation method based on image target recognition algorithm provided by the present invention obtains labeled data in image test samples, further uses multiple image target recognition algorithms simultaneously, performs target recognition on image test samples under the same recognition threshold, obtains recognition data calculated by each image target recognition algorithm, so as to test the algorithm performance under the same recognition threshold, and further determines the evaluation result of each image target recognition algorithm based on the recognition data calculated by each image target recognition algorithm and the original labeled data, thereby realizing automatic evaluation of the performance of image target recognition algorithms under the same recognition threshold, which can greatly reduce the workload of testers and greatly reduce the time and energy required for algorithm evaluation, thereby effectively improving the efficiency of image target recognition algorithm evaluation.

[0043] Further, based on step 103, when determining the evaluation result of each image object recognition algorithm based on the recognition data calculated by each image object recognition algorithm and the labeled data, the following steps are performed for each image object recognition algorithm: According to the recognition data calculated by the image target recognition algorithm and the annotation data, a plurality of second target objects in the image test sample are compared with a plurality of first target objects in the image test sample to obtain a target existence comparison result of the image target recognition algorithm; According to the recognition data calculated by the image target recognition algorithm and the annotation data, the type of each second target object in the image test sample is compared with the type of each first target object in the image test sample to obtain a target type comparison result of the image target recognition algorithm; Calculating target overlapping areas between a plurality of second target objects in the image test sample and a plurality of first target objects in the image test sample according to the recognition data calculated by the image target recognition algorithm and the annotation data; An algorithm evaluation is performed based on the target existence comparison result, the target type comparison result and the target overlapping area to obtain an algorithm performance evaluation result of the image target recognition algorithm.

[0044] Specifically, the evaluation device compares the multiple second target objects with the multiple first target objects originally marked in the image test sample based on the multiple second target objects in the image test sample calculated by the image target recognition algorithm, that is, confirms whether the target objects exist the same, and obtains the target existence comparison result of the image target recognition algorithm.

[0045] Furthermore, the evaluation device compares the type of each second target object in the image test sample calculated by the image target recognition algorithm and the type of each first target object originally marked in the image test sample, that is, confirms whether the types of the target objects are the same, and obtains the target type comparison result of the image target recognition algorithm.

[0046] Furthermore, the evaluation device calculates the target overlapping area between multiple second target objects and multiple first target objects based on the target frame information of each second target object in the image test sample calculated by the image target recognition algorithm and the target frame information of each first target object originally marked in the image test sample, that is, it calculates the target overlapping area between the second target object and the first target object when the same target object and the same target object type are identified.

[0047] Furthermore, the evaluation device performs algorithm evaluation based on the target existence comparison result, the target type comparison result and the target overlapping area to obtain an algorithm performance evaluation result of the image target recognition algorithm.

[0048] The embodiment of the present invention analyzes the target existence comparison results, target type comparison results and target overlapping area by comparing the recognition data and the annotation data, so as to realize the comprehensive performance evaluation of the image target recognition algorithm, thereby improving the accuracy of the evaluation of the image target recognition algorithm.

[0049] Furthermore, the algorithm evaluation is performed based on the target existence comparison result, the target type comparison result and the target overlap area to obtain the algorithm performance evaluation result of the image target recognition algorithm, including: Determine the number of true positive detections, the number of false positive detections, the number of missed detections, the accuracy rate and the recall rate of the image target recognition algorithm based on the target existence comparison result, the target type comparison result and the target overlap area; Based on the number of true detections, the number of false detections, the number of missed detections, the accuracy rate and the recall rate, an algorithm performance evaluation result of the image object recognition algorithm is determined.

[0050] Specifically, the evaluation device determines the number of true detections, the number of false detections, the number of missed detections, the accuracy rate and the recall rate of the image target recognition algorithm based on the target existence comparison results, the target type comparison results and the target overlapping area.

[0051] It should be noted that the number of true detections refers to the number of targets correctly detected by the algorithm; the number of false detections refers to the number of non-targets mistakenly identified as targets by the algorithm; the number of missed detections refers to the number of targets that the algorithm fails to detect; the accuracy rate refers to the proportion of targets correctly identified; and the recall rate refers to the proportion of targets that are correctly identified.

[0052] Furthermore, the evaluation device comprehensively analyzes the algorithm performance of the image target recognition algorithm based on the number of true detections, the number of false detections, the number of missed detections, the accuracy rate and the recall rate to determine the algorithm performance evaluation result of the image target recognition algorithm.

[0053] The embodiment of the present invention determines the image target recognition algorithm's multiple performance indicators, such as the number of true detections, the number of false detections, the number of missed detections, the accuracy rate, and the recall rate, through recognition data. Through the multiple performance indicators, the accuracy and completeness of the algorithm in the target recognition task are comprehensively evaluated, thereby obtaining a comprehensive evaluation result of the algorithm's performance, which effectively improves the accuracy of the evaluation of the image target recognition algorithm.

[0054] Further, based on step 103, when determining the evaluation result of each image object recognition algorithm based on the recognition data calculated by each image object recognition algorithm, the following steps are performed for each image object recognition algorithm: Based on the recognition data calculated by the image object recognition algorithm, determine the average service time, average inference time and number of images processed per second of the image object recognition algorithm; Based on the average service time, the average inference time and the number of images processed per second, the service performance evaluation result of the image object recognition algorithm is determined.

[0055] Specifically, the evaluation device determines the average service time, average inference time and number of images processed per second of the image target recognition algorithm based on the recognition data of the image test samples calculated by the image target recognition algorithm.

[0056] It should be noted that the average service time refers to the average time required to complete a target recognition task, usually including the time consumed in the entire process from request to response; the average inference time refers to the average time required for the algorithm to perform a single inference (i.e., single image processing); the number of images processed per second refers to the number of images that the algorithm can process per unit time.

[0057] Furthermore, the evaluation device comprehensively analyzes the service performance of the image target recognition algorithm based on the average service time, the average reasoning time and the number of images processed per second to determine the service performance evaluation result of the image target recognition algorithm.

[0058] The embodiment of the present invention determines the service performance indicators of the image target recognition algorithm, such as the average service time, the average inference time, and the number of images processed per second, by identifying data. Through the various service performance indicators, the algorithm's performance in processing speed and efficiency is comprehensively evaluated, thereby obtaining a comprehensive evaluation result of its service performance, which effectively improves the accuracy of the evaluation of the image target recognition algorithm.

[0059] Further, after determining the evaluation result of each image object recognition algorithm based on the recognition data calculated by each image object recognition algorithm and the labeled data, it includes: Determine the optimal image target recognition algorithm based on the evaluation results of each image target recognition algorithm; Based on the evaluation results of each image target recognition algorithm, provide a performance improvement strategy for each image target recognition algorithm; Based on the evaluation results of each image object recognition algorithm, a service performance improvement strategy for each image object recognition algorithm is provided.

[0060] Specifically, the evaluation device determines the optimal image target recognition algorithm based on the algorithm performance evaluation results and service performance evaluation results of each image target recognition algorithm and in accordance with the preset optimal algorithm evaluation principles, wherein the preset optimal algorithm evaluation principles are set according to actual conditions and can be evaluated by relevant algorithm experts or by a certain algorithm.

[0061] Furthermore, the evaluation device provides an algorithm performance improvement strategy for each image target recognition algorithm based on the algorithm performance evaluation results of each image target recognition algorithm, wherein the algorithm performance improvement strategy includes but is not limited to a labeled data set optimization strategy, a data labeling optimization strategy and a first recognition algorithm optimization strategy.

[0062] In one embodiment, the labeling data set optimization strategy may include data cleaning and deduplication, data enhancement, category balancing, and data labeling quality inspection.

[0063] Data cleaning and deduplication: Thoroughly clean the labeled data set to remove outliers, noise data, and duplicate samples. Use algorithms or manual inspections to ensure the purity and consistency of the data set.

[0064] Data augmentation: Apply various data augmentation techniques (such as rotation, scaling, cropping, color transformation, etc.) to increase the diversity of training samples, thereby improving the generalization ability of the model. Use advanced techniques such as generative adversarial networks to generate synthetic data that is close to the distribution of real data to further expand the dataset.

[0065] Class balance: Check the number of samples in each category in the data set to ensure that the number of samples in different categories is relatively balanced. For categories with a small number, oversampling or data synthesis techniques can be used to increase their sample size.

[0066] Data labeling quality check: Establish a strict data labeling quality check process, including automatic inspection and manual review. Use automated tools (such as label consistency check tools) to assist in discovering labeling errors, and invite experts to conduct manual review and correction.

[0067] In one embodiment, the data annotation optimization strategy may include methods such as annotation specification formulation, annotation tool optimization, and annotation process optimization.

[0068] Labeling specification formulation: formulate detailed labeling specifications, clarify the labeling standards and requirements for each category and attribute. Provide training for labelers to ensure that they fully understand the labeling specifications and can accurately implement them.

[0069] Optimization of annotation tools: Use advanced annotation tools to support efficient annotation operations, batch processing, and automated auxiliary functions (such as pre-annotation, automatic error correction, etc.). Customize and optimize annotation tools according to the characteristics and requirements of annotation tasks to improve annotation efficiency and accuracy.

[0070] Optimize the annotation process: Design a reasonable annotation process, including data preprocessing, annotation task allocation, annotation progress monitoring, annotation quality inspection, etc. Introduce incentive mechanisms and reward and punishment systems to stimulate the enthusiasm and sense of responsibility of annotation personnel and improve the quality of annotation.

[0071] In one embodiment, the first recognition algorithm optimization strategy may include methods such as algorithm selection and adjustment, model structure optimization, regularization and hyperparameter tuning, and integrated learning and model fusion.

[0072] Algorithm selection and adjustment: According to the specific needs and characteristics of the target recognition task, select the appropriate recognition algorithm (such as convolutional neural network, recurrent neural network, etc.). Adjust and optimize the algorithm parameters (such as learning rate, batch size, optimizer, etc.) to improve the algorithm's recognition performance and efficiency.

[0073] Model structure optimization: Design or adopt a more efficient model structure (such as lightweight network structure, deep separable convolution, etc.) to reduce the amount of calculation and parameters and improve the reasoning speed of the model. Introduce advanced technologies such as attention mechanism and feature pyramid to improve the model's recognition ability for complex scenes.

[0074] Regularization and hyperparameter tuning: Use regularization techniques (such as L1 regularization, L2 regularization, etc.) to limit model complexity and avoid overfitting. Use automated hyperparameter tuning tools (such as Bayesian optimization, grid search, etc.) to search for the optimal hyperparameter configuration and improve the recognition performance of the model.

[0075] Ensemble learning and model fusion: Multiple individually trained models are integrated to improve the stability and accuracy of the overall recognition performance. Model fusion technology (such as weighted average, voting mechanism, etc.) is used to fuse the prediction results of different models to obtain more accurate recognition results.

[0076] Furthermore, the evaluation device provides a service performance improvement strategy for each image target recognition algorithm based on the service performance evaluation results of each image target recognition algorithm, wherein the service performance improvement strategy includes but is not limited to a resource scheduling optimization strategy and a second recognition algorithm optimization strategy.

[0077] In one embodiment, the resource scheduling optimization strategy may include methods such as introduction of intelligent resource scheduling algorithms, distributed and heterogeneous computing, dynamic resource expansion, and resource utilization monitoring and tuning.

[0078] Introduction of intelligent resource scheduling algorithms: Introducing intelligent scheduling algorithms, such as scheduling strategies based on reinforcement learning or deep learning, to dynamically allocate computing resources. These algorithms can intelligently adjust resource allocation based on the current system load, task priority, and the performance of different computing nodes, thereby improving overall performance and efficiency.

[0079] Distributed and heterogeneous computing: Utilize distributed system architecture and heterogeneous computing platform to distribute target recognition tasks to multiple computing nodes. Reduce communication overhead and accelerate convergence speed through load balancing and gradient aggregation algorithms. For computationally intensive tasks, high-performance graphics processors or tensor processors are used for processing, while the central processing unit handles simpler tasks, so as to achieve efficient operation of the overall training process.

[0080] Dynamic resource expansion: Dynamic expansion of computing resources is achieved through cloud services. When the system load increases, computing resources can be automatically increased to meet demand; when the load decreases, excess resources are released to save costs. This elastic computing resource management mechanism can ensure that the system always maintains the best operating state.

[0081] Resource utilization monitoring and tuning: Regularly monitor the utilization and performance of computing resources, such as image processor utilization, memory usage, etc. Based on the monitoring results, adjust the training batch size, optimize the data loading method, or adjust the model structure to improve resource utilization and training efficiency.

[0082] In one embodiment, the second recognition algorithm optimization strategy may include methods such as algorithm and training strategy improvement, regularization and self-supervised learning, model structure innovation, transfer learning and incremental learning, model pruning and compression, synthetic data and data enhancement, and fairness and debiasing technology.

[0083] Improvements in algorithms and training strategies: Research energy-efficient training algorithms, such as sparse training and quantized training, to reduce computational and storage requirements. At the same time, dynamic model expansion technology is used to enable full model capacity only when necessary to save resources and increase training speed.

[0084] Regularization and self-supervised learning: Introduce regularization techniques (such as L1 regularization, L2 regularization) and self-supervised learning tasks to improve the generalization ability of the model and reduce overfitting. Through regularization Dropout, data enhancement Mixup, contrastive learning and other technologies, improve the performance of the model on unseen data.

[0085] Model structure innovation: Research new model structures, such as the sparsely activated Mixture of Experts (MoE) model, which activates only some components when needed to reduce unnecessary computing and storage burdens. At the same time, explore lightweight model design for deployment in resource-limited environments.

[0086] Transfer learning and incremental learning: Use pre-trained models for transfer learning to quickly adapt to new tasks. Or use incremental learning methods to allow the model to learn new tasks without losing existing knowledge, reducing dependence on large-scale data.

[0087] Model pruning and compression: Prune and quantize large models to remove redundant parameters or approximate them to low-precision values ​​to achieve model lightweighting. Pruning technology is used to reduce unnecessary connections and parameters in the model, and quantization technology is used to reduce the storage space and computing cost of the model.

[0088] Synthetic data and data enhancement: Use synthetic data generation technology and data enhancement strategies to expand and enrich training data sets. By increasing the diversity and complexity of data, the model's ability to recognize complex scenes is improved.

[0089] Fairness and debiasing techniques: Research algorithms and data preprocessing methods to eliminate implicit bias in the model and ensure fairness when the model processes data from various groups. Improve the fairness and unbiasedness of the model through diversified data sets and fairness indicators.

[0090] Furthermore, technical personnel can improve the algorithm performance of the image target recognition algorithm accordingly according to the algorithm performance improvement strategy of each image target recognition algorithm, and improve the service performance of the image target recognition algorithm accordingly according to the service performance improvement strategy of each image target recognition algorithm.

[0091] The embodiment of the present invention determines the optimal image target recognition algorithm through relatively accurate image target recognition algorithm evaluation results, which can significantly improve the actual effect of target recognition applications, and further provides corresponding improvement strategies in algorithm performance, which can enhance the algorithm's adaptability to complex scenarios. At the same time, it provides corresponding improvement strategies in service performance, which can improve the actual effect of the algorithm and actual application performance.

[0092] The following is a description of the evaluation device based on the image target recognition algorithm provided by the present invention. The evaluation device based on the image target recognition algorithm described below and the evaluation method based on the image target recognition algorithm described above can refer to each other.

[0093] Reference Figure 3 , Figure 3It is a structural schematic diagram of the evaluation device based on the image target recognition algorithm provided by the present invention.

[0094] The evaluation device based on the image target recognition algorithm comprises: The acquisition module 310 is used to acquire the labeling data of multiple first target objects in the image test sample.

[0095] The image target recognition module 320 is used to synchronously use multiple image target recognition algorithms to perform target recognition on the image test sample under the same recognition threshold, and obtain recognition data of multiple second target objects in the image test sample calculated by each image target recognition algorithm.

[0096] The evaluation result determination module 330 is used to determine the evaluation result of each image object recognition algorithm based on the recognition data calculated by each image object recognition algorithm and the annotation data.

[0097] The evaluation device based on image target recognition algorithm provided by the present invention obtains the labeled data in the image test sample, further uses multiple image target recognition algorithms synchronously, performs target recognition on the image test sample under the same recognition threshold, obtains the recognition data calculated by each image target recognition algorithm, so as to test the algorithm performance under the same recognition threshold, and further determines the evaluation result of each image target recognition algorithm based on the recognition data calculated by each image target recognition algorithm and the original labeled data, thereby realizing the automatic evaluation of the image target recognition algorithm performance under the same recognition threshold, which can greatly reduce the workload of testers and greatly reduce the time and energy required for algorithm evaluation, thereby effectively improving the efficiency of image target recognition algorithm evaluation.

[0098] Furthermore, the evaluation result determination module 330 is also used to: According to the recognition data calculated by the image target recognition algorithm and the annotation data, a plurality of second target objects in the image test sample are compared with a plurality of first target objects in the image test sample to obtain a target existence comparison result of the image target recognition algorithm; According to the recognition data calculated by the image target recognition algorithm and the annotation data, the type of each second target object in the image test sample is compared with the type of each first target object in the image test sample to obtain a target type comparison result of the image target recognition algorithm; Calculating target overlapping areas between a plurality of second target objects in the image test sample and a plurality of first target objects in the image test sample according to the recognition data calculated by the image target recognition algorithm and the annotation data; An algorithm evaluation is performed based on the target existence comparison result, the target type comparison result and the target overlapping area to obtain an algorithm performance evaluation result of the image target recognition algorithm.

[0099] Furthermore, the evaluation result determination module 330 is also used to: Determine the number of true positive detections, the number of false positive detections, the number of missed detections, the accuracy rate and the recall rate of the image target recognition algorithm based on the target existence comparison result, the target type comparison result and the target overlap area; Based on the number of true detections, the number of false detections, the number of missed detections, the accuracy rate and the recall rate, an algorithm performance evaluation result of the image object recognition algorithm is determined.

[0100] Furthermore, the evaluation result determination module 330 is also used to: Based on the recognition data calculated by the image object recognition algorithm, determine the average service time, average inference time and number of images processed per second of the image object recognition algorithm; Based on the average service time, the average inference time and the number of images processed per second, the service performance evaluation result of the image object recognition algorithm is determined.

[0101] Furthermore, the evaluation device based on the image target recognition algorithm is also used for: Determine the optimal image target recognition algorithm based on the evaluation results of each image target recognition algorithm; Based on the evaluation results of each image target recognition algorithm, provide a performance improvement strategy for each image target recognition algorithm; Based on the evaluation results of each image object recognition algorithm, a service performance improvement strategy for each image object recognition algorithm is provided.

[0102] Figure 4 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 4As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute an evaluation method based on an image target recognition algorithm, the method comprising: obtaining the labeling data of multiple first target objects in an image test sample; synchronously using multiple image target recognition algorithms to perform target recognition on the image test sample under the same recognition threshold, and obtaining the recognition data of multiple second target objects in the image test sample calculated by each image target recognition algorithm; and determining the evaluation result of each image target recognition algorithm based on the recognition data calculated by each image target recognition algorithm and the labeling data.

[0103] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0104] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the evaluation method based on the image target recognition algorithm provided in the above-mentioned embodiments, and the method includes: obtaining labeling data of multiple first target objects in an image test sample; synchronously using multiple image target recognition algorithms to perform target recognition on the image test sample under the same recognition threshold, and obtain recognition data of multiple second target objects in the image test sample calculated by each image target recognition algorithm; based on the recognition data calculated by each image target recognition algorithm and the labeling data, determining the evaluation result of each image target recognition algorithm.

[0105] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the evaluation method based on the image target recognition algorithm provided by the above-mentioned embodiments, the method comprising: obtaining labeling data of multiple first target objects in an image test sample; synchronously using multiple image target recognition algorithms to perform target recognition on the image test sample under the same recognition threshold, and obtaining recognition data of multiple second target objects in the image test sample calculated by each image target recognition algorithm; and determining the evaluation result of each image target recognition algorithm based on the recognition data calculated by each image target recognition algorithm and the labeling data.

[0106] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. Those of ordinary skill in the art may understand and implement the present invention without creative effort.

[0107] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An evaluation method based on an image target recognition algorithm, characterized in that: include: Acquire labeling data of a plurality of first target objects in an image test sample; Synchronously using multiple image target recognition algorithms to perform target recognition on the image test sample under the same recognition threshold, and obtaining recognition data of multiple second target objects in the image test sample calculated by each image target recognition algorithm; Based on the recognition data calculated by each image target recognition algorithm and the labeled data, an evaluation result of each image target recognition algorithm is determined.

2. The evaluation method based on the image target recognition algorithm according to claim 1 is characterized in that: The evaluation results include algorithm performance evaluation results; when determining the evaluation results of each image target recognition algorithm based on the recognition data calculated by each image target recognition algorithm and the labeled data, the following steps are performed for each image target recognition algorithm: According to the recognition data calculated by the image target recognition algorithm and the annotation data, a plurality of second target objects in the image test sample are compared with a plurality of first target objects in the image test sample to obtain a target existence comparison result of the image target recognition algorithm; According to the recognition data calculated by the image target recognition algorithm and the annotation data, the type of each second target object in the image test sample is compared with the type of each first target object in the image test sample to obtain a target type comparison result of the image target recognition algorithm; Calculating target overlapping areas between a plurality of second target objects in the image test sample and a plurality of first target objects in the image test sample according to the recognition data calculated by the image target recognition algorithm and the annotation data; An algorithm evaluation is performed based on the target existence comparison result, the target type comparison result and the target overlapping area to obtain an algorithm performance evaluation result of the image target recognition algorithm.

3. The evaluation method based on the image target recognition algorithm according to claim 2 is characterized in that: The algorithm evaluation is performed based on the target existence comparison result, the target type comparison result and the target overlap area to obtain the algorithm performance evaluation result of the image target recognition algorithm, including: Determine the number of true positive detections, the number of false positive detections, the number of missed detections, the accuracy rate and the recall rate of the image target recognition algorithm based on the target existence comparison result, the target type comparison result and the target overlap area; Based on the number of true detections, the number of false detections, the number of missed detections, the accuracy rate and the recall rate, an algorithm performance evaluation result of the image object recognition algorithm is determined.

4. The evaluation method based on the image target recognition algorithm according to claim 1 is characterized in that: The evaluation result includes a service performance evaluation result; when determining the evaluation result of each image target recognition algorithm based on the recognition data calculated by each image target recognition algorithm, the following steps are performed for each image target recognition algorithm: Based on the recognition data calculated by the image object recognition algorithm, determine the average service time, average inference time and number of images processed per second of the image object recognition algorithm; Based on the average service time, the average inference time and the number of images processed per second, the service performance evaluation result of the image object recognition algorithm is determined.

5. The evaluation method based on the image target recognition algorithm according to any one of claims 1 to 4, characterized in that: After determining the evaluation result of each image object recognition algorithm based on the recognition data calculated by each image object recognition algorithm and the labeled data, the method further comprises: Determine the optimal image target recognition algorithm based on the evaluation results of each image target recognition algorithm; Based on the evaluation results of each image target recognition algorithm, provide a performance improvement strategy for each image target recognition algorithm; Based on the evaluation results of each image object recognition algorithm, a service performance improvement strategy for each image object recognition algorithm is provided.

6. The evaluation method based on the image target recognition algorithm according to claim 5 is characterized in that: The algorithm performance improvement strategy includes at least a labeled data set optimization strategy, a data labeling optimization strategy and a first recognition algorithm optimization strategy.

7. The evaluation method based on the image target recognition algorithm according to claim 5 is characterized in that: The service performance improvement strategy at least includes a resource scheduling optimization strategy and a second identification algorithm optimization strategy.

8. An evaluation device based on an image target recognition algorithm, characterized in that: include: An acquisition module, used to acquire labeling data of a plurality of first target objects in an image test sample; An image target recognition module, used to synchronously use a plurality of image target recognition algorithms to perform target recognition on the image test sample under the same recognition threshold, and obtain recognition data of a plurality of second target objects in the image test sample calculated by each image target recognition algorithm; The evaluation result determination module is used to determine the evaluation result of each image target recognition algorithm based on the recognition data calculated by each image target recognition algorithm and the annotation data.

9. A device, comprising an electronic device, the electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the evaluation method based on the image target recognition algorithm as described in any one of claims 1 to 7 are implemented.

10. A medium, comprising a non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the evaluation method based on the image target recognition algorithm as described in any one of claims 1 to 7 are implemented.

Images