Algorithm model verification and precision evaluation method and device, equipment and storage medium

Abstract

The invention belongs to the field of artificial intelligence, finance and medical treatment, and relates to an algorithm model verification and precision evaluation method, which comprises the steps of calling a pre-constructed original algorithm model to process an original sample file set to obtain a to-be-evaluated model result, and associating an original sample file with the to-be-evaluated model result to obtain a to-be-evaluated model result; performing annotation task allocation according to the generated annotation task allocation scheme, receiving a truth value result submitted by the annotation personnel, comparing a to-be-evaluated model result with the corresponding truth value result to obtain an initial model precision evaluation result, optimizing the model according to the initial model precision evaluation result, and evaluating the optimization algorithm model to obtain an evaluation result. And generating a model performance comparison report. The invention further provides an algorithm model verification and precision evaluation device, equipment and a storage medium. In addition, the invention also relates to a block chain technology, and the original sample file can be stored in a block chain. According to the method, the model iteration efficiency can be greatly improved, and the precision and performance of the model are improved.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence, financial technology and digital medical technology, and in particular to an algorithm model verification and accuracy assessment method, device, equipment and storage medium. Background Art

[0002] With the rapid development of computer technology, more and more fields, especially the financial and medical fields, involve a wide variety of business scenarios and a large amount of file data to be processed, which is usually processed by developing corresponding algorithms. In the current field of algorithm development and application, although a variety of algorithm testing and annotation tools have emerged on the market, these tools generally face a core problem, that is, the lack of comprehensive integration and automated management of the entire life cycle of the algorithm, including development, testing, annotation and optimization. This situation directly leads to the algorithm development team, data annotation personnel and project managers having to rely on cumbersome manual operations and offline communication methods to complete various tasks, such as data file delivery, model prediction result sharing, annotation task allocation and feedback, etc. This working mode that is highly dependent on traditional communication media (such as email and instant messaging tools) not only greatly reduces the efficiency of algorithm development, but also causes confusion in file management, obstacles to team collaboration and concerns about data security.

[0003] Specifically, in the traditional algorithm model collaboration and accuracy assessment process, the business party provides the original data, the algorithm model generates the recognition results, and then the labeling assignor issues the labeling task to the labeling personnel through a non-integrated system. After the labeling is completed, the labeling results need to be manually summarized and the algorithm accuracy needs to be calculated. Finally, the algorithm is debugged and iterated. The whole process has the following problems: low file transfer efficiency; cumbersome accuracy assessment and model iteration processes, and the model accuracy cannot be quickly calculated and compared; model version management is chaotic, and it is difficult to track and trace different versions of files and model results. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose an algorithm model verification and accuracy assessment method, device, equipment and storage medium to solve the technical problems in the related technology of low file transmission efficiency, cumbersome accuracy assessment and model iteration process and difficult tracing.

[0005] In order to solve the above technical problems, the present application embodiment provides an algorithm model verification and accuracy evaluation method, which adopts the following technical solutions:

[0006] Obtaining an original sample file set, calling a pre-built original algorithm model to process the original sample file set to obtain a model result to be evaluated, and associating the original sample file with the model result to be evaluated according to a preset rule to obtain an associated sample set;

[0007] Generate a labeling task allocation plan based on the preset labeling personnel information and the associated sample set;

[0008] Obtaining a request for assigning a labeling task, and sending the labeling task to a corresponding labeler according to the labeling task assignment plan;

[0009] Receive the true value results submitted by the annotator, and associate the true value results with the corresponding original sample files and the results of the model to be evaluated to form a complete annotated sample data set;

[0010] Compare the model result to be evaluated with the corresponding true value result to obtain an initial model accuracy evaluation result;

[0011] When the initial model accuracy evaluation result is less than a preset evaluation threshold, a model optimization request is generated, and the original algorithm model is optimized according to the model optimization request to obtain an optimized algorithm model;

[0012] Calling the optimization algorithm model to process the labeled sample data set to obtain an optimized model result;

[0013] Compare the optimized model result with the corresponding true value file to obtain the optimized model accuracy evaluation result, compare the optimized model accuracy evaluation result with the initial model accuracy evaluation result, and generate a model performance comparison report;

[0014] According to the model performance comparison report, it is determined whether the optimization algorithm model meets the expected conditions. When the expected conditions are met, the optimization algorithm model is output as the final model.

[0015] In order to solve the above technical problems, the embodiment of the present application also provides an algorithm model verification and accuracy evaluation device, which adopts the following technical solution:

[0016] An acquisition module is used to acquire an original sample file set, call a pre-built original algorithm model to process the original sample file set, obtain a model result to be evaluated, and associate the original sample file with the model result to be evaluated according to a preset rule to obtain an associated sample set;

[0017] A scheme generation module, used to generate a labeling task allocation scheme according to preset labeling personnel information and the associated sample set;

[0018] An assignment module is used to obtain an assignment request for a labeling task and send the labeling task to a corresponding labeler according to the labeling task assignment plan;

[0019] An association module is used to receive the true value results submitted by the annotator, and associate the true value results with the corresponding original sample files and the model results to be evaluated to form a complete annotated sample data set;

[0020] A comparison module is used to compare the model result to be evaluated with the corresponding true value result to obtain an initial model accuracy evaluation result;

[0021] An optimization module, configured to generate a model optimization request when the initial model accuracy evaluation result is less than a preset evaluation threshold, and optimize the original algorithm model according to the model optimization request to obtain an optimized algorithm model;

[0022] A model processing module, used to call the optimization algorithm model to process the labeled sample data set to obtain an optimized model result;

[0023] A report generation module, used to compare the optimized model results with the corresponding true value files to obtain the optimized model accuracy evaluation results, compare the optimized model accuracy evaluation results with the initial model accuracy evaluation results, and generate a model performance comparison report;

[0024] The judgment module is used to judge whether the optimization algorithm model meets the expected conditions according to the model performance comparison report, and output the optimization algorithm model as the final model when the expected conditions are met.

[0025] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0026] The computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the algorithm model verification and accuracy assessment method as described above when executing the computer-readable instructions.

[0027] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0028] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the algorithm model verification and accuracy evaluation method as described above are implemented.

[0029] Compared with the prior art, this application has the following beneficial effects:

[0030] The present application provides an algorithm model verification and accuracy assessment method, which processes the original sample file set by calling a pre-built original algorithm model to obtain the model result to be evaluated, and associates the original sample file with the model result to be evaluated, thereby realizing the automatic association of the sample file with the model result, and providing data support for algorithm model optimization and application; by allocating labeling tasks according to the generated labeling task allocation plan, resources can be reasonably allocated and the completion progress of the labeling task can be improved; the model result to be evaluated is compared with the corresponding true value result to obtain the initial model accuracy assessment result, which can significantly improve the evaluation efficiency and accuracy; the model is optimized according to the initial model accuracy assessment result, which can optimize the model in a targeted manner and improve the optimization efficiency and accuracy; the optimized algorithm model is evaluated, and a model performance comparison report is generated, thereby realizing the automatic analysis and optimization of the model, greatly improving the model iteration efficiency, and thus continuously improving the accuracy and performance of the algorithm model. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0032] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0033] Figure 2 is a flowchart of an embodiment of the algorithm model verification and accuracy assessment method according to the present application;

[0034] Figure 3 It is a structural schematic diagram of an embodiment of an algorithm model verification and accuracy evaluation device according to the present application;

[0035] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0037] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0038] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0039] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0040] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0041] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0042] The server 103 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal device 101 .

[0043] It should be noted that the algorithm model verification and accuracy assessment method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the algorithm model verification and accuracy assessment device is generally arranged in the server / terminal device.

[0044] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0045] Continue to refer Figure 2 , shows a flowchart of an embodiment of the algorithm model verification and accuracy evaluation method according to the present application, comprising the following steps:

[0046] Step S201, obtain an original sample file set, call a pre-built original algorithm model to process the original sample file set, obtain a model result to be evaluated, associate the original sample file and the model result to be evaluated according to a preset rule, and obtain an associated sample set.

[0047] According to the task type, the corresponding original algorithm model and original sample file set are obtained. The original sample file set is a collection of original sample files. For example, in the medical field, for the classification of medical images of different disease categories, the original sample files obtained are image data sets of each corresponding disease category, such as head MRI images, dental images, lung CT images, pathological slice images, etc.; in the financial field, for the insurance risk assessment in the insurance field, the original sample files obtained include historical insurance records, historical claims data, etc.

[0048] The original sample file set is input into the corresponding pre-built original algorithm model, and the data is analyzed and processed by the model to obtain the result of the model to be evaluated. The result of the model to be evaluated is different for different task types. If the original algorithm model is a medical image classification model, the result of the model to be evaluated is the image classification result; if the original algorithm model is a risk assessment model, the result of the model to be evaluated is the risk assessment result; if the original algorithm model is a text recognition model, the result of the model to be evaluated is the text recognition result.

[0049] In this embodiment, the algorithm model verification and accuracy evaluation method is executed on the electronic device (eg Figure 1 The server / terminal device shown in the figure can obtain the original sample file set through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0050] In this embodiment, the original sample file and the model result to be evaluated are automatically associated according to preset rules to obtain an associated sample set, wherein the preset rules include file name matching, timestamp corresponding, unique identifier matching, etc.

[0051] It should be emphasized that in order to further ensure the privacy and security of the original sample files, the above-mentioned original sample files can also be stored in a blockchain node.

[0052] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0053] In some optional implementations of this embodiment, the step of associating the original sample file with the model result to be evaluated according to a preset rule to obtain an associated sample set includes:

[0054] Extract metadata of the original sample file and the model result to be evaluated to obtain file metadata and result metadata;

[0055] Perform correlation analysis on the file metadata and the result metadata to obtain a correlation score;

[0056] When the correlation score is greater than or equal to the preset correlation threshold, the original sample file is associated with the corresponding model result to be evaluated to obtain a correlation result;

[0057] An association mapping table is generated according to the association result, and the path and version information of the model result file to be evaluated corresponding to each original sample file is recorded in the association mapping table to obtain an associated sample set.

[0058] The metadata of the original sample file and the model result to be evaluated are extracted respectively, and the metadata includes the file name, file type, file size, file format, timestamp, etc. The similarity between the file metadata and the result metadata is calculated by the edit distance algorithm, and the similarity is used as the association score. When the association score is greater than or equal to the preset association threshold, the original sample file and the corresponding model result to be evaluated are associated. Among them, the preset association threshold can be set according to the actual situation. For example, it is set to 8. When the association score is greater than or equal to 8, the original sample file and the model result to be evaluated are associated.

[0059] In this embodiment, metadata can be extracted by regular expression matching. If the original sample file is in PDF format, the PDF parsing library is called to parse the PDF file and extract metadata information such as the number of pages, author, and creation time of the PDF; if the original sample file is in picture format, the image processing algorithm is used to analyze the picture content and obtain metadata information such as the shooting device, shooting time, and picture size of the picture. The extracted metadata information is formatted according to a predefined data structure to generate a structured metadata record.

[0060] Generate an association mapping table based on the association results, and record the path and version information of the model results to be evaluated corresponding to each original sample file. Store the association mapping table persistently in the database to establish a long-term association between the original sample file and the model results, and provide a data basis for subsequent version tracing and comparison analysis. For example, store the association mapping table persistently in the MongoDB database, and store the mapping relationship in JSON format.

[0061] In some optional implementations, for original sample files and model results to be evaluated that cannot be directly associated through metadata, a content similarity analysis algorithm is used to extract key features of the original sample files and the model results to be evaluated, respectively, and the similarity between the original sample files and the model results to be evaluated is determined by feature vector comparison; the original sample files and the model results to be evaluated whose similarity is higher than a preset similarity threshold are associated and added to the existing associated results; the associated results are deduplicated and duplicated association relationships are merged to ensure that there is only a unique mapping relationship between each original sample file and the model result to be evaluated.

[0062] By associating the original sample files with the model results to be evaluated, the automatic association of sample files and model results is achieved, providing data support for algorithm model optimization and application.

[0063] Step S202: Generate a labeling task allocation plan based on preset labeling personnel information and associated sample sets.

[0064] In this embodiment, based on the preset labeling personnel information and the associated sample set, the labeling workload of the labeling personnel and the number of samples and the distribution of sample types in the associated sample set are obtained; and a preset task allocation algorithm is used to generate a labeling task allocation plan based on the labeling workload, the number of samples and the distribution of sample types.

[0065] Among them, the labeling personnel information includes the labeling personnel's unique identification and the labeling personnel's current labeling workload; the sample type distribution is the proportion of each sample type in the total number of samples. For example, the sample types include text samples, image samples, and voice samples, and the sample type distributions are 60%, 30%, and 10% respectively.

[0066] A preset task allocation algorithm is used to comprehensively consider the labeling workload, sample quantity and sample type distribution, and an annotation task allocation plan is automatically generated. The task allocation algorithm can adopt the Hungarian algorithm.

[0067] By generating a labeling task allocation plan based on the labeling workload, sample quantity, and sample type distribution, we can achieve reasonable resource allocation, improve labeling efficiency, and ensure that labeling tasks can be completed on schedule.

[0068] In some optional implementations, after the step of generating the labeling task allocation scheme, the following steps are further included:

[0069] Determine whether the number of labeling tasks included in the labeling task allocation plan is greater than a preset number threshold. If it is greater than the preset number threshold, trigger the task splitting process;

[0070] The preset task splitting algorithm is called to split the labeling task allocation plan into multiple sub-labeling task allocation plans, wherein the labeling task amount in each sub-labeling task allocation plan is less than a preset quantity threshold.

[0071] Obtain a labeling task allocation plan, and determine a preset quantity threshold according to the number and complexity of the labeling tasks; if the number of labeling tasks included in the labeling task allocation plan is greater than the preset quantity threshold, use a task splitting algorithm to divide the labeling tasks into multiple subtask sets by analyzing the characteristics and relevance of the labeling tasks; determine whether the amount of the subtask set exceeds the preset quantity threshold according to the task volume, and if so, further split the subtask set until a sub-labeling task allocation plan that meets the preset quantity threshold is obtained; according to the sub-labeling task allocation plan, use the Hungarian task allocation algorithm to allocate the sub-labeling task allocation plan to different labeling personnel.

[0072] In this embodiment, the task completion status and efficiency data of each labeler are obtained, and the task allocation strategy is adjusted dynamically. Specifically, based on the real-time task progress of the labelers, it is determined whether the task allocation needs to be rebalanced. If necessary, the task reallocation process is triggered; by summarizing the subtask results completed by each labeler, a complete labeling task result data set is obtained, and task reallocation is performed based on the labeling task result data set.

[0073] By splitting the tasks according to the number of labeling tasks included in the labeling task allocation plan, it is possible to further balance resource allocation and speed up the task process.

[0074] Step S203: obtaining a labeling task assignment request, and sending the labeling task to a corresponding labeler according to the labeling task assignment plan.

[0075] In this embodiment, according to the labeling task assignment request, labeling task information is obtained, and the labeling task information includes the labeling task ID, labeling requirements, etc.; according to the labeling task assignment plan or the sub-labeling task assignment plan, the labeling personnel list corresponding to each labeling task or sub-labeling task is determined, and the task list of each labeling personnel is automatically generated; through the message push module of the labeling platform, a labeling task notification is sent to each labeling personnel in the labeling personnel list, and the notification content includes the labeling task ID, the task list download address, the labeling requirements description, etc.; after receiving the labeling task notification, the labeling personnel downloads the task list from the labeling platform, labels the files in the task list according to the labeling requirements, and uploads the labeling results to the labeling platform in the specified format; the data processing module of the labeling platform performs format verification and integrity check on the labeling results submitted by the labeling personnel, and stores the labeling results that pass the check as true value results in the labeling result database.

[0076] Among them, the labeling task can be performed through labeling auxiliary tools or through custom trained labeling models.

[0077] In this embodiment, the completion progress of all labeling tasks can be monitored, and the labeling task allocation scheme can be dynamically adjusted according to actual conditions until all labeling tasks are completed.

[0078] Step S204: receiving the true value results submitted by the annotators, and associating the true value results with the corresponding original sample files and the results of the model to be evaluated to form a complete annotated sample data set.

[0079] Specifically, according to the file name corresponding to the true value result submitted by the annotator, the storage path of the corresponding original sample file and the model result file to be evaluated is obtained; the contents of the true value result, the original sample file and the model result to be evaluated are read to obtain the data corresponding to the three files; the data of the true value result is parsed to extract the annotation information therein; it is determined whether the annotation information is complete, and if not, an error message is returned, and the information is modified and resubmitted according to the error message; according to the field name in the annotation information, the corresponding field is searched in the original sample file and the model result to be evaluated; through field matching, the annotation information in the true value result is associated with the corresponding field in the original sample file and the model result to be evaluated; a uniqueness verification algorithm is used to deduplicate the associated data to ensure that each annotation information corresponds to a unique original sample file and model result to be evaluated; the associated and deduplicated true value result, original sample file and model result to be evaluated are used to generate a complete annotated sample data set in a predefined data format; the generated sample data set is stored in a specified path, and the metadata information of the sample data set, including the creation time, number of samples, etc., is recorded to facilitate subsequent management and use.

[0080] By associating the true value results with the corresponding original sample files and the model results to be evaluated, the automatic association of labeled data with the original sample files and model results is achieved, thereby efficiently generating high-quality labeled data sets and providing data support for subsequent model training and evaluation.

[0081] Step S205, comparing the model result to be evaluated with the corresponding true value result to obtain the initial model accuracy evaluation result.

[0082] Before the comparison, the file format and content of the model result to be evaluated and the corresponding true value result are verified to ensure the integrity and consistency of the file. For example, regular expressions can be used to verify the file format and content, such as checking whether the file ends with a specified format such as txt or csv, and whether the content contains necessary fields such as predicted labels, confidence, and true labels.

[0083] After verification, the model result to be evaluated is compared with the corresponding true value result. Specifically, text parsing technology is used to extract key fields in the model result to be evaluated and the true value result to obtain model fields and true value fields; the field values ​​corresponding to the model fields and the true value fields are structured to obtain model structured data and true value structured data; field matching is performed on the model structured data and the true value structured data, and when the model field matches the true value field, it is determined whether the field value corresponding to the model field is consistent with the field value corresponding to the true value field; for the model field with consistent matching, it is marked as a correct field, and for the model field with inconsistent matching, it is marked as an error field, and the error type and error value are recorded to obtain the marking result; according to the marking result, the number of correct fields and the number of error fields are counted, and the accuracy of the original algorithm model is calculated as the initial model accuracy evaluation result.

[0084] In this embodiment, the model result to be evaluated is compared with the corresponding true value result according to the preset evaluation rules, and the preset evaluation rules include the calculation methods of indicators such as accuracy, recall rate, precision rate, etc. Among them, accuracy rate = number of correct samples / total number of samples, recall rate = number of correct samples / number of true value positive samples, precision rate = number of correct samples / number of predicted positive samples.

[0085] For example, the text parsing technology is a pre-set regular expression, which is used to match the key fields in the model results to be evaluated and the true value results. The extracted key field information can be stored in the database, with the field name as the key and the field value as the value, to form a structured data representation to obtain the corresponding model structured data and true value structured data. By traversing the model structured data, it is determined whether each model field exists in the true value structured data. If so, the value is compared. For model fields and true value fields that match consistently, the model field can be marked as true, and inconsistent model fields can be marked as false, and the error type and error value are recorded to obtain the marking result. The marking result includes the number of correct fields and the number of incorrect fields, and the accuracy of the model is calculated based on the number of correct fields and the number of incorrect fields.

[0086] In some optional implementations, detailed information of the error fields is summarized to generate an error report, including the error field name, error value, correct value, etc. The error report is analyzed to identify common error patterns and rules, providing a reference for algorithm optimization. The evaluation results such as model accuracy and error reports are associated with the model version information and stored in a preset algorithm accuracy evaluation database to facilitate subsequent version tracking and comparative analysis.

[0087] By comparing the results of the model to be evaluated with the corresponding true value results, the evaluation efficiency and accuracy can be significantly improved, and the iteration direction of the model can be further guided and optimized.

[0088] Step S206: When the initial model accuracy evaluation result is less than a preset evaluation threshold, a model optimization request is generated, and the original algorithm model is optimized according to the model optimization request to obtain an optimized algorithm model.

[0089] Determine the preset evaluation threshold for model accuracy evaluation based on business needs and historical model performance. Obtain the initial model accuracy evaluation result and determine whether it is lower than the preset evaluation threshold. If the initial model accuracy evaluation result is lower than the preset threshold, a model optimization request is automatically generated.

[0090] Specifically, the key information of the evaluation results is extracted from the initial model accuracy evaluation results, and the key information of the evaluation results includes the accuracy value and the error type; according to the error type, the target sample file in the sample data set is located through data association analysis; the file location information of the target sample file is obtained, and a model optimization request is generated according to the key information of the evaluation results and the file location information.

[0091] Use text analysis techniques such as the bag-of-words model, TF-IDF algorithm, or TextRank algorithm to extract key information from the model accuracy evaluation results, including accuracy values, error types, etc. Through data association analysis, find the target sample files that cause low model accuracy. Integrate the extracted key information and file location information to generate a structured model optimization request. According to the model optimization request, automatically call the algorithm model, and use the located target sample files to retrain and optimize the model. After the model optimization is completed, automatically perform a new round of model accuracy evaluation, and compare the evaluation results with the accuracy values ​​in the model optimization request. If the accuracy of the optimized model is higher than the preset evaluation threshold, the model optimization is judged to be successful; otherwise, continue to iterate until the model accuracy meets the requirements or the maximum number of iterations is reached.

[0092] When the model accuracy assessment results do not meet the requirements, a model optimization request is automatically generated to perform model optimization iterations, realizing automatic analysis and optimization of low-precision models and greatly improving the efficiency of model iterations.

[0093] Step S207, calling the optimization algorithm model to process the labeled sample data set to obtain the optimized model result.

[0094] In this embodiment, the configuration parameters and model file of the optimization algorithm model are obtained; the model loading interface is called to load the configuration parameters and model file of the optimization algorithm model into the system memory to obtain a new version of the algorithm model. Each sample in the labeled sample data set is traversed, and the current sample is input into the new version of the algorithm model for processing through the inference interface of the algorithm model to obtain the model result of the sample; it is determined whether there are any unprocessed samples, and if so, the new version of the algorithm model is continuously called to process the next sample until all samples are processed; the model results of all samples are summarized to generate a model result file; the storage path of the model result file is obtained, and the model result file is written to the path; the relevant information of this model processing is recorded in the system database, including the model version, the processed sample set, the model result file path, etc., to determine the unique identifier of this model processing record; according to the model processing record identifier, the sample set, the model version and the model result file are associated to form a complete model processing traceability chain.

[0095] Step S208, compare the optimized model results with the corresponding true value files to obtain the optimized model accuracy evaluation results, compare the optimized model accuracy evaluation results with the initial model accuracy evaluation results, and generate a model performance comparison report.

[0096] Specifically, the corresponding evaluation index data is extracted from the optimized model accuracy evaluation results and the initial model accuracy evaluation results; data visualization technology is used to generate comparison charts of the optimized algorithm model and the original algorithm model in various indicators based on the evaluation index data; according to the preset model performance evaluation rules, it is judged whether the optimized algorithm model meets the standards for iterative launch, and quantitative analysis conclusions are given; a model performance comparison report is generated based on the comparison charts and analysis conclusions, and the model performance comparison report is automatically sent to developers for subsequent algorithm model iterative optimization.

[0097] By generating a model performance comparison report, you can more intuitively evaluate the optimization effect of the model, make more targeted improvements to the model, and improve the efficiency of model optimization iterations.

[0098] Step S209, judging whether the optimization algorithm model meets the expected conditions according to the model performance comparison report, and outputting the optimization algorithm model as the final model when the expected conditions are met.

[0099] According to the model performance comparison report, determine whether the new version of the optimization algorithm model has a significant improvement in key indicators. If there is a significant improvement, the model update process is triggered, the model deployment interface is called, and the new version model file is deployed to the inference server in the production environment to replace the original version. If the new version model does not meet the expected conditions, the model rollback process is triggered to obtain the production version model file from the model warehouse. Through the model deployment interface, the production version model is redeployed to the inference server to overwrite the new version model to ensure the stability of online services. Obtain the training parameters and iteration records of the new version model that did not meet expectations, analyze the model optimization direction, and generate optimization suggestions. According to the optimization suggestions, adjust the model training parameters, features or model structure through technical means such as data enhancement and regularization, re-train to generate a new model version, submit it to the model warehouse, and re-initiate the model evaluation process, iterate and optimize continuously until the performance of the new version model meets the online requirements, and then update and deploy the model to complete a complete model iteration process.

[0100] This application processes the original sample file set by calling the pre-built original algorithm model to obtain the model result to be evaluated, and associates the original sample file with the model result to be evaluated, thereby realizing the automatic association of the sample file with the model result, and providing data support for the optimization and application of the algorithm model; by allocating the labeling task according to the generated labeling task allocation plan, it is possible to reasonably allocate resources and improve the completion progress of the labeling task; by comparing the model result to be evaluated with the corresponding true value result to obtain the initial model accuracy evaluation result, the evaluation efficiency and accuracy can be significantly improved; by optimizing the model according to the initial model accuracy evaluation result, the model can be optimized in a targeted manner to improve the optimization efficiency and accuracy; by evaluating the optimized algorithm model and generating a model performance comparison report, it is possible to realize the automatic analysis and optimization of the model, greatly improve the model iteration efficiency, and thus continuously improve the accuracy and performance of the algorithm model.

[0101] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0102] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0103] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0104] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0105] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of an algorithm model verification and accuracy evaluation device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0106] like Figure 3 As shown, the algorithm model verification and accuracy evaluation device 300 described in this embodiment includes: an acquisition module 301, a solution generation module 302, an allocation module 303, an association module 304, a comparison module 305, an optimization module 306, a model processing module 307, a report generation module 308 and a judgment module 309. Among them:

[0107] The acquisition module 301 is used to acquire an original sample file set, call a pre-built original algorithm model to process the original sample file set, obtain a model result to be evaluated, and associate the original sample file with the model result to be evaluated according to a preset rule to obtain an associated sample set;

[0108] The solution generation module 302 is used to generate a labeling task allocation solution according to the preset labeling personnel information and the associated sample set;

[0109] The assignment module 303 is used to obtain an assignment request for a labeling task, and send the labeling task to a corresponding labeler according to the labeling task assignment plan;

[0110] The association module 304 is used to receive the true value results submitted by the annotator, and associate the true value results with the corresponding original sample files and the model results to be evaluated to form a complete annotated sample data set;

[0111] The comparison module 305 is used to compare the model result to be evaluated with the corresponding true value result to obtain the initial model accuracy evaluation result;

[0112] The optimization module 306 is used to generate a model optimization request when the initial model accuracy evaluation result is less than a preset evaluation threshold, and optimize the original algorithm model according to the model optimization request to obtain an optimized algorithm model;

[0113] The model processing module 307 is used to call the optimization algorithm model to process the labeled sample data set to obtain the optimized model result;

[0114] The report generation module 308 is used to compare the optimized model result with the corresponding true value file to obtain the optimized model accuracy evaluation result, compare the optimized model accuracy evaluation result with the initial model accuracy evaluation result, and generate a model performance comparison report;

[0115] The judgment module 309 is used to judge whether the optimization algorithm model meets the expected conditions according to the model performance comparison report, and output the optimization algorithm model as the final model when the expected conditions are met.

[0116] It should be emphasized that in order to further ensure the privacy and security of the original sample files, the above-mentioned original sample files can also be stored in a blockchain node.

[0117] Based on the above-mentioned algorithm model verification and accuracy evaluation device 300, the original sample file set is processed by calling the pre-built original algorithm model to obtain the model result to be evaluated, and the original sample file and the model result to be evaluated are associated, thereby realizing the automatic association of the sample file and the model result, and providing data support for algorithm model optimization and application; by assigning labeling tasks according to the generated labeling task allocation plan, resources can be reasonably allocated and the completion progress of the labeling task can be improved; the model result to be evaluated is compared with the corresponding true value result to obtain the initial model accuracy evaluation result, which can significantly improve the evaluation efficiency and accuracy; the model is optimized according to the initial model accuracy evaluation result, which can optimize the model in a targeted manner and improve the optimization efficiency and accuracy; the optimized algorithm model is evaluated, and a model performance comparison report is generated, thereby realizing automatic analysis and optimization of the model, greatly improving the model iteration efficiency, and thus continuously improving the accuracy and performance of the algorithm model.

[0118] In some optional implementations, the acquisition module 301 includes:

[0119] An extraction submodule, used to extract metadata of the original sample file and the model result to be evaluated, to obtain file metadata and result metadata;

[0120] A correlation analysis submodule, used to perform correlation analysis on the file metadata and the result metadata to obtain a correlation score;

[0121] An association submodule, used for associating the original sample file with the corresponding model result to be evaluated to obtain an association result when the association score is greater than or equal to a preset association threshold;

[0122] The mapping submodule is used to generate an association mapping table according to the association result, and record the path and version information of the to-be-evaluated model result file corresponding to each original sample file in the association mapping table to obtain an associated sample set.

[0123] By associating the original sample files with the model results to be evaluated, the automatic association of sample files and model results is achieved, providing data support for algorithm model optimization and application.

[0124] In some optional implementations, the solution generation module 302 is further configured to:

[0125] According to the preset labeling personnel information and the associated sample set, the labeling workload of the labeling personnel and the number of samples and the distribution of sample types in the associated sample set are obtained;

[0126] A preset task allocation algorithm is used to generate a labeling task allocation plan according to the labeling workload, the number of samples and the distribution of sample types.

[0127] By generating a labeling task allocation plan based on the labeling workload, sample quantity, and sample type distribution, we can achieve reasonable resource allocation, improve labeling efficiency, and ensure that labeling tasks can be completed on schedule.

[0128] In some optional implementations of this embodiment, the solution generation module 302 further includes a task splitting submodule, which is used to:

[0129] Determine whether the number of labeling tasks included in the labeling task allocation plan is greater than a preset number threshold, and if so, trigger a task splitting process;

[0130] A preset task splitting algorithm is called to split the labeling task allocation scheme into a plurality of sub-labeling task allocation schemes, wherein the labeling task amount in each of the sub-labeling task allocation schemes is less than a preset quantity threshold.

[0131] By splitting the tasks according to the number of labeling tasks included in the labeling task allocation plan, it is possible to further balance resource allocation and speed up the task process.

[0132] In some optional implementations, the comparison module 305 includes:

[0133] A text parsing submodule, used to extract key fields from the model result to be evaluated and the true value result by using text parsing technology, and obtain model fields and true value fields;

[0134] A structuring submodule, used for structuring the field values ​​corresponding to the model field and the true value field to obtain model structured data and true value structured data;

[0135] A matching submodule, used for performing field matching on the model structured data and the true value structured data, and determining whether a field value corresponding to the model field is consistent with a field value corresponding to the true value field when the model field matches the true value field;

[0136] The marking submodule is used to mark the model fields with consistent matching as correct fields, and mark the model fields with inconsistent matching as error fields, and record the error type and error value to obtain the marking result;

[0137] The statistical submodule is used to count the number of the correct fields and the number of the erroneous fields according to the marking results, and calculate the accuracy of the original algorithm model as the initial model accuracy evaluation result.

[0138] By comparing the results of the model to be evaluated with the corresponding true value results, the evaluation efficiency and accuracy can be significantly improved, and the iteration direction of the model can be further guided and optimized.

[0139] In some optional implementations, the optimization module 306 includes:

[0140] An information extraction submodule, used to extract key information of the evaluation result from the initial model accuracy evaluation result, wherein the key information of the evaluation result includes an accuracy value and an error type;

[0141] A query submodule, used to locate a target sample file in the sample data set through data association analysis according to the error type;

[0142] The request submodule is used to obtain the file location information of the target sample file and generate a model optimization request according to the evaluation result key information and the file location information.

[0143] When the model accuracy assessment results do not meet the requirements, a model optimization request is automatically generated to perform model optimization iterations, realizing automatic analysis and optimization of low-precision models and greatly improving the efficiency of model iterations.

[0144] In some optional implementations, the report generation module 308 includes:

[0145] A data acquisition submodule, used to extract corresponding evaluation index data from the optimized model accuracy evaluation result and the initial model accuracy evaluation result;

[0146] A visualization submodule, for using data visualization technology to generate a comparison chart of the optimization algorithm model and the original algorithm model on various indicators according to the evaluation index data;

[0147] The evaluation submodule is used to determine whether the optimization algorithm model meets the standard for iterative launch according to the preset model performance evaluation rules, and to give a quantitative analysis conclusion;

[0148] The report submodule is used to generate a model performance comparison report based on the comparison chart and the analysis conclusion.

[0149] By generating a model performance comparison report, you can more intuitively evaluate the optimization effect of the model, make more targeted improvements to the model, and improve the efficiency of model optimization iterations.

[0150] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0151] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 4 with a memory 41, a processor 42, and a network interface 43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0152] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0153] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions for algorithm model verification and accuracy evaluation methods, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0154] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer-readable instructions stored in the memory 41 or process data, such as computer-readable instructions for running the algorithm model verification and accuracy assessment method.

[0155] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0156] By calling the pre-built original algorithm model to process the original sample file set, the results of the model to be evaluated are obtained, and the original sample files and the results of the model to be evaluated are associated, which realizes the automatic association of sample files and model results, and provides data support for algorithm model optimization and application; by allocating labeling tasks according to the generated labeling task allocation plan, resources can be reasonably allocated and the completion progress of labeling tasks can be improved; the results of the model to be evaluated are compared with the corresponding true value results to obtain the initial model accuracy evaluation results, which can significantly improve the evaluation efficiency and accuracy; the model is optimized according to the initial model accuracy evaluation results, which can optimize the model in a targeted manner and improve the optimization efficiency and accuracy; the optimized algorithm model is evaluated and a model performance comparison report is generated, which realizes the automatic analysis and optimization of the model, greatly improves the model iteration efficiency, and continuously improves the accuracy and performance of the algorithm model.

[0157] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the algorithm model verification and accuracy evaluation method as described above.

[0158] By calling the pre-built original algorithm model to process the original sample file set, the results of the model to be evaluated are obtained, and the original sample files and the results of the model to be evaluated are associated, which realizes the automatic association of sample files and model results, and provides data support for algorithm model optimization and application; by allocating labeling tasks according to the generated labeling task allocation plan, resources can be reasonably allocated and the completion progress of labeling tasks can be improved; the results of the model to be evaluated are compared with the corresponding true value results to obtain the initial model accuracy evaluation results, which can significantly improve the evaluation efficiency and accuracy; the model is optimized according to the initial model accuracy evaluation results, which can optimize the model in a targeted manner and improve the optimization efficiency and accuracy; the optimized algorithm model is evaluated and a model performance comparison report is generated, which realizes the automatic analysis and optimization of the model, greatly improves the model iteration efficiency, and continuously improves the accuracy and performance of the algorithm model.

[0159] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0160] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. An algorithm model verification and accuracy assessment method, characterized in that: The steps include: Obtaining an original sample file set, calling a pre-built original algorithm model to process the original sample file set to obtain a model result to be evaluated, and associating the original sample file with the model result to be evaluated according to a preset rule to obtain an associated sample set; Generate a labeling task allocation plan based on the preset labeling personnel information and the associated sample set; Obtaining a request for assigning a labeling task, and sending the labeling task to a corresponding labeler according to the labeling task assignment plan; Receive the true value results submitted by the annotator, and associate the true value results with the corresponding original sample files and the results of the model to be evaluated to form a complete annotated sample data set; Compare the model result to be evaluated with the corresponding true value result to obtain an initial model accuracy evaluation result; When the initial model accuracy evaluation result is less than a preset evaluation threshold, a model optimization request is generated, and the original algorithm model is optimized according to the model optimization request to obtain an optimized algorithm model; Calling the optimization algorithm model to process the labeled sample data set to obtain an optimized model result; Compare the optimized model result with the corresponding true value file to obtain the optimized model accuracy evaluation result, compare the optimized model accuracy evaluation result with the initial model accuracy evaluation result, and generate a model performance comparison report; According to the model performance comparison report, it is determined whether the optimization algorithm model meets the expected conditions. When the expected conditions are met, the optimization algorithm model is output as the final model.

2. The algorithm model verification and accuracy assessment method according to claim 1, characterized in that: The step of associating the original sample file with the model result to be evaluated according to a preset rule to obtain an associated sample set comprises: Extracting metadata of the original sample file and the model result to be evaluated to obtain file metadata and result metadata; Performing a correlation analysis on the file metadata and the result metadata to obtain a correlation score; When the correlation score is greater than or equal to a preset correlation threshold, the original sample file is associated with the corresponding model result to be evaluated to obtain a correlation result; An association mapping table is generated according to the association result, and the path and version information of the to-be-evaluated model result file corresponding to each of the original sample files is recorded in the association mapping table to obtain an associated sample set.

3. The algorithm model verification and accuracy assessment method according to claim 1, characterized in that: The step of generating a labeling task allocation scheme according to the preset labeling personnel information and the associated sample set includes: According to the preset labeling personnel information and the associated sample set, the labeling workload of the labeling personnel and the number of samples and the distribution of sample types in the associated sample set are obtained; A preset task allocation algorithm is used to generate a labeling task allocation plan according to the labeling workload, the number of samples and the distribution of sample types.

4. The algorithm model verification and accuracy assessment method according to claim 3, characterized in that: After the step of generating the labeling task allocation scheme, the method further includes: Determine whether the number of labeling tasks included in the labeling task allocation plan is greater than a preset number threshold, and if so, trigger a task splitting process; A preset task splitting algorithm is called to split the labeling task allocation scheme into a plurality of sub-labeling task allocation schemes, wherein the labeling task amount in each of the sub-labeling task allocation schemes is less than a preset quantity threshold.

5. The algorithm model verification and accuracy assessment method according to claim 1, characterized in that: The step of comparing the model result to be evaluated with the corresponding true value result to obtain the initial model accuracy evaluation result comprises: Using text parsing technology, extract key fields from the model result to be evaluated and the true value result to obtain model fields and true value fields; Structuring the field values ​​corresponding to the model field and the true value field to obtain model structured data and true value structured data; Performing field matching on the model structured data and the true value structured data, and when the model field matches the true value field, determining whether a field value corresponding to the model field is consistent with a field value corresponding to the true value field; For the model fields that match consistently, mark them as correct fields, and for the model fields that do not match consistently, mark them as error fields, and record the error type and error value to obtain the marking result; According to the marking results, the number of the correct fields and the number of the erroneous fields are counted, and the accuracy of the original algorithm model is calculated as the initial model accuracy evaluation result.

6. The algorithm model verification and accuracy assessment method according to claim 1, characterized in that: When the initial model accuracy evaluation result is less than a preset evaluation threshold, the step of generating a model optimization request includes: Extracting evaluation result key information from the initial model accuracy evaluation result, wherein the evaluation result key information includes an accuracy value and an error type; According to the error type, locating the target sample file in the sample data set through data association analysis; The file location information of the target sample file is obtained, and a model optimization request is generated according to the evaluation result key information and the file location information.

7. The algorithm model verification and accuracy assessment method according to claim 1, characterized in that: The step of comparing the optimized model accuracy evaluation result with the initial model accuracy evaluation result to generate a model performance comparison report includes: Extracting corresponding evaluation index data from the optimized model accuracy evaluation result and the initial model accuracy evaluation result; Using data visualization technology, a comparison chart of the optimized algorithm model and the original algorithm model on various indicators is generated according to the evaluation index data; According to the preset model performance evaluation rules, determine whether the optimization algorithm model meets the standard for iterative launch, and give a quantitative analysis conclusion; A model performance comparison report is generated based on the comparison chart and the analysis conclusion.

8. An algorithm model verification and accuracy assessment device, characterized in that: include: An acquisition module is used to acquire an original sample file set, call a pre-built original algorithm model to process the original sample file set, obtain a model result to be evaluated, and associate the original sample file with the model result to be evaluated according to a preset rule to obtain an associated sample set; A scheme generation module, used to generate a labeling task allocation scheme according to preset labeling personnel information and the associated sample set; An assignment module is used to obtain an assignment request for a labeling task and send the labeling task to a corresponding labeler according to the labeling task assignment plan; An association module is used to receive the true value results submitted by the annotator, and associate the true value results with the corresponding original sample files and the model results to be evaluated to form a complete annotated sample data set; A comparison module is used to compare the model result to be evaluated with the corresponding true value result to obtain an initial model accuracy evaluation result; An optimization module, configured to generate a model optimization request when the initial model accuracy evaluation result is less than a preset evaluation threshold, and optimize the original algorithm model according to the model optimization request to obtain an optimized algorithm model; A model processing module, used to call the optimization algorithm model to process the labeled sample data set to obtain an optimized model result; A report generation module, used to compare the optimized model results with the corresponding true value files to obtain the optimized model accuracy evaluation results, compare the optimized model accuracy evaluation results with the initial model accuracy evaluation results, and generate a model performance comparison report; The judgment module is used to judge whether the optimization algorithm model meets the expected conditions according to the model performance comparison report, and output the optimization algorithm model as the final model when the expected conditions are met.

9. A computer device, characterized in that: It includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the algorithm model verification and accuracy assessment method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the algorithm model verification and accuracy assessment method according to any one of claims 1 to 7.

Images