Database algorithm model accuracy verification method and device, equipment and medium

Abstract

The present application relates to the technical field of data processing, and particularly relates to a database algorithm model accuracy verification method and device, equipment and medium. N algorithm models in a database are obtained, and distribution values are generated for the N algorithm models. User requests for each algorithm model are generated, and response results based on the corresponding user requests are obtained. If the response result corresponds to the i-th algorithm model, the response result is determined to be of the i-th type. The types of the response results corresponding to each algorithm model are counted, and verification results for each algorithm model are obtained based on the counting results and the distribution values. In the present application, user requests are generated to obtain a large amount of verification data. Different types of response results corresponding to the algorithm models in the verification data can be used to calculate the accuracy of the database algorithm models. The accuracy of the algorithm models is analyzed from multiple angles based on the different types of response results, thereby improving the calculation accuracy of the accuracy.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, device, and medium for verifying the accuracy of a database algorithm model. Background Technology

[0002] With the advent of the big data era, big data recommendation algorithms and their applications have become a key research focus both domestically and internationally. Domestic research largely focuses on applying big data algorithms to recommendation practices within a big data framework, particularly in areas like electronic communications and personalized news recommendations. Developers typically design multiple recommendation algorithms and conduct online testing to select the best-performing one. Currently, the validation of recommendation algorithms generally relies on simple data analysis and comparison. This method is relatively simplistic, resulting in low accuracy verification of big data algorithms and failing to provide comprehensive and effective data support for improvement. Therefore, improving the accuracy verification of big data algorithms is a pressing issue that needs to be addressed. Summary of the Invention

[0003] Therefore, it is necessary to provide a method, apparatus, device, and medium for verifying the accuracy of a database algorithm model, in order to solve the problem of low accuracy verification precision in the process of verifying the accuracy of a database algorithm model.

[0004] The first aspect of this application provides a method for verifying the accuracy of a database algorithm model, characterized in that the defect detection method includes: Obtain N algorithm models from the database, generate allocation values ​​for the N algorithm models, generate user requests for each algorithm model, and the number of user requests for each algorithm model is the same as the allocation value for the corresponding algorithm model, where N is an integer greater than 1; All user requests are sent to the database, and the database's response results based on the corresponding user requests are obtained. If the algorithm model corresponding to the response result is the i-th algorithm model, then the response result is determined to be of type i, 0. <i<N; The types of response results corresponding to each algorithm model are statistically analyzed to obtain statistical results. Based on the statistical results and the assigned values, the verification results of each algorithm model are obtained.

[0005] A second aspect of this application provides a verification device for the accuracy of a database algorithm model, characterized in that the defect detection device includes: a user request generation module, configured to obtain N algorithm models in the database, generate an allocation value for each of the N algorithm models, and generate user requests for each algorithm model, wherein the number of user requests for each algorithm model is the same as the allocation value of the corresponding algorithm model, and N is an integer greater than 1; a response result acquisition module, configured to send all user requests to the database, acquire response results of the database based on the corresponding user requests, and determine that the response result is of the i-th type if the algorithm model corresponding to the response result is the i-th algorithm model, where 0<i<N; a verification module, configured to count types of response results corresponding to each algorithm model to obtain a statistical result, and obtain a verification result of each algorithm model according to the statistical result and the allocation value.

[0006] In a third aspect, an embodiment of the present invention provides a computer device, wherein the computer device comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the method for verifying accuracy of a database algorithm model according to the first aspect is implemented.

[0007] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for verifying accuracy of a database algorithm model according to the first aspect is implemented.

[0008] Compared with the prior art, the present invention has the following beneficial effects: N algorithm models in a database are obtained, allocation values are generated for the N algorithm models, user requests are generated for each algorithm model, the number of user requests of each algorithm model is the same as the allocation value of the corresponding algorithm model, N is an integer greater than 1, all user requests are sent to the database, response results of the database based on the corresponding user requests are obtained, if the algorithm model corresponding to a response result is the i-th algorithm model, the response result is determined to be of the i-th type, 0<i<N, types of response results corresponding to each algorithm model are counted to obtain a statistical result, and a verification result of each algorithm model is obtained according to the statistical result and the allocation value. In the present invention, a large amount of verification data is obtained by generating user requests, the accuracy of a database algorithm model can be calculated according to response results of different types obtained from corresponding algorithm models in the verification data, and the accuracy of the algorithm model can be obtained through analysis from multiple perspectives based on response results of different types, which improves the calculation accuracy of accuracy. Description of Drawings

[0009] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a schematic diagram of an application environment for a method for verifying the accuracy of a database algorithm model according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating a method for verifying the accuracy of a database algorithm model according to an embodiment of the present invention. Figure 3 This is a flowchart illustrating a method for verifying the accuracy of a database algorithm model according to an embodiment of the present invention. Figure 4 This is a schematic diagram of the structure of a database algorithm model accuracy verification device provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0012] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0013] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0014] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0015] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0016] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0017] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0018] To illustrate the technical solution of the present invention, specific embodiments are described below.

[0019] An embodiment of the present invention provides a method for verifying the accuracy of a database algorithm model, which can be applied to, for example... Figure 1 In this application environment, the client communicates with the server. Clients include, but are not limited to, PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). The server can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0020] See Figure 2This is a flowchart illustrating a method for verifying the accuracy of a database algorithm model according to an embodiment of the present invention. This method for verifying the accuracy of a database algorithm model can be applied to... Figure 1 The server in the above-mentioned configuration connects to the corresponding client, such as... Figure 2 As shown, the method for verifying the accuracy of the database algorithm model may include the following steps.

[0021] S201: Obtain N algorithm models from the database, generate allocation values ​​for the N algorithm models, and generate user requests for each algorithm model. The number of user requests for each algorithm model is the same as the allocation value for the corresponding algorithm model.

[0022] In step S201, N algorithm models are obtained from the database, where N is an integer greater than 1. A corresponding prediction value is assigned to each of the N algorithm models. Based on the prediction value of each algorithm model, a corresponding user request is generated. The number of user requests corresponding to each algorithm model is equal to the assigned value of each algorithm model.

[0023] In this embodiment, N algorithm models are retrieved from the database, and a corresponding algorithm identifier is assigned to each model. The algorithm identifier uniquely corresponds to each algorithm model. Random data is generated using a random function in the simulation tool. This random data is then allocated to each algorithm model, or different allocation values ​​can be configured for each model in the configuration interface. After each algorithm model receives its assigned value, a user request equal to that value is generated for each model. This generated user request simulates a real user request.

[0024] For example, CTR prediction models, payment rate models, and viewing duration models are retrieved from the database. Data is randomly allocated to each model using a modeling tool to obtain the allocation number for each model. If the allocation numbers for the CTR prediction model, payment rate model, and viewing duration model are 80, 10, and 10 respectively, then when generating corresponding user requests, 80 user requests with the target algorithm model being the CTR prediction model will be generated, 10 user requests with the target algorithm model being the payment rate model will be generated, and 10 user requests with the target algorithm model being the viewing duration model will be generated.

[0025] Optionally, obtain N algorithm models from the database and generate assigned values ​​for the N algorithm models, including: Obtain N algorithm models from the database, use a preset simulation tool to assign probability values ​​to the N algorithm models, and the sum of the probability values ​​assigned to the N algorithm models is 1; Based on the target value and the probability value of each algorithm model, the allocation value for each algorithm model is obtained.

[0026] In this embodiment, the predicted values ​​assigned to N algorithm models and the corresponding algorithm identifier information for each algorithm model are obtained. For example, the CTR prediction model, the payment rate model, and the viewing duration model are obtained from the database. An algorithm identifier is set for each algorithm model. The CTR prediction model can be set to algorithm identifier 0, the payment rate model to algorithm identifier 1, and the viewing duration model to algorithm identifier 2. When generating a user request, the corresponding user request is generated according to the algorithm identifier corresponding to each algorithm model. For example, when generating a user request corresponding to the CTR prediction model, the generated user request includes the algorithm identifier corresponding to the CTR prediction model, so that the algorithm model with the same algorithm identifier can be obtained from the database according to the algorithm identifier corresponding to the algorithm model in the user request.

[0027] In this embodiment, the preset simulation tool used is JMeter. JMeter is used to simulate real user requests, including the user's device model, processor, MAC address, etc. JMeter can be used to simulate users performing actual operations. JMeter generates simulated requests, sends them to the server, and retrieves the results returned by the server. It can act as a proxy gateway between the middleware server and the client to capture the client's requests and the middleware server's responses. This makes it easy to accurately verify the script. The message is initiated by the JMeter tool, sent to the middleware server, processed by the middleware server, and then sent to a third-party server. The third-party server performs database operations, returns the results to the third-party server, and then sends the response content to the middleware server. Finally, the corresponding response result can be obtained in the JMeter response module.

[0028] Based on different request methods, such as POST, corresponding POST requests are generated, and parameters are transmitted in POST format. The generated simulated user requests ensure that the submitted data is complete and will not be truncated if the submitted data is too long, thus guaranteeing data integrity and preventing data loss. In one embodiment, for requests that do not require data submission, such as GET requests, only the corresponding interface address and request headers need to be obtained to send the corresponding simulated user request to the server in GET format.

[0029] Using the JMeter simulation tool, verification can be performed without a user interface, simply by passing parameters through an API. Verification doesn't require a complete understanding of the business logic and processes; you only need to find the corresponding API and perform verification based on the scenario it represents. Verification scripts are easy to maintain, with one script per scenario. Scripts can be copied and reused for future verifications. You can even bypass the user interface and send data directly to the backend API.

[0030] It should be noted that the simulation tool first assigns a corresponding probability value to each algorithm model. The sum of the probability values ​​assigned to N algorithm models is 1. Based on the total target value of the user requests to be generated, and the probability value of each algorithm model, the assigned value is calculated. The simulation tool then generates user requests equal to the assigned value.

[0031] S202: Send all user requests to the database, obtain the database's response results based on the corresponding user requests, and if the algorithm model corresponding to the response result is the i-th algorithm model, then determine that the response result is of the i-th type.

[0032] In step S202, the generated user request is sent to the database, and the database receives the response result based on the corresponding user request. The response result includes the call result of the interface to the user request. The call result may be equal to or different from the algorithm model in the user request.

[0033] In this embodiment, the algorithm model is executable code developed by developers. Deploying the algorithm model allows the system to integrate it, with each model forming an independent service. Calling the algorithm model is necessary to provide the corresponding service capabilities. The message originates from the JMeter tool, is sent to the middleware server, processes it, and then forwards it to a third-party server. The third-party server performs database operations, returns the results, and then sends the response to the middleware server. Finally, the corresponding response result can be obtained in the JMeter response module.

[0034] When the algorithm model corresponding to the response result is the i-th algorithm model, the response result is classified as the i-th type. For example, if the algorithm models are CTR prediction model, payment rate model, and viewing duration model, and the algorithm models are sorted in the order of CTR prediction model, payment rate model, and viewing duration model, then if the algorithm model corresponding to the response result is the payment rate model, the response result is type 2; if the algorithm model corresponding to the response result is the CTR prediction model, the response result is type 1.

[0035] Optionally, all user requests are sent to the database, and the database's response based on the corresponding user request is obtained, including: Send all user requests to the database and obtain the algorithm identifier corresponding to the algorithm model returned by the database based on the corresponding user request; Count the number of each algorithm identifier; Based on the number of each algorithm identifier, the database response result for the corresponding user request is determined.

[0036] In this embodiment, all user requests are sent to the database, and the algorithm identifier corresponding to the algorithm model returned by the database based on the corresponding user request is obtained. The algorithm model identifier is statistically analyzed to obtain the response result corresponding to each algorithm model. For example, if the CTR prediction model is set to algorithm identifier 0, the payment rate model is set to algorithm identifier 1, and the viewing duration model is set to algorithm identifier 2, if there are 10 user requests, and the algorithm identifier obtained in the response result is 0012012010, then there are 5 CTR prediction models, 3 payment rate models, and 2 viewing duration models returned in the response result.

[0037] S203: Statistically analyze the types of response results corresponding to each algorithm model, obtain statistical results, and obtain the verification results of each algorithm model based on the statistical results and the assigned values.

[0038] In step S203, the types of response results corresponding to each algorithm model are statistically analyzed. This involves statistically analyzing the same algorithm models in the response results so that the verification result of the algorithm model can be calculated based on the statistical results and the allocation value corresponding to each algorithm model.

[0039] In this embodiment, different types of response results correspond to different algorithm models. The types of response results corresponding to each algorithm model are statistically analyzed to obtain the response result value returned by each algorithm model. For example, if the database includes 5 algorithm models, the 5 algorithm models are sorted, and the response results of the i-th type are statistically analyzed. Then, the i-th algorithm model in the response results is statistically analyzed.

[0040] Once the number of each algorithm model in the image results is obtained, it is carefully compared with the corresponding algorithm model's assigned value to verify the accuracy of the algorithm model.

[0041] Optionally, the types of response results corresponding to each algorithm model are statistically analyzed to obtain statistical results. Based on the statistical results and the assigned values, the validation results of each algorithm model are obtained, including: If the algorithm model corresponding to the response result is the same as the algorithm model of the user request, then the response result is of the target type; For any given algorithm model, count the number of response results corresponding to the target type. Determine this number as the response value of the algorithm model. Based on the response values ​​and target values ​​of all algorithm models, obtain the accuracy of the algorithm models in the database and determine the validation results of the algorithm models.

[0042] In this embodiment, the response result returned by each user request is obtained, and it is determined whether the algorithm model in the user request and the algorithm model in the response result are the same algorithm model. If the algorithm model corresponding to the response result is the algorithm model corresponding to the user request, then the response result is the target type. When the response result is the target type, the corresponding interface call is the algorithm model corresponding to the user request, and the algorithm model called by the interface is the correct algorithm model. Otherwise, when the corresponding interface call is different from the algorithm model corresponding to the user request, the algorithm model called by the interface is the wrong algorithm model.

[0043] The accuracy of the algorithm model in the database is obtained by counting the number of target types in the response results and, for any algorithm model, calculating the ratio of the total number of target types to the target value. The target value is the total number of user requests.

[0044] Optionally, the types of response results corresponding to each algorithm model are statistically analyzed to obtain statistical results. Based on the statistical results and the assigned values, the validation results of each algorithm model are obtained, including: The types of response results corresponding to each algorithm model are statistically analyzed to obtain the statistical results for each algorithm model. The number of responses of each algorithm model to the target type is counted. Based on the statistical results corresponding to each algorithm model, the accuracy of each algorithm model in the database is calculated. Based on the number of target types in the response results of each algorithm model, and the corresponding allocation value for each algorithm model, the sensitivity of each algorithm model in the database is calculated. Based on the accuracy and sensitivity of each algorithm model in the database, the F1 score of each algorithm model is determined, and the verification results are obtained.

[0045] In this embodiment, the types of response results corresponding to each algorithm model are statistically analyzed to obtain the statistical results for each algorithm model. The number of responses returned by each algorithm model in the response results is obtained. Based on the statistical results for each algorithm model, the accuracy of each algorithm model in the database is calculated. That is, the ratio of the number of target types corresponding to each algorithm model in the response results to the number of responses returned by the corresponding algorithm model in the response results is calculated. For example, if the number of types corresponding to the CTR prediction model in the response results is 80, and the number of target types for which the user request and response results are equal is 50, then the accuracy of the CTR prediction model is 62.5%.

[0046] The sensitivity of each algorithm model in the database is calculated according to the number of target types in the response result of each algorithm model and the allocation value corresponding to each algorithm model. For example, the number of target types for which the user request corresponding to the CTR prediction model is equal to the response result is 50, and the allocation value corresponding to the CTR prediction model is 100, so the sensitivity corresponding to the CTR prediction model is 50%.

[0047] According to the precision and sensitivity of each algorithm model in the database, the F1 Score of each algorithm model is determined. The F1 Score reconciles the precision and sensitivity of each algorithm model and is an indicator used to measure the verification result of each algorithm model.

[0048] In this embodiment, the verification results of the algorithm model include verification of the accuracy, precision, sensitivity and F1 Score of the algorithm models in the database. Analyzing the accuracy of the algorithm models in the database based on these verification results is beneficial to improving the accuracy of accuracy analysis.

[0049] Obtain N algorithm models in a database, generate allocation values for the N algorithm models, generate user requests for each algorithm model, wherein the number of user requests for each algorithm model is the same as the allocation value of the corresponding algorithm model, and N is an integer greater than 1; send all user requests to the database, obtain response results based on the corresponding user requests from the database, if the algorithm model corresponding to the response result is the i-th algorithm model, determine that the response result is of the i-th type, 0 < i < N; count the types of response results corresponding to each algorithm model to obtain a statistical result, and obtain the verification result of each algorithm model according to the statistical result and the allocation value. In the present invention, a large amount of verification data is obtained by generating user requests, the accuracy of the database algorithm model can be calculated according to response results of different types obtained from the corresponding algorithm models in the verification data, and the accuracy of the algorithm model is analyzed from multiple perspectives through response results of different types, which improves the calculation accuracy of the accuracy.

[0050] See Figure 3 , which is a schematic flowchart of a method for verifying accuracy of a database algorithm model provided by an embodiment of the present invention, as shown in Figure 3 , the method for verifying accuracy of a database algorithm model may comprise the following steps: S301: obtaining N algorithm models in a database, generating allocation values for the N algorithm models, and generating user requests for each algorithm model, wherein the number of user requests for each algorithm model is the same as the allocation value of the corresponding algorithm model, and N is an integer greater than 1; S302: Send all user requests to the database, obtain the database's response based on the corresponding user request, and if the algorithm model corresponding to the response is the i-th algorithm model, then determine the response as type i, 0 <i<N; S303: Statistically analyze the types of response results corresponding to each algorithm model, obtain statistical results, and obtain the verification results of each algorithm model based on the statistical results and the assigned values.

[0051] The contents of steps S301 to S303 are the same as those of steps S201 to S203, and can be referred to the description of steps S201 to S203, which will not be repeated here.

[0052] S304: If the algorithm model corresponding to the user request is the j-th algorithm model, construct a confusion matrix based on the i-th algorithm model corresponding to the response result and the j-th algorithm model corresponding to the user request. 0 <j<N; S305: Use the preset visualization tool to visualize the confusion matrix and obtain a visualization heatmap of the verification results for each algorithm model.

[0053] In this embodiment, a confusion matrix is ​​constructed based on the algorithm model in the user request and the algorithm model corresponding to the response result. The i-th algorithm model corresponding to the response result and the j-th algorithm model corresponding to the user request are used to fill the confusion matrix sequentially. A preset visualization tool is used to visualize the confusion matrix to obtain a visualization heatmap of the verification result of each algorithm model.

[0054] In this embodiment, the confusion matrix is ​​visualized using a heatmap generated by Python's seaborn library. The horizontal axis represents the assignment value of the algorithm model, and the vertical axis represents the response value of the algorithm model. Different colors are used to render the data based on the magnitude of the corresponding verification result values, so as to facilitate the analysis of the verification results of each algorithm model.

[0055] Please see Figure 4 , Figure 4 This is a schematic diagram of a database algorithm model accuracy verification device provided in an embodiment of the present invention. In this embodiment, the terminal includes units used for execution... Figures 2 to 3 The steps in the corresponding embodiments. Please refer to the details. Figures 2 to 3 as well as Figures 2 to 3 The relevant descriptions in the corresponding embodiments are shown below. For ease of explanation, only the parts relevant to this embodiment are shown. See also... Figure 4 The verification device 40 includes: a user request generation module 41, a response result acquisition module 42, and a verification module 43.

[0056] User request generation module 41 is used to obtain N algorithm models from the database, generate allocation values ​​for the N algorithm models, generate user requests for each algorithm model, and the number of user requests for each algorithm model is the same as the allocation value of the corresponding algorithm model, where N is an integer greater than 1. The response result acquisition module 42 is used to send all user requests to the database, obtain the response results from the database based on the corresponding user requests, and determine the response result as type i if the algorithm model corresponding to the response result is the i-th algorithm model. <i<N; The verification module 43 is used to statistically analyze the types of response results corresponding to each algorithm model, obtain statistical results, and obtain the verification results of each algorithm model based on the statistical results and the assigned values.

[0057] Optionally, the verification device 40 further includes: The confusion matrix construction module is used to construct a confusion matrix based on the i-th algorithm model corresponding to the response result and the j-th algorithm model corresponding to the user request if the algorithm model corresponding to the user request is the j-th algorithm model.

[0058] The visualization module is used to visualize the confusion matrix using preset visualization tools, and obtain a visualized heatmap of the verification results for each algorithm model.

[0059] Optionally, the user request generation module 41 mentioned above includes: The probability value acquisition unit is used to acquire N algorithm models from the database, use a preset simulation tool to assign probability values ​​to the N algorithm models, and the sum of the probability values ​​assigned to the N algorithm models is 1.

[0060] The allocation value acquisition unit is used to obtain the allocation value for each algorithm model based on the target value and the probability value of each algorithm model.

[0061] Optionally, the response result acquisition module 42 mentioned above includes: The algorithm identifier acquisition unit is used to send all user requests to the database and obtain the algorithm identifier corresponding to the algorithm model returned by the database based on the corresponding user request.

[0062] The statistics unit is used to count the number of algorithm identifiers and obtain the number of each algorithm identifier.

[0063] The user request response result determination unit is used to determine the database response result based on the corresponding user request according to the number of each algorithm identifier.

[0064] Optionally, the verification module 43 mentioned above includes: The target type determination unit is used to determine the target type if the algorithm model corresponding to the response result is the same as the algorithm model requested by the user.

[0065] The accuracy determination unit is used to count the number of response results of any given algorithm model that are of the target type, determine the number as the response value of the algorithm model, and obtain the accuracy of the algorithm model in the database based on the response values ​​and target values ​​of all algorithm models, thus determining the verification result of the algorithm model.

[0066] Optionally, the verification module 43 mentioned above includes: The statistical result determination unit for each algorithm model is used to statistically analyze the types of response results corresponding to each algorithm model and obtain the statistical results for each algorithm model.

[0067] The accuracy determination unit is used to count the number of times each algorithm model's response result is of the target type, and to calculate the accuracy of each algorithm model in the database based on the statistical results corresponding to each algorithm model.

[0068] The sensitivity determination unit is used to calculate the sensitivity of each algorithm model in the database based on the number of target types in the response results of each algorithm model and the corresponding allocation value for each algorithm model.

[0069] The F1 Score determination unit is used to determine the F1 Score of each algorithm model based on the accuracy and sensitivity of each algorithm model in the database, and obtain the verification results.

[0070] It should be noted that the information interaction and execution process between the above-mentioned units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0071] Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Figure 5 As shown, the computer device of this embodiment includes: at least one processor ( Figure 5 Only one is shown in the diagram), a memory, and a computer program stored in the memory and capable of running on at least one processor. When the processor executes the computer program, it implements the steps in the embodiments of the method for verifying the accuracy of any of the above-described database algorithm models.

[0072] This computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 5The examples of computer devices are merely examples and do not constitute a limitation on computer devices. Computer devices may include more or fewer components than shown in the illustration, or combinations of certain components, or different components, such as network interfaces, displays, and input devices.

[0073] The processor referred to can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0074] Memory includes readable storage media, internal memory, etc., wherein internal memory can be the RAM of a computer device, providing an environment for the operation of the operating system and computer-readable instructions stored in the readable storage media. The readable storage media can be the hard drive of a computer device, or in other embodiments, it can be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, memory can include both internal storage units and external storage devices of the computer device. Memory is used to store the operating system, applications, bootloader, data, and other programs, such as program code for computer programs. Memory can also be used to temporarily store data that has been output or will be output.

[0075] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the functions described above can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention can implement all or part of the processes in the methods of the above embodiments by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code, a recording medium, a computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0076] The present invention can implement all or part of the processes in the methods of the above embodiments, or it can be accomplished by a computer program product. When the computer program product is run on a computer device, the computer device executes the steps in the above method embodiments.

[0077] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0078] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0079] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0080] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0081] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for verifying the accuracy of a database algorithm model, characterized in that, The verification method comprises: Acquiring N algorithm models from the database, generating an allocation value for each of the N algorithm models, and generating user requests for each algorithm model, wherein the number of user requests for each algorithm model is identical to the allocation value of the corresponding algorithm model, and N is an integer greater than 1; the allocation value is the number of user requests corresponding to the algorithm model; Acquiring N algorithm models from the database, and generating an allocation value for each of the N algorithm models comprises: Acquiring N algorithm models from the database, and allocating a probability value to each of the N algorithm models by using a preset simulation tool, wherein the sum of the probability values allocated to the N algorithm models is 1; Calculating the allocation value of each algorithm model according to the total target value of user requests to be generated and the probability value of each algorithm model; Sending all user requests to the database, acquiring response results of the database based on the corresponding user requests, and determining that the response result is of the i-th type if the algorithm model corresponding to the response result is the i-th algorithm model, wherein 0<i<N; sending all user requests to the database and acquiring response results of the database based on the corresponding user requests comprises: Sending all user requests to the database, and acquiring the algorithm identifier corresponding to the algorithm model returned by the database based on the corresponding user requests; Counting the algorithm identifiers to acquire the number of each algorithm identifier; Determining the response results of the database based on the corresponding user requests according to the number of each algorithm identifier; Counting types of response results corresponding to each algorithm model to obtain a statistical result, and obtaining a verification result of each algorithm model according to the statistical result and the allocation values; Counting types of response results corresponding to each algorithm model to obtain a statistical result, and obtaining a verification result of each algorithm model according to the statistical result and the allocation values comprises: Counting types of response results corresponding to each algorithm model to obtain a statistical result corresponding to each algorithm model; Counting the number of response results of each algorithm model that are of a target type, and calculating the precision of each algorithm model in the database according to the statistical result corresponding to each algorithm model; Calculating the sensitivity of each algorithm model in the database according to the number of response results of each algorithm model that are of the target type and the allocation value corresponding to each algorithm model; Determining an F1 Score of each algorithm model according to the precision and the sensitivity of each algorithm model in the database, to obtain a verification result.

2. The verification method as described in claim 1, characterized in that, After counting types of response results corresponding to each algorithm model to obtain a statistical result, and obtaining a verification result of each algorithm model according to the statistical result and the allocation values, the method further comprises: If the algorithm model corresponding to a user request is the j-th algorithm model, constructing a confusion matrix based on the i-th algorithm model corresponding to the response result and the j-th algorithm model corresponding to the user request, wherein 0<j<N; Performing visualization processing on the confusion matrix of the verification result of each algorithm model by using a preset visualization tool, to obtain a visualization heat map of the verification result of each algorithm model.

3. The verification method as described in claim 1, characterized in that, Counting types of response results corresponding to each algorithm model to obtain a statistical result, and obtaining a verification result of each algorithm model according to the statistical result and the allocated value comprises: if the algorithm model corresponding to the response result is the algorithm model corresponding to the user request, the response result is of a target type; for any algorithm model, counting the number of response results of the algorithm model that are of the target type, determining the number as a response value of the algorithm model, obtaining an accuracy rate of the algorithm models in the database according to response values and a target total value of all algorithm models, and determining the verification result of the algorithm model.

4. A device for verifying the accuracy of a database algorithm model, characterized in that, The verification device comprises: a user request generation module, configured to acquire N algorithm models in the database, generate an allocated value for each of the N algorithm models, and generate a user request for each algorithm model, wherein the number of user requests of each algorithm model is the same as the allocated value of the corresponding algorithm model, and N is an integer greater than 1; and the allocated value is the number of user requests of the corresponding algorithm model; acquiring N algorithm models in the database and generating allocated values for the N algorithm models comprises: acquiring N algorithm models in the database, and allocating a probability value to each of the N algorithm models by using a preset simulation tool, wherein the sum of the probability values allocated to the N algorithm models is 1; calculating the allocated value of each algorithm model according to a total target value of user requests that need to be generated and the probability value of each algorithm model; a response result acquisition module, configured to send all user requests to the database, acquire a response result based on the corresponding user request from the database, and if the algorithm model corresponding to the response result is an i-th algorithm model, determine that the response result is of an i-th type, 0 < i < N; sending all user requests to the database and acquiring a response result based on the corresponding user request from the database comprises: sending all user requests to the database, and acquiring an algorithm identifier corresponding to the algorithm returned by the database based on the corresponding user request; counting the algorithm identifiers, and acquiring the number of each algorithm identifier; determining a response result based on the corresponding user request from the database according to the number of each algorithm identifier; a verification module, configured to count types of response results corresponding to each algorithm model to obtain a statistical result, and obtain a verification result of each algorithm model according to the statistical result and the allocated value; counting types of response results corresponding to each algorithm model to obtain a statistical result, and obtaining a verification result of each algorithm model according to the statistical result and the allocated value comprises: counting types of response results corresponding to each algorithm model, to obtain a statistical result corresponding to each algorithm model; counting the number of response results that are of the target type for each algorithm model, and calculating a precision rate of each algorithm model in the database according to the statistical result corresponding to each algorithm model; calculating a sensitivity of each algorithm model in the database according to the number of response results that are of the target type for each algorithm model and the allocated value corresponding to each algorithm model; Based on the accuracy and sensitivity of each algorithm model in the database, the F1 score of each algorithm model is determined, and the verification results are obtained.

5. The verification device as described in claim 4, characterized in that, The verification device further includes: The confusion matrix construction module is used to construct a confusion matrix based on the i-th algorithm model corresponding to the response result and the j-th algorithm model corresponding to the user request if the algorithm model corresponding to the user request is the j-th algorithm model. The visualization module is used to visualize the confusion matrix using preset visualization tools, and obtain a visualized heatmap of the verification results for each algorithm model.

6. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a method for verifying the accuracy of a database algorithm model as described in any one of claims 1 to 3.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements a method for verifying the accuracy of a database algorithm model as described in any one of claims 1 to 3.

Images