Audit quality assessment methods, devices, equipment, media and program products

Abstract

The present application relates to an audit quality assessment method, device, equipment, medium and program product, which can be applied to the field of financial technology or other related fields, including: first obtaining multiple assessment information of the target document, and then using the information assessment model to quantitatively analyze the correspondence between the elements to be assessed and the content of the elements in each assessment information, and obtain the assessment quantitative value of each assessment information, and determine the quality pass status of the audit result of the target document according to the assessment quantitative value of each assessment information. This method reduces the cost of manual audit, avoids the subjective factors of manual audit, and ensures the audit quality.

Description

Technical Field

[0001] The present application relates to the field of financial technology, and in particular to an audit quality assessment method, device, equipment, medium and program product. Background Art

[0002] In international trade, international documents are important credentials required for the transportation, commodity inspection, payment, insurance, and settlement of foreign exchange of goods. Therefore, banks and other financial institutions pay special attention to the quality of document review.

[0003] At present, document review is to identify international documents through artificial intelligence technology, and then evaluate the accuracy of the review results through offline manual review. The reviewers review the automatically extracted document elements one by one to determine whether the content of the elements is accurate and complete.

[0004] However, the labor cost of evaluating the accuracy of the review results through offline manual review is high, and manual review has subjective factors, and the quality of the review cannot be guaranteed. Summary of the invention

[0005] Based on this, it is necessary to provide an audit quality assessment method, device, equipment, medium and program product to address the above-mentioned technical issues, reduce the cost of manual audits, avoid the subjective factors of manual audits, and ensure the quality of audits.

[0006] In a first aspect, the present application provides a method for evaluating audit quality, the method comprising:

[0007] Acquire multiple evaluation information of the target document; each evaluation information includes a set of correspondences between elements to be evaluated and the content of the elements; the multiple evaluation information of the target document represents the audit results generated after the document audit device audits the target document;

[0008] Through the information evaluation model, the corresponding relationship between the elements to be evaluated and the content of the elements in each evaluation information is quantitatively analyzed to obtain the evaluation quantitative value of each evaluation information;

[0009] The quality pass status of the audit result of the target document is determined based on the evaluation quantitative value of each evaluation information.

[0010] In one embodiment, the information evaluation model includes a plurality of unit information evaluation models, each unit information evaluation model corresponding to a factor to be evaluated;

[0011] Then, through the information evaluation model, the corresponding relationship between the elements to be evaluated and the content of the elements in each evaluation information is quantitatively analyzed to obtain the evaluation quantitative value of each evaluation information, including:

[0012] Through the unit information evaluation model corresponding to each to-be-evaluated element, the corresponding relationship between each to-be-evaluated element and the element content is quantitatively analyzed to obtain the evaluation quantitative value of each evaluation information.

[0013] In one embodiment, determining the quality pass status of the audit result of the target document according to the evaluation quantified value of each evaluation information includes:

[0014] Obtain the weights corresponding to the elements to be evaluated in each evaluation information;

[0015] Determine the total quantitative evaluation value of the target document based on the quantitative evaluation value of each evaluation information and the weight of the elements to be evaluated in each evaluation information;

[0016] Determine the quality pass status of the audit results of the target document based on the total quantitative assessment value.

[0017] In one embodiment, determining the quality pass status of the audit result of the target document according to the total evaluation quantification value includes:

[0018] If the total evaluation quantification value is greater than or equal to the first preset threshold, the quality pass status of the audit result of the target document is determined to be fully passed;

[0019] If the total evaluation quantification value is less than or equal to the second preset threshold, the quality pass status of the audit result of the target document is determined to be failed; the second preset threshold is less than the first preset threshold;

[0020] If the total evaluation quantification value is greater than the second preset threshold and less than the first preset threshold, the quality pass status of the audit result of the target document is determined to be pending pass.

[0021] In one embodiment, the method further comprises:

[0022] If the quality pass status of the audit result of the target document is fully passed, it is determined that the audit result of the target document does not need to be reviewed;

[0023] If the quality pass status of the audit result of the target document is Failed, it is determined that the audit result of the target document needs to be reviewed as a whole;

[0024] If the quality approval status of the audit result of the target document is pending approval, the audit result of the target document needs to be reviewed by sampling.

[0025] In one embodiment, obtaining multiple evaluation information of the target document includes:

[0026] In response to the evaluation request of the target document, the evaluation request is sent to the target task process; the task process of each evaluation request is allocated according to the accumulation of evaluation requests of each task process;

[0027] Obtain multiple evaluation information of the target document in the evaluation request through the target task process.

[0028] In one embodiment, the process of building the information assessment model includes:

[0029] Obtaining sample information corresponding to multiple sample elements to be evaluated; the sample information includes positive sample information and negative sample information; the positive sample information indicates a sample that completely meets the corresponding element information to be evaluated, and the negative sample information indicates a sample that does not completely meet the corresponding element information to be evaluated;

[0030] According to the sample information corresponding to each sample element to be evaluated, the initial information evaluation model is trained to obtain the information evaluation model.

[0031] In one embodiment, obtaining sample information corresponding to a plurality of sample elements to be evaluated includes:

[0032] Obtain historical inventory review data;

[0033] Based on the historical inventory of document review data, determine the positive sample information corresponding to each sample element to be evaluated;

[0034] The positive sample information is simulated to obtain the negative sample information.

[0035] In one embodiment, the multiple evaluation information of the target document includes the document type, the elements to be evaluated and the corresponding element content extracted by the document review device from the information in the image text of the target document.

[0036] In a second aspect, the present application also provides an audit quality assessment device, the device comprising:

[0037] The acquisition module is used to acquire multiple evaluation information of the target document; each evaluation information includes a set of correspondences between elements to be evaluated and the content of the elements; the multiple evaluation information of the target document represents the audit results generated after the document audit device audits the target document;

[0038] The analysis module is used to perform quantitative analysis on the correspondence between the elements to be evaluated and the content of the elements in each evaluation information through the information evaluation model, so as to obtain the evaluation quantitative value of each evaluation information;

[0039] The determination module is used to determine the quality pass status of the audit result of the target document according to the evaluation quantitative value of each evaluation information.

[0040] In a third aspect, an embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any one of the methods provided in the embodiment of the first aspect are implemented.

[0041] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any one of the methods provided in the embodiment of the first aspect above are implemented.

[0042] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of any one of the methods provided in the embodiments of the first aspect above.

[0043] The embodiments of the present application provide an audit quality assessment method, device, equipment, medium and program product, which obtain multiple assessment information of a target document, and then use an information assessment model to quantitatively analyze the correspondence between the elements to be assessed and the element content in each assessment information to obtain an assessment quantitative value of each assessment information, and determine the quality pass status of the audit result of the target document based on the assessment quantitative value of each assessment information. In the method, each evaluation information includes a set of correspondences between elements to be evaluated and element contents, and multiple evaluation information of the target document represent the audit results of the target document after the audit equipment has audited it. Therefore, the elements to be evaluated and the element contents in each evaluation information are quantitatively analyzed through the information evaluation model to obtain the evaluation quantitative value of each evaluation information. The evaluation quantitative value of each evaluation information is equivalent to the quantitative value of the audit result obtained after the audit equipment has audited the target document. The quantitative value can intuitively measure the quality of the audit result of each evaluation information. The evaluation information is quantified through the information evaluation model, which avoids the subjective factors of manual audit and ensures the quality of the audit result of the target document. Therefore, according to the evaluation quantitative value of each evaluation information, the quality pass status of the audit result of the target document is determined, and the self-assessment of the quality of the audit result of the target document is realized, avoiding the link of directly reviewing the audit result of the target document by manual labor, thereby reducing labor costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 An application environment diagram of an audit quality assessment method in one embodiment;

[0045] Figure 2 A schematic diagram of a flow chart of an audit quality assessment method in one embodiment;

[0046] Figure 3 A flowchart of an audit quality assessment method in another embodiment;

[0047] Figure 4 A flowchart of an audit quality assessment method in another embodiment;

[0048] Figure 5 A flowchart of an audit quality assessment method in another embodiment;

[0049] Figure 6 A flowchart of an audit quality assessment method in another embodiment;

[0050] Figure 7 A flowchart of an audit quality assessment method in another embodiment;

[0051] Figure 8 A flowchart of an audit quality assessment method in another embodiment;

[0052] Fig. 9 A schematic diagram of the structure of an audit quality assessment system in one embodiment;

[0053] Fig.10 A schematic diagram of the structure of an audit quality assessment system in another embodiment;

[0054] Fig.11 It is a structural block diagram of an audit quality assessment device in one embodiment;

[0055] Fig.12 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0057] The audit quality assessment method provided in the embodiment of the present application can be as follows Figure 1 In the application environment shown. Among them, the single-checking device 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the single-checking device 102 can be, but is not limited to, various personal computers, laptops, smart phones, and tablet computers. The server 104 can be implemented with an independent server or a server cluster consisting of multiple servers.

[0058] At present, many financial institutions use artificial intelligence technology to achieve intelligent review in order to reduce labor costs. However, the quality of intelligent review results is greatly affected by image quality and recognition model accuracy, and the quality of review varies greatly.

[0059] In the existing technology, the accuracy of intelligent document review results is mainly evaluated through offline manual review. The reviewers review the automatically extracted document elements one by one to determine whether the elements automatically extracted from the electronic image are correct. The offline manual review method requires high labor costs to evaluate the accuracy of intelligent document review results, which greatly reduces the benefits of intelligent document review. In addition, manual review cannot avoid human interference, and the timeliness and quality of the review are entirely dependent on the experience and business proficiency of the reviewers, which easily leads to uneven review efficiency and quality.

[0060] Based on this, the embodiments of the present application provide an audit quality assessment method, device, equipment, medium and program product, which reduces the cost of manual audit, avoids the subjective factors of manual audit, and ensures the quality of audit.

[0061] The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments.

[0062] In one embodiment, a method for evaluating audit quality is provided. Figure 1 Taking the application environment in as an example, this embodiment involves a specific process of obtaining multiple evaluation information of a target document, and then quantitatively analyzing the correspondence between the elements to be evaluated and the content of the elements in each evaluation information through an information evaluation model to obtain an evaluation quantitative value of each evaluation information, and determining the quality pass status of the audit result of the target document according to the evaluation quantitative value of each evaluation information, such as Figure 2 As shown, this embodiment includes the following steps:

[0063] S201, obtaining multiple evaluation information of the target document; each evaluation information includes a set of correspondence between elements to be evaluated and element contents; the multiple evaluation information of the target document represents the audit results generated after the document audit device audits the target document.

[0064] In international trade, documents are important credentials required for the transportation, commodity inspection, payment, insurance, and settlement of foreign exchange of goods. Common documents include financial documents, commercial documents, shipping documents, insurance documents, and other documents. Financial documents include bills of exchange, promissory notes, and checks. Commercial documents include commercial invoices and customs invoices. Shipping documents include ocean bills of lading, charter party bills of lading, multimodal transport documents, air waybills, and other documents include commodity inspection documents, certificates of origin, and other documents (document mailing certificate, sample mailing certificate, shipping notice, vessel age certificate, etc.).

[0065] After receiving the target document, the document review device reviews the target document and determines the review result. The document review device may review the target document by performing structured extraction of text information in the electronic image of the target document to obtain element information required for the evaluation of various types of documents. The review result includes the elements and element contents extracted from the target document.

[0066] The elements and the element contents are taken as one evaluation information. The target document includes multiple elements. Therefore, the target document includes multiple evaluation information. For example, if the target document type is "ocean bill of lading", the elements to be evaluated of the target document include "bill of lading number", "shipping company name", "carrier", "loading port", "unloading port", "departure date", "commodity name", etc., and each element to be evaluated corresponds to an element content. Then the multiple evaluation information of the target document include: the bill of lading number element and the content corresponding to the bill of lading number element, the shipping company name element and the content corresponding to the shipping company name element, the carrier element and the content corresponding to the carrier element, the loading port element and the content corresponding to the loading port element, the unloading port element and the content corresponding to the unloading port element, the departure date element and the content corresponding to the departure date element, the commodity name element and the content corresponding to the commodity name element.

[0067] S202, using the information evaluation model, quantitatively analyzing the correspondence between the elements to be evaluated and the element contents in each evaluation information, and obtaining an evaluation quantitative value of each evaluation information.

[0068] Quantitative analysis is to express some non-specific and vague factors with specific data, so as to achieve the purpose of analysis and comparison, which can help us measure risks and returns more conveniently and intuitively.

[0069] The information evaluation model is a model that evaluates whether the audit results of the target document by the audit equipment are accurate, that is, the evaluation results are obtained by evaluating whether the element content of the elements to be evaluated in each evaluation information is accurate. By quantifying the evaluation results, the audit quality of the target document audit results can be intuitively determined.

[0070] The information evaluation model is used to perform quantitative analysis on the correspondence between the elements to be evaluated and the element contents in each evaluation information to obtain the evaluation quantitative value of each evaluation information. It should be noted that one evaluation information corresponds to one evaluation quantitative value, and multiple evaluation information corresponds to multiple evaluation quantitative values.

[0071] In one embodiment, the elements to be evaluated and the corresponding element contents in each evaluation information are respectively input into the information evaluation model to obtain the evaluation quantitative value of each evaluation information.

[0072] The above describes calling the information evaluation model to determine the evaluation quantization value of each evaluation information. There is also an embodiment. If the element content of the element to be evaluated in the evaluation information is empty, it is not necessary to use the information evaluation model to perform quantitative analysis on the correspondence between the element to be evaluated and the element content in the evaluation information, and the evaluation quantization value of the evaluation information is directly determined to be 0.

[0073] S203, determining the quality pass status of the audit result of the target document according to the evaluation quantification value of each evaluation information.

[0074] The quality passing status may be an evaluation result obtained after evaluating the target document review result in the document review device.

[0075] In one embodiment, the quality pass status of the audit result of the target document can be determined by a neural network model. Specifically, the evaluation quantization value of each evaluation information is used as the input of the neural network model. Through the analysis of the neural network, the quality pass status of the audit result of the target document is finally output. The quality pass status can be "pass" or "fail". "Pass" indicates that the quality of the audit result of the target document meets the standards, and "fail" indicates that the quality of the audit result of the target document does not meet the standards.

[0076] In the above-mentioned audit quality assessment method, multiple assessment information of the target document is obtained, and then the correspondence between the elements to be assessed and the element content in each assessment information is quantitatively analyzed through the information assessment model to obtain the assessment quantitative value of each assessment information, and the quality pass status of the audit result of the target document is determined based on the assessment quantitative value of each assessment information. In the method, each evaluation information includes a set of correspondences between elements to be evaluated and element contents, and multiple evaluation information of the target document represent the audit results of the target document after the audit equipment has audited it. Therefore, the elements to be evaluated and the element contents in each evaluation information are quantitatively analyzed through the information evaluation model to obtain the evaluation quantitative value of each evaluation information. The evaluation quantitative value of each evaluation information is equivalent to the quantitative value of the audit result obtained after the audit equipment has audited the target document. The quantitative value can intuitively measure the quality of the audit result of each evaluation information. The evaluation information is quantified through the information evaluation model, which avoids the subjective factors of manual audit and ensures the quality of the audit result of the target document. Therefore, according to the evaluation quantitative value of each evaluation information, the quality pass status of the audit result of the target document is determined, and the self-assessment of the quality of the audit result of the target document is realized, avoiding the link of directly reviewing the audit result of the target document by manual labor, thereby reducing labor costs.

[0077] In one embodiment, the information evaluation model includes multiple unit information evaluation models, each unit information evaluation model corresponds to an element to be evaluated; the information evaluation model is used to quantitatively analyze the correspondence between the element to be evaluated and the element content in each evaluation information to obtain an evaluation quantitative value of each evaluation information, including: through the unit information evaluation model corresponding to each element to be evaluated, the correspondence between each element to be evaluated and the element content is quantitatively analyzed to obtain an evaluation quantitative value of each evaluation information.

[0078] The information evaluation model includes a plurality of unit information evaluation models, each of which corresponds to an element to be evaluated; there is a one-to-one correspondence between the element to be evaluated and the unit information evaluation model.

[0079] When it is necessary to evaluate the element to be evaluated, the information evaluation model corresponding to the element to be evaluated is called, and the element content corresponding to the element to be evaluated is quantitatively analyzed to determine the evaluation quantitative value of the element to be evaluated.

[0080] For example, if the factors to be evaluated include product name and bill of lading number, when evaluating the product name and bill of lading number, the product name factor calls the product name evaluation model, and the bill of lading number factor calls the bill of lading number evaluation model. The product name evaluation model performs quantitative analysis on the product name content to obtain the evaluation quantitative value of the product name factor; the bill of lading number evaluation model performs quantitative analysis on the bill of lading number content to obtain the evaluation quantitative value of the bill of lading number factor.

[0081] It should be noted that the information evaluation models corresponding to different elements to be evaluated are different.

[0082] Based on the above-mentioned embodiment, the following describes an embodiment of determining the quality pass status of the audit result of the target document according to the evaluation quantification value of each evaluation information. In one embodiment, Figure 3 As shown, this embodiment includes the following steps:

[0083] S301, obtaining the weights corresponding to the elements to be evaluated in each evaluation information.

[0084] When the server obtains multiple evaluation information of the target document, it also obtains the document type corresponding to the target document.

[0085] Before conducting an audit quality assessment on the audit results of the documents, a structured decomposition will be performed on the different bill types of the documents. The rule is to decompose each document type into the required element fields during the intelligent document review. The audit quality assessment of the audit results of the documents is mainly to evaluate whether the element content automatically extracted during the document review is accurate.

[0086] Different document types correspond to different elements, so the importance of the elements of different document types is also different. Therefore, the corresponding weight value is set according to the importance of the element in the corresponding document type, and the configuration information is stored in the relational database.

[0087] It should be noted that the elements in different document types are different. Correspondingly, the weights corresponding to the same elements in different document types may also be different, and the weights of elements in the same document type are also different.

[0088] The method of obtaining the weights corresponding to the elements to be evaluated in each evaluation information is to obtain the weights corresponding to each evaluation element in the database according to the document type of the target document.

[0089] In one embodiment, the weight corresponding to each to-be-evaluated element is directly queried in a database according to each to-be-evaluated element.

[0090] S302, determining the total evaluation quantification value of the target document according to the evaluation quantification value of each evaluation information and the weight of the to-be-evaluated element in each evaluation information.

[0091] The evaluation quantitative value of each evaluation information and the weight of the element to be evaluated in each evaluation information are weighted and summed to determine the total evaluation quantitative value of the target document. For example, there are two evaluation information in the target document, namely evaluation information 1 and evaluation information 2. The evaluation quantitative value of evaluation information 1 is 80, and the element to be evaluated in evaluation information 1 is the product name, and the weight of the product name is 0.4. The evaluation quantitative value of evaluation information 2 is 70, and the element to be evaluated in evaluation information 2 is the bill of lading number, and the weight of the bill of lading number is 0.8. Then the total evaluation quantitative value of the target document is 80×0.4+70×0.8=88.

[0092] In one embodiment, the method for determining the total quantitative evaluation value of the target document can also be determined through a preset neural network model. Specifically, the quantitative evaluation value of each evaluation information and the weight of the element to be evaluated in each evaluation information are used as input of the neural network model. Through the analysis of the neural network model, the total quantitative evaluation value of the target document is finally output.

[0093] S303, determining the quality pass status of the audit result of the target document according to the total evaluation quantification value.

[0094] Different assessment levels are set according to different document types. For example, if the total assessment value of "Ocean Bill of Lading" is greater than or equal to 80, the corresponding quality status is "Pass". If the total assessment value is less than 80, the corresponding quality status is "Fail".

[0095] In actual applications, different document types correspond to different assessment levels.

[0096] Therefore, if the document type of the target document obtained above is "ocean bill of lading type" and the total evaluation quantitative value is 88, the quality pass status of the audit result of the target document is "passed", indicating that the quality of the audit result of the target document is good.

[0097] In one embodiment, the quality passing status of the audit results of the target document can also be determined by a preset evaluation algorithm. Specifically, the total evaluation quantification value is used as the input of the preset evaluation algorithm, and the quality passing status of the audit results of the target document is directly output by running the evaluation algorithm.

[0098] In the above audit quality assessment method, the weights corresponding to the elements to be assessed in each assessment information are first obtained, and the total assessment quantification value of the target document is determined based on the assessment quantification value of each assessment information and the weights of the elements to be assessed in each assessment information, and the quality pass status of the audit result of the target document is determined based on the assessment quantification value of each assessment information and the weights corresponding to the elements to be assessed in each assessment information. Because different elements to be assessed have different corresponding weights, and the weights are set according to the importance of the elements to be assessed, this method improves the accuracy of the quality assessment of the audit results of the target documents.

[0099] In one embodiment, Figure 4 As shown, according to the total evaluation quantification value, determining the quality pass status of the audit result of the target document includes the following steps:

[0100] S401, if the total evaluation quantification value is greater than or equal to the first preset threshold, the quality pass status of the audit result of the target document is determined to be fully passed.

[0101] The first preset threshold is an evaluation mechanism pre-set for the document type of the target document. Different single-needle types correspond to different first preset thresholds. In the embodiment of the present application, no limitation is made to the first preset threshold. In actual applications, the first preset threshold is assigned according to actual conditions.

[0102] If the total evaluation quantification value of the target document is greater than or equal to the first preset threshold, the audit result of the target document is "excellent", and the quality pass status of the audit result of the target document is determined to be fully passed.

[0103] If the quality pass status of the audit result of the target document is fully passed, it is determined that the audit result of the target document does not need to be reviewed. The audit result of the target document does not need to be reviewed, which means that the audit result of the target document is good and does not need to be reviewed again, that is, it can be regarded that the evaluation information of the target document is correct.

[0104] S402: If the total evaluation quantification value is less than or equal to the second preset threshold, the quality pass status of the audit result of the target document is determined to be failed.

[0105] Among them, the second preset threshold is smaller than the first preset threshold, and the second preset threshold is also an evaluation mechanism pre-set for the single-needle type of the target document. Different single-needle types correspond to different second preset thresholds, and in the embodiment of the present application, no limitation is made on the second preset threshold. In actual applications, the second preset threshold is assigned according to actual conditions.

[0106] If the total evaluation quantization value of the target single needle is less than or equal to the second preset threshold, the audit result of the target single needle is "unqualified", and the quality pass status of the audit result of the target document is determined to be unqualified.

[0107] If the quality pass status of the audit result of the target document is Fail, it is determined that the audit result of the target document needs to be reviewed as a whole; the audit result of the target document needs to be reviewed as a whole, which means that the audit result of the target document is poor and needs to be reviewed again, that is, it can be regarded as that there are at least multiple errors in the evaluation information of the target document, which needs to be reviewed again.

[0108] The re-examination can be carried out by manual review or by re-examination and re-evaluation of the target document.

[0109] S403: If the total evaluation quantification value is greater than the second preset threshold and less than the first preset threshold, the quality pass status of the audit result of the target document is determined to be pending pass.

[0110] If the total evaluation quantification value of the target document is greater than the second preset threshold and less than the first preset threshold, the audit result of the target document is "good", and the quality pass status of the audit result of the target document is determined to be pending pass.

[0111] If the quality pass status of the audit result of the target document is pending pass, it is determined that the audit result of the target document needs to be reviewed by sampling; the audit result of the target document needs to be reviewed by sampling, which means that the audit result of the target document is general, and the evaluation information in the target document can be re-reviewed by sampling; the re-review method can be manual review or re-review of the sampled evaluation information in the target document.

[0112] In the above audit quality assessment method, if the total evaluation quantification value is greater than or equal to the first preset threshold, the quality pass status of the audit result of the target document is determined to be fully passed; if the total evaluation quantification value is less than or equal to the second preset threshold, the quality pass status of the audit result of the target document is determined to be not passed; the second preset threshold is less than the first preset threshold; if the total evaluation quantification value is greater than the second preset threshold and less than the first preset threshold, the quality pass status of the audit result of the target document is determined to be pending pass. In this method, according to the total evaluation quantification value of the target document, the quality pass status of the audit result of the target document is determined by the evaluation level, and it can be determined whether to conduct a review based on the quality pass status, avoiding unnecessary reviews and reducing labor costs.

[0113] In one embodiment, Figure 5 As shown, multiple evaluation information of the target document is obtained, including:

[0114] S501, in response to an evaluation request of a target document, sending the evaluation request to a target task process; the task process for each evaluation request is allocated according to the accumulation of evaluation requests of each task process.

[0115] From the time the review is initiated, there are five statuses for a document: "pending review", "under review", "not reviewed yet", "under review", and "assessed".

[0116] Among them, the "pending review" status means that after submitting the document to the review request, a review request with the status of "pending review" is obtained; when the document in the "pending review" status is being reviewed, the status of the document is "under review"; after the review request is reviewed, an evaluation request with the status of "pending evaluation" is obtained; when the document in the "pending evaluation" status is being evaluated, the status of the document is "under evaluation"; after the evaluation request is evaluated, the document with the status of "evaluated" is obtained.

[0117] Therefore, before evaluating the audit result of the target document, the server will receive an evaluation request for the target document and store the evaluation request for the target document in a relational database.

[0118] The master control process in the server obtains the evaluation request of the target document from the database. The responsibility of the master control process is to implement the task allocation, assign the task to the task process, and the task process completes the corresponding task. Each task process runs independently and in parallel.

[0119] The way to send the evaluation request to the target task process is that the master control process allocates the evaluation request to the least busy task process, that is, the target task process, according to the busy and idle status reported by each task process; the busy and idle status is determined by the number of evaluation requests currently accumulated in each task process, and the more accumulated, the busier it is.

[0120] Optionally, each task process actively sends the current accumulation status of its own evaluation requests to the main control process through the heartbeat mechanism, that is, each task process will report the current accumulation status of its own evaluation requests to the main control process in a preset time period; and, it can also be horizontally expanded or reduced according to the changes in the demand for evaluation requests. Specifically, when the demand for evaluation requests is large, the task processes can be increased to improve the evaluation efficiency of the audit results. When the demand for evaluation requests is small, the task processes can be reduced to improve the utilization rate of the equipment.

[0121] S502, obtaining multiple evaluation information of the target document in the evaluation request through the target task process.

[0122] After the target task process obtains the evaluation request, it can obtain the document type of the target document in the evaluation request, and according to the document type, obtain the to-be-evaluated elements corresponding to the document type from a preset database, traverse the element contents in the evaluation request according to the to-be-evaluated elements, and take the correspondence between the to-be-evaluated elements and the element contents corresponding to the to-be-evaluated elements as one evaluation information; multiple to-be-evaluated elements and element contents correspond to multiple evaluation information.

[0123] Optionally, multiple evaluation information of the target document may also be directly obtained from an evaluation request in the target process, where the evaluation request includes multiple evaluation information of the target document.

[0124] In the above audit quality assessment method, in response to the assessment request of the target document, the assessment request is sent to the target task process, and the task process of each assessment request is allocated according to the accumulation of assessment requests of each task process, and multiple assessment information of the target document in the assessment request is obtained through the target task process. In this method, the target task process is allocated according to the accumulation of assessment requests of each task process, which improves the assessment efficiency of the audit results of the target document.

[0125] Based on the above information evaluation model, the correspondence between the elements to be evaluated and the content of the elements in each evaluation information is quantitatively analyzed to obtain the evaluation quantitative value of each evaluation information. The construction process of the information evaluation model is described below through an embodiment. In one embodiment, Figure 6 As shown, this embodiment includes the following steps:

[0126] S601, obtaining sample information corresponding to multiple sample elements to be evaluated; the sample information includes positive sample information and negative sample information; the positive sample information represents samples that fully conform to the corresponding element information to be evaluated, and the negative sample information represents samples that do not fully conform to the corresponding element information to be evaluated.

[0127] The information evaluation model mainly determines the probability of the extracted content being the current element from the part of speech and semantics, so the sample information used to train the information evaluation model directly determines the accuracy of the model recognition. For the embodiment of the present application, it is mainly to determine whether the element content of the element to be evaluated is the element, so the accuracy of the information evaluation model mainly depends on the quality and quantity of the sample information of the corresponding element. Sample information includes positive sample information and negative sample information. Positive sample information is a sample that fully conforms to the corresponding element information to be evaluated, and negative sample information represents a sample that does not conform to the corresponding element information to be evaluated. Taking the "sea bill of lading" category as an example, "frozen beef" and "textiles" are positive samples of the element "trade name", and "frozen" and "textiles" are negative samples of the element "trade name".

[0128] Sample information corresponding to multiple sample elements to be evaluated can be obtained from the sample data set. The sample data set is logically divided according to different document types and stored in the data lake. The data lake is a big data platform built based on an open source distributed computing and storage framework. The open source distributed computing and storage framework can be Hadoop.

[0129] The positive sample information and the negative sample information in the sample data set may be stored according to the audit results of the historical audit.

[0130] It should be noted that the sample information corresponding to each sample element to be evaluated is different. For example, the sample information corresponding to the product name element is information of the product name type, while the sample information corresponding to the date element is information of the date type.

[0131] S602: Train the initial information evaluation model according to the sample information corresponding to each sample element to be evaluated to obtain an information evaluation model.

[0132] According to the sample information corresponding to each sample element to be evaluated, the initial information evaluation model is trained until the initial information evaluation model meets the preset convergence conditions, then the unit information evaluation model corresponding to each sample element to be evaluated is obtained, and all unit information evaluation models are determined as information evaluation models.

[0133] The initial information evaluation model may be a fast text classification (Fasttext) model.

[0134] After obtaining the sample information corresponding to the above-mentioned sample elements to be evaluated, the feature values ​​are extracted, and the sample information is processed by labeling, with the positive sample marked as 1, indicating that it is the sample element, and the negative sample marked as 0, indicating that it is not the element; based on the labeled sample information, the Fasttext model is trained to obtain the information evaluation model.

[0135] When the information evaluation model is applied, the corresponding element content is taken as input, and based on the training results, the probability that the element content belongs to the element is output. In the embodiment of the present application, the information evaluation model outputs a quantitative value belonging to the element, for example 80, and the quantitative value is a total score of 100.

[0136] In the above audit quality assessment method, sample information corresponding to multiple sample elements to be evaluated is obtained, and the sample information includes positive sample information and negative sample information. The positive sample information represents samples that fully conform to the corresponding element information to be evaluated, and the negative sample information represents samples that do not fully conform to the corresponding element information to be evaluated. According to the sample information corresponding to each sample element to be evaluated, the initial information assessment model is trained to obtain the information assessment model. In this method, the information assessment model is obtained by training the initial information assessment model according to the positive sample information and negative sample information in the sample information corresponding to each sample element to be evaluated, and the information assessment model improves the accuracy of the target document audit result assessment.

[0137] Based on the above embodiment, the method of obtaining sample information corresponding to multiple sample elements to be evaluated is described. In one embodiment, Figure 7 As shown, this embodiment includes the following steps:

[0138] S701, obtain the historical inventory of document review data.

[0139] After reviewing the documents, the document information will be stored in the database, so the accumulated data of historical inventory review can be obtained in the database.

[0140] In addition, the historical inventory of document review data includes offline document review data and intelligent document review data. The offline document review data may be data after manual review, and the intelligent document review data may be data after the document is reviewed through computer technology.

[0141] S702: Determine the positive sample information corresponding to each sample element to be evaluated based on the historical inventory of order review data.

[0142] The information corresponding to each element to be evaluated is queried in the historical inventory of document review data, and the positive sample information corresponding to each sample element to be evaluated in the historical inventory of document review data is obtained.

[0143] S703, performing simulation processing on the positive sample information to obtain negative sample information.

[0144] In the historical inventory of document review data, the corresponding elements of recognition errors have accumulated less, resulting in less data of negative sample information.

[0145] Therefore, on the one hand, negative sample information is obtained from the data source accumulated by intelligent document review, and on the other hand, a negative sample set needs to be simulated through a program; for example, in intelligent document review, there are often missed words, incomplete content, or misrecognition of text or letters due to poor quality caused by crossing lines, such as recognizing the letter O as the number 0.

[0146] The process of simulating the positive sample information and obtaining the negative sample information is as follows: extract a certain proportion of data from the positive sample, and simulate a batch of negative sample information through a program according to the situations that often occur in intelligent order review and extraction, such as word truncation, word omission, confusion of certain letters and numbers, and store them in the sample information set.

[0147] In the above-mentioned audit quality assessment method, historical inventory of audit data is obtained, and the positive sample information corresponding to each sample element to be evaluated is determined based on the historical inventory of audit data, and the positive sample information is simulated to obtain negative sample information. In this method, the positive sample information is simulated to obtain negative sample information. By enriching the negative sample information, the diversity of sample information when training the information assessment model is guaranteed, and the accuracy of the information assessment model is improved.

[0148] In one embodiment, the multiple evaluation information of the target document includes the document type, the elements to be evaluated and the corresponding element content extracted by the document examination device from the information in the image text of the target document.

[0149] The document review device extracts the image text of the target document to obtain the document type, elements to be evaluated and corresponding element content of the target document.

[0150] First, define the decomposition of these document types according to their structures and divide them into different elements. The principle of decomposition is to implement it based on the elements required for document review, and store the document types and corresponding elements in a relational database. Then, the electronic image of the received target document is recognized, and the image text corresponding to the target document is extracted through optical character recognition (OCR). According to the rule model, the document type corresponding to the target document is determined. Then, according to the document type, the required elements and element contents are obtained from the extracted image text, that is, the elements to be evaluated and the corresponding element contents. Finally, the extracted elements are checked according to the verification rules of the document type to generate an evaluation request.

[0151] In one embodiment, there is also a way to obtain positive sample information and negative sample information. Based on multiple evaluation information of the target document that needs to be reviewed, the element content in each evaluation information is corrected according to its actual situation, and the element content corresponding to each element to be evaluated is added to the sample information; for example, "mutton with bones" should be "trade name", and the information evaluation model determines that it is not a "trade name", then it is corrected and "mutton with bones" is added to the sample information as positive sample information; "bone" is not a "trade name", and the information evaluation model determines that it is not a "trade name", if the sample is not in the negative sample set, then the element content is added to the sample information. By continuously supplementing sample information, the accuracy of the information evaluation model is improved.

[0152] Set a nightly scheduled task. For example, set it to automatically check whether the sample information content is updated at 11 o'clock every night. If the sample information is updated, the information evaluation model is triggered to obtain the latest sample information for training, and the accuracy of the information evaluation model is optimized through continuous iteration.

[0153] In one embodiment, the information evaluation model of this embodiment is a FastText model, which is an open source word vector and text classification tool and a natural language processing training model. Compared with other natural language processing technologies, FastText can provide a simple and efficient method for text classification and feature learning, and is very suitable for application in the self-assessment scoring model of the present invention.

[0154] The present invention fills the gap that intelligent document review cannot self-evaluate the quality of its review results. With the help of data lake and FastText text classification technology, it realizes the self-learning of the document element review self-evaluation model, which can automatically evaluate the review results of intelligent document review and prompt the quality risk of intelligent document review by providing result rating. Through the evaluation method of the present invention, firstly, it can effectively reduce the occurrence of document review errors caused by the failure to implement manual review, which leads to the failure to discover the incorrect recognition of the intelligent document review model; secondly, through the self-evaluation rating, it can be decided whether to conduct manual review according to the risk prompt, avoid unnecessary manual review, and reduce labor costs. At the same time, the system architecture of the background master-slave mode is very easy to expand the system to ensure efficient and stable operation of the system.

[0155] In one embodiment, Figure 8 As shown, taking the information evaluation model as a scoring model as an example, the embodiment includes the following steps:

[0156] S801, receiving a request for evaluation of a target document, wherein the request for evaluation includes a document type, elements to be evaluated and corresponding element contents.

[0157] S802, obtaining the current task accumulation status corresponding to each task process, and allocating the task to be evaluated to the task process with the least amount of evaluation task accumulation.

[0158] S803, the task process receives the request to be evaluated, and obtains the elements to be evaluated and the corresponding weights according to the document type of the request to be evaluated.

[0159] S804, scoring the element content of each element to be evaluated based on the preset scoring model corresponding to each element to be evaluated, and obtaining the score corresponding to each element to be evaluated;

[0160] The construction process of the scoring model of each element includes: according to the corresponding element, obtaining the sample data corresponding to the element from the sample data set, the sample data includes positive samples and negative samples, the positive sample is the correct content corresponding to the element, the negative sample is the wrong content corresponding to the element, determining the positive sample as 1, the negative sample as 0, the positive sample and the negative sample, training the FastText model to obtain the scoring model;

[0161] For the evaluation requests that require manual review, the sample set is supplemented according to the actual situation. If the element content belongs to the element, the element is added to the positive sample corresponding to the element in the sample set, otherwise, a negative sample is added; if an update to the sample set is detected in the preset night-time scheduled task, the updated sample set is trained and the scoring model is updated.

[0162] S805: Obtain a self-assessment score of the request to be assessed according to the score of each element to be assessed and the corresponding weight.

[0163] S806, determining an evaluation result according to the self-evaluation score of the request to be evaluated and the document type; the evaluation results include no need for manual review, sampling review, and need for manual review;

[0164] Among them, if the self-assessment score is greater than the high threshold, no manual review is required; if the self-assessment score is less than the low threshold, manual review is required, otherwise, sampling review can be performed.

[0165] The specific limitations of the audit quality assessment method provided in this embodiment can be found in the above-mentioned step limitations of each embodiment of the audit quality assessment method, which will not be repeated here.

[0166] It should be understood that, although each step in the attached flow chart in the above-described embodiment is shown in sequence according to the indication of the arrow, these steps are not necessarily performed in sequence according to the order indicated by the arrow. Unless there is clear explanation in this article, the execution of these steps does not have strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the attached figure in the above-described embodiment may include a plurality of steps or a plurality of stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of the steps or stages in other steps or other steps.

[0167] In one embodiment, Fig. 9 As shown, the embodiment of the present application also provides an audit quality assessment system 900, which includes: a bill structure definition module 901, a self-assessment task allocation module 902, an intelligent assessment module 903, a self-assessment result feedback module 904, and an element scoring model self-learning module 905, wherein:

[0168] The bill structure definition module 901 is used to perform structural decomposition on different bill types of documents, query the elements required for intelligent document review from the database according to the bill type, set the corresponding weight value according to the importance of the element in the corresponding document type, and set the self-assessment level rules for intelligent document review of the corresponding bill type.

[0169] The self-assessment task allocation module 902 is used to allocate tasks to be evaluated. The self-assessment task allocation module 902 is a master-slave architecture, where the master process is responsible for implementing task allocation, and the task process is a slave process, whose main responsibility is to complete the corresponding task.

[0170] The intelligent evaluation module 903 is used to obtain the elements and corresponding weight values ​​required for evaluation of the corresponding bill type in the bill structure definition module 901, and then send the element content obtained by the intelligent review to the corresponding scoring model to obtain the score, and then multiply the score by the corresponding weight value to obtain the final score of the element, and then add up the scores of all elements, which is the self-assessment score of the intelligent review of this type of document. Finally, based on the score rating of this type, the quality of the intelligent review result is rated, and the subsequent processing is determined according to the evaluation result.

[0171] The self-assessment result feedback module 904 is used to determine the next step according to the rating result. If it is excellent, no manual review is required; if it is good, it can be reviewed by sampling; if it is unqualified, manual review is required; and it is also used to provide a sample data source for the factor scoring model.

[0172] The factor scoring model self-learning module 905 is used to train the factor scoring model and continuously supplement the positive and negative sample sets through the feedback module.

[0173] In one embodiment, Fig.10 As shown, the embodiment of the present application also provides an audit quality assessment system 1000 including: an intelligent audit module 1001 and an intelligent audit result self-assessment module 1002, wherein:

[0174] The intelligent document review module 1001 is used to perform structured processing on the text information in the electronic image of the document and obtain the element content required for reviewing various types of documents.

[0175] The intelligent document review result self-assessment module 1002 is used to evaluate the quality of each element information extracted by the intelligent document review, so as to determine whether manual review is required.

[0176] For the specific definition of the audit quality assessment system, please refer to the definition of each step in the audit quality assessment method above, which will not be repeated here. Each module in the above audit quality assessment system can be implemented in whole or in part by software, hardware and their combination. The above modules can be embedded in or independent of the target device in the form of hardware, or stored in the memory of the target device in the form of software, so that the target device can call and execute the operations corresponding to the above modules.

[0177] In one embodiment, Fig.11 As shown, the embodiment of the present application further provides an audit quality assessment device 1100, the device 1100 includes: an acquisition module 1101, an analysis module 1102 and a determination module 1103, wherein:

[0178] The acquisition module 1101 is used to acquire multiple evaluation information of the target document; each evaluation information includes a set of correspondences between elements to be evaluated and the content of the elements; the multiple evaluation information of the target document represents the audit results generated after the document audit device audits the target document;

[0179] The analysis module 1102 is used to perform quantitative analysis on the correspondence between the elements to be evaluated and the element contents in each evaluation information through the information evaluation model, and obtain the evaluation quantitative value of each evaluation information;

[0180] The determination module 1103 is used to determine the quality pass status of the audit result of the target document according to the evaluation quantification value of each evaluation information.

[0181] In one embodiment, the analysis module 1102 includes:

[0182] The analysis unit is used to quantitatively analyze the corresponding relationship between each element to be evaluated and the element content through the unit information evaluation model corresponding to each element to be evaluated, so as to obtain the evaluation quantitative value of each evaluation information.

[0183] In one embodiment, the determination module 1103 includes:

[0184] A first acquisition unit is used to acquire the weight corresponding to the to-be-evaluated element in each evaluation information;

[0185] A first determining unit, configured to determine the total evaluation quantification value of the target document according to the evaluation quantification value of each evaluation information and the weight of the to-be-evaluated element in each evaluation information;

[0186] The second determination unit is used to determine the quality pass status of the audit result of the target document according to the total evaluation quantification value.

[0187] In one embodiment, the second determining unit includes:

[0188] A first determination subunit is used to determine that the quality pass status of the audit result of the target document is completely passed if the total evaluation quantification value is greater than or equal to a first preset threshold value;

[0189] A second determination subunit is used to determine that the quality pass status of the audit result of the target document is not passed if the total evaluation quantification value is less than or equal to a second preset threshold value; and the second preset threshold value is less than the first preset threshold value;

[0190] The third determination subunit is used to determine the quality pass status of the audit result of the target document as pending pass if the total evaluation quantification value is greater than the second preset threshold and less than the first preset threshold.

[0191] In one embodiment, the apparatus 1100 further includes:

[0192] The first judgment module is used to determine that the audit result of the target document does not need to be reviewed if the quality pass status of the audit result of the target document is completely passed;

[0193] The second judgment module is used to determine that the audit result of the target document needs to be reviewed as a whole if the quality pass status of the audit result of the target document is not passed;

[0194] The third judgment module is used to determine whether the audit result of the target document needs to be reviewed by sampling if the quality pass status of the audit result of the target document is pending pass.

[0195] In one embodiment, the acquisition module 1101 includes:

[0196] A generating unit, for responding to an evaluation request of a target document, and sending the evaluation request to a target task process; the task process of each evaluation request is allocated according to the accumulation of evaluation requests of each task process;

[0197] The second acquisition unit is used to acquire multiple evaluation information of the target document in the evaluation request through the target task process.

[0198] In one embodiment, the apparatus 1100 further includes:

[0199] A sample module is used to obtain sample information corresponding to multiple sample elements to be evaluated; the sample information includes positive sample information and negative sample information; the positive sample information indicates a sample that completely meets the corresponding element information to be evaluated, and the negative sample information indicates a sample that does not completely meet the corresponding element information to be evaluated;

[0200] The training module is used to train the initial information evaluation model according to the sample information corresponding to each sample element to be evaluated to obtain the information evaluation model.

[0201] In one embodiment, the sample module includes:

[0202] The third acquisition unit is used to acquire the historical inventory of document review data;

[0203] The third determination unit is used to determine the positive sample information corresponding to each sample element to be evaluated based on the historical inventory of document review data;

[0204] The obtaining unit is used to simulate the positive sample information to obtain the negative sample information.

[0205] In one embodiment, the multiple evaluation information of the target document includes the document type, the elements to be evaluated and the corresponding element content extracted by the document examination device from the information in the image text of the target document.

[0206] For the specific definition of the audit quality assessment device, please refer to the definition of each step in the audit quality assessment method above, which will not be repeated here. Each module in the above-mentioned audit quality assessment device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the target device in the form of hardware, or can be stored in the memory of the target device in the form of software, so that the target device can call and execute the operations corresponding to the above modules.

[0207] In one embodiment, a computer device is provided, such as Fig.12As shown, the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an audit quality assessment method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0208] Those skilled in the art will appreciate that the structural description of the above-mentioned computer device is only a partial structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0209] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0210] The implementation principles and technical effects of each step implemented by the processor in this embodiment are similar to the principles of the above-mentioned audit quality assessment method, and will not be repeated here.

[0211] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0212] The implementation principles and technical effects of the steps implemented when the computer program in this embodiment is executed by the processor are similar to the principles of the above-mentioned audit quality assessment method, and will not be repeated here.

[0213] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0214] The implementation principles and technical effects of the steps implemented when the computer program in this embodiment is executed by the processor are similar to the principles of the above-mentioned audit quality assessment method, and will not be repeated here.

[0215] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0216] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0217] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0218] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for evaluating audit quality, characterized in that: The method comprises: Acquire multiple evaluation information of the target document; each of the evaluation information includes a set of correspondences between elements to be evaluated and element contents; the multiple evaluation information of the target document represents the audit result generated after the document audit device audits the target document; the audit result includes the elements and element contents obtained after the audit of the target document; Through the information evaluation model, the corresponding relationship between the to-be-evaluated elements and the element contents in each of the evaluation information is quantitatively analyzed to obtain an evaluation quantitative value of each of the evaluation information; the evaluation quantitative value is used to characterize the accuracy of the element contents corresponding to the to-be-evaluated elements; According to the evaluation quantification value of each of the evaluation information, the quality pass status of the audit result of the target document is determined.

2. The method according to claim 1, characterized in that The information evaluation model includes a plurality of unit information evaluation models, each unit information evaluation model corresponds to an element to be evaluated; Then, the information evaluation model is used to quantitatively analyze the correspondence between the elements to be evaluated and the element contents in each of the evaluation information to obtain the evaluation quantitative value of each of the evaluation information, including: Through the unit information evaluation model corresponding to each of the elements to be evaluated, the corresponding relationship between each of the elements to be evaluated and the element content is quantitatively analyzed to obtain the evaluation quantization value of each of the evaluation information.

3. The method according to claim 1 or 2, characterized in that: Determining the quality pass status of the audit result of the target document according to the evaluation quantified value of each evaluation information includes: Obtaining the weights corresponding to the elements to be evaluated in each of the evaluation information; Determining the total quantitative evaluation value of the target document according to the quantitative evaluation value of each evaluation information and the weight of the to-be-evaluated element in each evaluation information; The quality pass status of the audit result of the target document is determined based on the total evaluation quantification value.

4. The method according to claim 3, characterized in that Determining the quality pass status of the audit result of the target document according to the total evaluation quantification value includes: If the total evaluation quantification value is greater than or equal to a first preset threshold, determining that the quality pass status of the audit result of the target document is fully passed; If the total evaluation quantification value is less than or equal to a second preset threshold, the quality pass status of the audit result of the target document is determined to be failed; and the second preset threshold is less than the first preset threshold; If the total evaluation quantification value is greater than the second preset threshold and less than the first preset threshold, the quality pass status of the audit result of the target document is determined to be pending pass.

5. The method according to claim 4, characterized in that The method further comprises: If the quality pass status of the audit result of the target document is completely passed, it is determined that the audit result of the target document does not need to be reviewed; If the quality pass status of the audit result of the target document is failed, it is determined that the audit result of the target document needs to be reviewed as a whole; If the quality pass status of the audit result of the target document is pending pass, it is determined that the audit result of the target document needs to be reviewed by sampling.

6. The method according to claim 1 or 2, characterized in that: The obtaining of multiple evaluation information of the target document includes: In response to the evaluation request of the target document, the evaluation request is sent to the target task process; the task process of each evaluation request is allocated according to the accumulation of evaluation requests of each task process; A plurality of evaluation information of the target document in the evaluation request is obtained through the target task process.

7. The method according to claim 1 or 2, characterized in that: The construction process of the information evaluation model includes: Acquire sample information corresponding to a plurality of sample elements to be evaluated; the sample information includes positive sample information and negative sample information; the positive sample information indicates a sample that completely conforms to the corresponding element information to be evaluated, and the negative sample information indicates a sample that does not completely conform to the corresponding element information to be evaluated; The initial information evaluation model is trained according to the sample information corresponding to each of the sample elements to be evaluated to obtain the information evaluation model.

8. The method according to claim 7, characterized in that The obtaining of sample information corresponding to the plurality of sample elements to be evaluated includes: Obtain historical inventory review data; Determine the positive sample information corresponding to each sample element to be evaluated based on the historical inventory of document review data; The positive sample information is simulated to obtain the negative sample information.

9. The method according to claim 1 or 2, characterized in that: The multiple evaluation information of the target document includes the document type, the elements to be evaluated and the corresponding element content extracted by the document examination device from the information in the image text of the target document.

10. An audit quality assessment device, characterized in that: The device comprises: An acquisition module is used to acquire multiple evaluation information of a target document; each of the evaluation information includes a set of correspondences between elements to be evaluated and element contents; the multiple evaluation information of the target document represents the audit result generated after the document audit device audits the target document; the audit result includes the elements and element contents obtained after the audit of the target document; An analysis module, used to perform quantitative analysis on the correspondence between the elements to be evaluated and the element contents in each of the evaluation information through an information evaluation model, and obtain an evaluation quantitative value of each of the evaluation information; the evaluation quantitative value is used to characterize the accuracy of the element contents corresponding to the elements to be evaluated; The determination module is used to determine the quality pass status of the audit result of the target document according to the evaluation quantification value of each evaluation information.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images