Work order auditing method and device and storage medium

Abstract

The invention discloses a work order auditing method and device and a storage medium, and the method comprises the steps: reserving a work order which accords with a work order form as a target work order for each obtained work order; performing auditing level matching on each target work order by using screening standards of different levels in sequence, and determining an auditing level corresponding to each target work order; wherein the target work order with the auditing level does not need to be screened according to the screening standard of the next level; and performing manual auditing or automatic auditing on the target work order by using the corresponding auditing level. According to the technical scheme, the work order auditing efficiency and accuracy are improved.

Description

Technical Field

[0001] The present application belongs to the field of work order review, and specifically relates to a work order review method, device, and storage medium. Background Art

[0002] With the development and progress of the times, the number of work orders of telecom operators is increasing, and the types are complex, requiring a large number of manual review. Manual review is labor-intensive, time-consuming, and prone to bottlenecks. In addition, for some simple work orders, there is a lot of mechanical work. Reviewers are more prone to fatigue and errors. With the increase in work order types, there are more processing rules. The processing procedures of various work orders are cumbersome, with many nodes, bloated processing systems, and slow processing efficiency.

[0003] Therefore, how to improve the review efficiency and accuracy of work orders is a technical problem that needs to be solved urgently. Summary of the invention

[0004] The purpose of this application is to improve the efficiency and accuracy of work order review.

[0005] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0006] According to one aspect of an embodiment of the present application, a work order review method is provided, including:

[0007] For each work order obtained, keep the work order that meets the work order format as the target work order;

[0008] Using different levels of screening criteria to match the audit level of each target work order in turn, and determining the audit level corresponding to each target work order; wherein, the target work order with the audit level does not need to be screened by the next level of screening criteria;

[0009] The target work order is manually reviewed or automatically reviewed using the corresponding review level.

[0010] According to one aspect of an embodiment of the present application, for each work order obtained, a work order that conforms to the work order format is retained as a target work order, including:

[0011] Based on the preset attributes of each filling unit in the work order, if the filled content in the filling unit meets the preset attributes, the work order is used as the target work order; if the filled content in the filling unit does not meet the preset attributes, the work order is rejected.

[0012] According to one aspect of an embodiment of the present application, each target work order is sequentially matched with a review level using screening criteria of different levels to determine the review level corresponding to each target work order, including:

[0013] Determine a first-level target work order corresponding to a first-level review level according to the basic attributes of the target work order;

[0014] For a target work order without a corresponding review level, a secondary target work order corresponding to the secondary review level is determined according to the processing difficulty of the target work order;

[0015] For a target work order without a corresponding review level, a third-level target work order corresponding to the third-level review level is determined according to the risk attribute of the target work order;

[0016] Target work orders that do not have corresponding review levels will be treated as manual target work orders.

[0017] According to one aspect of an embodiment of the present application, determining a first-level target work order corresponding to a first-level review level according to the basic attributes of the target work order includes:

[0018] Perform simple rule matching on each of the target work orders to obtain the processing simplicity of each of the target work orders;

[0019] Performing standard work order matching on each of the target work orders to determine the standardization of each of the target work orders;

[0020] Calculate the priority of each target work order according to the urgency, customer level, and importance of each target work order;

[0021] Determining the time limit tension of the target work order according to the remaining processing time of each target work order;

[0022] The first-level confidence of each target work order is calculated according to the processing simplicity, standardization, priority and time limit tension of each target work order, and the target work order with a first-level confidence greater than a first threshold is taken as the first-level target work order.

[0023] According to one aspect of an embodiment of the present application, for a target work order without a corresponding review level, determining a secondary target work order corresponding to a secondary review level according to the processing difficulty of the target work order includes:

[0024] For target tickets without corresponding review levels:

[0025] Extracting information from each of the target work orders to obtain feature information of each of the target work orders;

[0026] According to the characteristic information of each target work order, processing rule matching, historical case matching, and processing scenario matching are performed to obtain the processing rule matching degree, the historical case matching degree, and the processing scenario matching degree respectively;

[0027] The secondary confidence is calculated according to the processing rule matching degree, historical case matching degree and processing scenario matching degree, and the target work order with the secondary confidence greater than the second threshold is taken as the secondary target work order. The secondary confidence is inversely proportional to the processing difficulty of the target work order.

[0028] According to one aspect of an embodiment of the present application, for a target work order without a corresponding review level, determining a third-level target work order corresponding to a third-level review level according to a risk attribute of the target work order includes:

[0029] For target tickets without corresponding review levels:

[0030] Determine the field to which each target work order belongs according to the filled-in content in each target work order;

[0031] According to the field to which each target work order belongs, a preset evaluation model is used to evaluate the model score, security score, compliance score, and influence score of each target work order in the field to which it belongs, and the database in the evaluation model includes historical work orders and professional knowledge of the industry to which the work order belongs;

[0032] According to the model score, security score, compliance score, and influence score of each target work order, the third-level confidence of the target work order is calculated, and the target work order with a third-level confidence greater than a third threshold is taken as a third-level target work order, and the third-level confidence is inversely proportional to the risk attribute of the target work order.

[0033] According to one aspect of an embodiment of the present application, the work order review method is configured to run in a review model, and the method further includes:

[0034] Get feedback on the audit results of target work orders at each audit level;

[0035] If the proportion of negative feedback in the feedback is greater than the negative review threshold, then the proportion of negative reviews of the target work orders at each review level in the negative feedback is obtained;

[0036] According to the proportion of negative reviews, the manual review work orders corresponding to each review level are obtained as training work orders, and the proportion of training work orders corresponding to each review level in the overall training work orders is the same as the proportion of negative reviews;

[0037] The audit model is trained using the training work order.

[0038] According to one aspect of the embodiment of the present application, the method further includes:

[0039] Obtaining a sample work order, where the sample work order is marked with a manual processing result;

[0040] Inputting the sample work order into the initial audit model, and determining the error of the initial audit model based on the comparison result of the initial audit model output result of the sample work order with the manual processing result;

[0041] According to the error of the initial audit model, the initial audit model is trained until the error of the initial audit model is less than a set error threshold, thereby obtaining an audit model provided with the audit method of the work order.

[0042] According to one aspect of an embodiment of the present application, the present application provides a work order review device, including a memory, a processor, and a readable program stored on the memory, and the processor executes the readable program to implement any of the methods described above.

[0043] According to one aspect of an embodiment of the present application, a readable storage medium is provided, on which a readable program / instruction is stored. When the readable program / instruction is executed by a processor, any of the methods described above is implemented.

[0044] In the technical solution provided in the embodiment of the present application, first of all, for each work order obtained, the work order that conforms to the work order form is retained as the target work order; the audit level is matched for each of the target work orders in turn using different levels of screening criteria to determine the audit level corresponding to each of the target work orders; wherein, the target work order with the audit level does not need to be screened by the next level of screening criteria; the target work order is manually audited or automatically audited using the corresponding audit level. Through the above method, firstly, the work orders with low-level errors are rejected, and then different screening criteria are used to screen and grade the work orders with a large number, and automatic grading is achieved to avoid waste of human resources. Then, the work orders of different levels are automatically audited or manually audited accordingly. Automatic audit greatly reduces the burden on auditors, and increases the efficiency and accuracy of the audit of work orders, avoids the mechanical work of auditors, reduces the fatigue of personnel, and also ensures the accuracy of manual audit of work orders.

[0045] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0046] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0048] Figure 1 A flow chart of a work order review method according to an embodiment of the present application is shown.

[0049] Figure 2 A flowchart is shown for matching the audit levels of target work orders using screening criteria of different levels in turn to determine the audit level corresponding to each target work order according to an embodiment of the present application.

[0050] Figure 3 A flow chart of determining a first-level target work order corresponding to a first-level review level according to basic attributes of the target work order according to an embodiment of the present application is shown.

[0051] Figure 4 A flow chart is shown for determining a secondary target work order corresponding to a secondary review level according to the processing difficulty of a target work order without a corresponding review level according to an embodiment of the present application.

[0052] Figure 5 A flow chart is shown for determining a three-level target work order corresponding to a three-level review level according to the risk attributes of the target work order, for a target work order without a corresponding review level, according to an embodiment of the present application.

[0053] Figure 6 A training flowchart of an audit model according to an embodiment of the present application is shown.

[0054] Figure 7 A flowchart for adjusting the audit model according to an embodiment of the present application is shown.

[0055] Figure 8 A block diagram of a computer system structure for implementing a method for auditing a workpiece according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0056] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.

[0057] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0058] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0059] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0060] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0061] See also Figure 1 , Figure 1 A flowchart of a work order review method according to an embodiment of the present application is shown. The present application embodiment provides a work order review method, including:

[0062] Step S110, for each work order obtained, retain the work order that meets the work order format as the target work order;

[0063] Step S120, using different levels of screening criteria to match the audit levels of each target work order in turn, and determining the audit level corresponding to each target work order; wherein, the target work order with the audit level does not need to be screened by the next level of screening criteria;

[0064] Step S130: Perform manual or automatic review on the target work order using the corresponding review level.

[0065] The above three steps are described in detail below.

[0066] In step S110, each work order is reviewed for its form. Low-level errors such as incomplete information, wrong filling position, wrong filling content format, etc. that cause the work order to fail to be reviewed normally. All these work orders that do not conform to the work order form are rejected to make the work order filler re-fill. The work order that conforms to the work order form is taken as the target work order.

[0067] In some embodiments, an error mark is added to the position in the work order with a form error that does not conform to the work order form, and the work order with the form error mark is returned to the person who filled out the work order, so that the person who filled out the work order can more conveniently determine where the work order is filled out incorrectly.

[0068] In some embodiments, based on the preset attributes of each filling unit in the work order, if the filled content in the filling unit meets the preset attributes, the work order is used as the target work order; if the filled content in the filling unit does not meet the preset attributes, the work order is rejected.

[0069] In step S120, different levels of screening criteria are pre-set, and different levels of screening criteria correspond to work orders of different complexity and difficulty. According to the level of the screening criteria from low to high, each target work order is screened in turn. The lower the level of the screening criteria, the simpler the target work order is. For target work orders that have been screened by one screening criterion, there is no need to screen by other levels of screening criteria. That is, each level of screening criteria only screens target work orders that have not passed the screening criteria. The target work order has the same level of review level as the screening criteria that it passes, and each target work order has one and only one review level.

[0070] Different audit levels are configured with corresponding audit models or manual audits.

[0071] See also Figure 2 , Figure 2 A flowchart is shown of using different levels of screening criteria to match the audit level of each target work order in turn, and determining the audit level corresponding to each target work order according to an embodiment of the present application. The present application embodiment provides a step S120 of using different levels of screening criteria to match the audit level of each target work order in turn, and determining the audit level corresponding to each target work order, including:

[0072] Step S121, determining a first-level target work order corresponding to a first-level review level according to the basic attributes of the target work order;

[0073] Step S122, for a target work order that has no corresponding review level, determine a secondary target work order corresponding to the secondary review level according to the processing difficulty of the target work order;

[0074] Step S123, for a target work order without a corresponding review level, determine a third-level target work order corresponding to the third-level review level according to the risk attribute of the target work order;

[0075] Step S124, treating the target work order without a corresponding review level as a manual target work order.

[0076] The above four steps are described in detail below.

[0077] It needs to be made clear in advance that target work orders include first-level target work orders, second-level work standard work orders, third-level target work orders, and manual target work orders.

[0078] In step S121, preliminary data collection and identification is performed on the target work orders, and the target work orders with obviously simple processing rules and processed completely in accordance with the processing rules are screened out as the first-level target work orders. That is, if the first-level target work orders are processed manually, it is a simple mechanical work, so the first-level target work orders can be directly processed automatically and mechanically according to the processing rules through a simple review model.

[0079] See also Figure 3 , Figure 3 A flowchart of determining a first-level target work order corresponding to a first-level review level according to the basic attributes of the target work order according to an embodiment of the present application is shown. The embodiment of the present application provides a step S121 of determining a first-level target work order corresponding to a first-level review level according to the basic attributes of the target work order, including:

[0080] Step S201, performing simple rule matching on each target work order to obtain the processing simplicity of each target work order;

[0081] Step S202, matching each target work order with a standard work order to determine the standardization of each target work order;

[0082] Step S203, calculating the priority of each target work order according to the urgency, customer level, and importance of each target work order;

[0083] Step S204, determining the time limit tension of the target work order according to the remaining processing time of each target work order;

[0084] Step S205, calculate the first-level confidence of each target work order according to the processing simplicity, standard, priority, and time limit of each target work order, and take the target work order with a first-level confidence greater than the first threshold as the first-level target work order.

[0085] The above five steps are described in detail below.

[0086] The basic attributes of the target work order include the processing ease, standardization, priority, and time limit of the target work order.

[0087] In step S201, the target work order is matched with simple rules, and the simple rules are used to indicate the simplicity of the target work order. In this process, it can be understood that the processing methods of the target work order are analyzed to obtain the processing methods of each target work order, and the processing simplicity of the target work order is determined for the processing methods corresponding to each target work order. The higher the processing simplicity, the simpler the processing flow, and the lower the processing simplicity, the more complex the processing flow.

[0088] In step S202, each target work order is matched with a standard work order, that is, it is determined whether the target work order is a standard work order. A standard work order refers to a work order that can be processed according to a standard process. Such a work order only needs to be processed according to the standard process without manual intervention. It should be clear that a standard work order does not specifically refer to a work order, but a type of work order, which can include multiple work orders.

[0089] For example, various work orders have standard forms, such as various work order templates, which are also called standard work orders. If a target work order is more consistent with a work order template, its matching degree with the standard work order will be relatively high or exceed the set matching degree, and it can be processed completely according to the preset processing rules of the corresponding work order template, making the processing method traceable and relatively simple.

[0090] If a target work order does not quite match the work order template, its matching degree with the standard work order is low or does not reach the set matching degree, this target work order is a composite work order and cannot be simply processed according to the processing rules corresponding to a work order template. The processing method is more complicated.

[0091] Get the matching degree between the target work order and the standard work order. The higher the matching degree, the simpler the target work order. The lower the matching degree, the more complex the target work order.

[0092] In some embodiments, the standardization of the work order (ie, the degree of matching with a standard work order) is determined by keyword matching.

[0093] In step S203, the priority of each target work order is calculated according to the urgency, customer level, and importance of each target work order.

[0094] The urgency of the target work order is determined by the creator of the target work order. In some embodiments, the urgency of the target work order is determined based on the customer, region, and project targeted by the target work order. Different customers and regions correspond to different urgency levels. In some embodiments, the urgency levels corresponding to customers, regions, and projects change dynamically.

[0095] Customer levels are pre-set by the user, and each customer has a corresponding level.

[0096] The importance of the target work order refers to the importance of the project type that the target work order targets, and is pre-set by the user.

[0097] The priority of the target work order = w1*C+w2*U+w3*T, where C is the customer level of the target work order, U is the urgency of the target work order, and T is the importance of the target work order. In some embodiments, the value range of C is 1-5, the value range of U is 1-3, and the value range of T is 0-1. w1, w2, and w3 are preset coefficients. In some embodiments, the sum of w1, w2, and w3 is 1.

[0098] In step S204, the time limit tension of the target work order is determined according to the remaining processing time of each target work order. The shorter the remaining processing time, the higher the time limit tension of the target work order.

[0099] In step S205, the first-level confidence of each target work order is calculated based on the processing simplicity, standardization, priority, and time limit of each target work order. It should be clear that the processing simplicity, standardization, priority, and time limit of the target work order are positively correlated with the first-level confidence. That is, the higher the first-level confidence, the simpler, more standardized, more prioritized, and more time-limited the target work order.

[0100] In some embodiments, the first-level confidence level = w4*R+w5*K+w6*P+w7*O, where: R is the processing ease of the target work order; K is the standard of the target work order; P is the priority of the target work order; O is the time limit of the target work order. w4, w5, w6, and w7 are preset coefficients. In some embodiments, the sum of w4, w5, w6, and w7 is 1.

[0101] In the embodiment of the present application, the target work order with the first-level confidence greater than the first threshold is regarded as the first-level target work order. That is, the first-level target work order is relatively simple and standard, and the processing method is simple, which is convenient for the audit model to directly process. At the same time, due to the high working efficiency of the audit model, the more prioritized and urgent target work orders can be processed first, which improves the user experience.

[0102] In step S122, for target work orders other than the first-level target work orders, that is, target work orders that have not been reviewed for several times, the difficulty of processing the work orders is analyzed, and the target work orders with a processing difficulty lower than the set standard are regarded as second-level target work orders. Although the second-level target work orders cannot be processed mechanically, the processing difficulty is relatively small, and some trained review models are fully capable of handling it. Therefore, the second-level target work orders can be automatically reviewed.

[0103] See also Figure 4 , Figure 4 A flowchart of determining a secondary target work order corresponding to a secondary review level according to the processing difficulty of the target work order is shown for a target work order without a corresponding review level according to an embodiment of the present application. The embodiment of the present application provides a step S122 of determining a secondary target work order corresponding to a secondary review level according to the processing difficulty of the target work order for a target work order without a corresponding review level, including:

[0104] Step S301, extracting information from each target work order to obtain feature information of each target work order;

[0105] Step S302, performing processing rule matching, historical case matching, and processing scenario matching according to the characteristic information of each target work order, and obtaining the processing rule matching degree, historical case matching degree, and processing scenario matching degree respectively;

[0106] Step S303, calculate the secondary confidence according to the processing rule matching degree, historical case matching degree, and processing scenario matching degree, and take the target work order with the secondary confidence greater than the second threshold as the secondary target work order, and the secondary confidence is inversely proportional to the processing difficulty of the target work order.

[0107] The above three steps are described in detail below.

[0108] It should be noted that, in this embodiment, the target work orders described are all target work orders without corresponding review levels, that is, target work orders other than the first-level target work orders.

[0109] In step S301, information is extracted from each target work order to obtain feature information of each target work order. The feature information of the target work order includes but is not limited to the project targeted by the target work order, the application scenario, and the required processing rules.

[0110] In step S302, processing rule matching, historical case matching, and processing scenario matching are performed according to the characteristic information of each target work order to obtain the processing rule matching degree, historical case matching degree, and processing scenario matching degree respectively.

[0111] According to the characteristic information of the target work order, the processing rules are matched. The higher the matching degree, the clearer the processing rules of the target work order are. The clearer the processing rules of the target work order are, the lower the difficulty of processing the target work order will be. In some embodiments, the processing rules include conventional rules and expert rules. Expert rules refer to special processing rules established for work orders. The matching degree of processing rules includes conventional rule matching degree and expert rule matching degree.

[0112] According to the matching of the characteristic information of the target work order with the historical cases, the higher the matching degree of the historical cases, the more historical experience can be used as a reference for the target work order, which will make the processing difficulty of the target work order lower. It is convenient for the audit model to audit the work order.

[0113] According to the matching of the characteristic information of the target work order with the application scenario, the application scenario refers to the preset application environment. Each application scenario has corresponding processing suggestions. The higher the matching degree between the target work order and the application scenario, the more reference the processing suggestions of the application scenario can generate for the target work order, and it is easier for the target work order to be processed by the audit model.

[0114] In step S303, the secondary confidence is calculated according to the processing rule matching degree, the historical case matching degree, and the processing scenario matching degree, and the target work order with the secondary confidence greater than the second threshold is taken as the secondary target work order. The secondary confidence is inversely proportional to the processing difficulty of the target work order.

[0115] In some embodiments, secondary confidence = w8*S+w9*H+w 10 *B+w 11 *E. S is the matching degree of the processing scenario, H is the matching degree of historical cases, B is the matching degree of conventional rules, and E is the matching degree of expert rules. w8, w9, w 10 、w 11 are preset coefficients. In some embodiments, w8, w9, w 10 、w 11 The sum of the three is 1. The secondary confidence is positively correlated with the processing scenario matching, historical case matching, general rule matching, and expert rule matching. The higher the secondary confidence, the lower the difficulty of processing the target work order.

[0116] In the embodiment of the present application, the first screening takes simple target work orders as the first-level target work orders, and the remaining target work orders are relatively complex target work orders. For these relatively complex target work orders, those with lower processing difficulty are taken as second-level target work orders. Although the second-level target work orders cannot be processed mechanically, the processing is relatively small, and some trained audit models are fully capable of handling it. Therefore, the second-level target work orders can be automatically audited.

[0117] In step S123, for target work orders other than the first-level target work orders and the second-level target work orders, that is, target work orders that have not passed the two screenings, the risk attributes of the target work orders are calculated. If the risk attribute of the target work order is less than a certain level, the target work order is regarded as a third-level target work order. Although such work orders are more difficult to handle, they have less influence. Specially trained audit models can also be competent, and because of their smaller influence and risk, they have greater fault tolerance. Therefore, the third-level target work order can be automatically audited.

[0118] See also Figure 5 , Figure 5 A flowchart of determining a level 3 target work order corresponding to a level 3 audit level according to the risk attribute of the target work order is shown for a target work order without a corresponding audit level according to an embodiment of the present application. The present application embodiment provides a step S123 of determining a level 3 target work order corresponding to a level 3 audit level according to the risk attribute of the target work order for a target work order without a corresponding audit level, including:

[0119] Step S401, determining the field to which each target work order belongs according to the filled content in each target work order;

[0120] Step S402: Based on the fields to which each target work order belongs, a preset evaluation model is used to evaluate the model score, security score, compliance score, and influence score of each target work order in the field to which it belongs. The database in the evaluation model includes historical work orders and professional knowledge of the industry to which the work order belongs.

[0121] Step S403, calculate the third-level confidence of the target work order according to the model score, security score, compliance score, and influence score of each target work order, and take the target work order with the third-level confidence greater than the third threshold as the third-level target work order, and the third-level confidence is inversely proportional to the risk attribute of the target work order.

[0122] The above three steps are described in detail below.

[0123] It should be noted that, in this embodiment, the target work orders described are all target work orders without corresponding review levels, that is, target work orders other than the first-level target work orders and the second-level target work orders.

[0124] The risk attributes of the target work order include the model score, security score, compliance score, and impact score of the target work order in its field.

[0125] In step S401, the field to which each target work order belongs is determined based on the content filled in each target work order. In some embodiments, keyword matching of each field is performed through the content filled in the target work order to determine the field to which the target work order belongs.

[0126] In step S402, based on the field to which each target work order belongs, a preset evaluation model is used to evaluate the model score, security score, compliance score, and influence score of each target work order in the field to which it belongs. The database in the evaluation model includes historical work orders and professional knowledge of the industry to which the work order belongs.

[0127] In some embodiments, the field to which the target work order belongs may include multiple fields, and the model score, security score, compliance score, and influence score of the target work order in each field are evaluated respectively.

[0128] The security score is an assessment of the risk of executing the target work order. The compliance score refers to the judgment of whether the content filled in the target work order is legal and compliant. The influence score refers to the reverse mapping of the influence in the field to which the target work order belongs. That is, the higher the influence score, the smaller the influence of the target work order in its field. The model score is also called the expert score, which is used to evaluate the ease of automatic review of the target work order. The higher the model score, the easier it is to automatically review the target work order.

[0129] In some embodiments, if the field to which the target work order belongs includes multiple fields, the security score, compliance score, and impact score of the target work order in each field are added together to obtain the final security score, compliance score, and impact score.

[0130] In some embodiments, the database in the evaluation model includes historical work orders and professional knowledge of the industry to which the work orders belong, so that the evaluation model's scoring of the target work orders is more professional.

[0131] In step S403, the third-level confidence of the target work order is calculated based on the security score, compliance score, and influence score of each target work order, and the target work order with a third-level confidence greater than the third threshold is taken as a third-level target work order. The third-level confidence is inversely proportional to the risk attribute of the target work order.

[0132] In some embodiments, the third level confidence level = w 12 *Q+w 13 *L+w 14 *M+w 15 *N. Where Q is the security score, L is the compliance score, M is the impact score, and N is the model score. 12 、w 13 、w 14 、w 15 is a preset coefficient. In some embodiments, w 12 、w 13 、w 14 、w 15 The sum of the three is 1.

[0133] In the embodiment of the present application, although the third-level target work order is more difficult to handle, it has less influence. A specially trained audit model can also handle it, and because of its small influence, it has greater fault tolerance. Therefore, the third-level target work order can be automatically audited.

[0134] In step S124, the target work orders that do not have corresponding audit levels, that is, the target work orders that have not passed the above screening, are manual target work orders. Because the target work orders that have not been selected after three screenings are difficult to handle, complex, and have high risk attributes. The audit model is no longer competent, and because the risk is relatively high, it is most appropriate to hand it over to manual review. On the one hand, it avoids the occupation of audit employees by simple and low-level work orders, saving human resources. On the other hand, the accuracy of automatic review of simple and low-level work orders is higher, and the most important and complex work orders are handed over to manual review, making full use of human resources. The audit employees only handle work orders that can only be handled manually, and people are put to good use. It also increases the accuracy of the work order review.

[0135] In step S130, the target work order is manually reviewed or automatically reviewed using the corresponding review level. Each review level has a corresponding review method, and some target work orders of the review level are automatically reviewed. Some target work orders of the appropriate level are manually reviewed.

[0136] See also Figure 6 , Figure 6 The following is a flow chart of the training of an audit model according to an embodiment of the present application. The present application embodiment provides the training steps of the audit model, including:

[0137] Step S501, obtaining a sample work order, on which a manual processing result is marked;

[0138] Step S502, inputting the sample work order into the initial audit model, and determining the error of the initial audit model based on the comparison result between the initial audit model output result of the sample work order and the manual processing result;

[0139] Step S503, according to the error of the initial audit model, the initial audit model is trained until the error of the initial audit model is less than a set error threshold, and an audit model with an audit method for the work order is obtained.

[0140] The above three steps are described in detail below.

[0141] It should be made clear that the audit model can be an AI model or a neural network training model.

[0142] In step S501, a sample work order is obtained, on which a manual processing result is marked;

[0143] In step S502, the sample work order is input into the initial audit model, and the error of the initial audit model is determined based on the comparison result between the initial audit model output result of the sample work order and the manual processing result;

[0144] In some embodiments, sample work orders at different audit levels are obtained, and the sample work orders at different audit levels are respectively input into corresponding audit modules, and the output results of the audit modules are compared with the manual audit results of the corresponding sample work orders to determine the errors of each audit module.

[0145] In step S503, according to the error of the initial audit model, the initial audit model is trained until the error of the initial audit model is less than a set error threshold, thereby obtaining an audit model having an audit method for the work order.

[0146] In some embodiments, the audit model includes multiple audit modules to automatically audit target work orders at different audit levels. Sample work orders at different audit levels are obtained, and the sample work orders at different audit levels are respectively input into the corresponding initial audit modules, and the output results of the initial audit modules are compared with the manual audit results of the corresponding sample work orders to determine the errors of each initial audit module. According to the errors of each initial audit module, the parameters of each initial audit module are adjusted until the error of the audit module at each audit level is less than the set error threshold, and each audit module is obtained. Then the training of the audit model is completed to obtain the audit model.

[0147] See also Figure 7 , Figure 7 A flowchart of adjusting the audit model according to an embodiment of the present application is shown. The present application embodiment provides steps for adjusting the audit model, including:

[0148] Step S601, obtaining feedback on the audit results of target work orders at each audit level;

[0149] Step S602: if the proportion of negative feedback in the feedback is greater than the negative review threshold, then obtain the negative review proportion of the target work orders at each review level in the negative feedback;

[0150] Step S603, according to the proportion of negative reviews, respectively obtain the manual review work orders corresponding to each review level as training work orders, and the proportion of training work orders corresponding to each review level in the overall training work orders is the same as the proportion of negative reviews;

[0151] Step S604: Use the training work order to train the audit model.

[0152] The above four steps are described in detail below.

[0153] In step S601, feedback on the audit results of target work orders at each audit level is obtained;

[0154] In step S602, if the proportion of negative feedback in the feedback is greater than the negative review threshold, the negative review proportion of the target work orders at each review level in the negative feedback is obtained, that is, the proportion of negative feedback at each review level in the overall negative feedback is obtained.

[0155] In step S603, the number of training work orders corresponding to each audit level is calculated based on the proportion of negative feedback at each audit level in the overall negative feedback and the set number of training work orders. Among them, training work orders refer to sample work orders used to adjust the audit model. The product of the set number of training work orders and the proportion of negative reviews at each audit level is taken as the number of training work orders corresponding to each audit level. That is, the proportion of training work orders corresponding to each audit level in the overall training work orders is the same as the proportion of negative reviews.

[0156] In step S604, the audit modules corresponding to each audit level in the audit model are trained using the training worksheets corresponding to each audit level until the error value of the audit module corresponding to each audit level is less than the training error threshold, thus completing the adjustment of the audit model.

[0157] Figure 8 A block diagram of a computer system structure for implementing a work order review method according to an embodiment of the present application is shown.

[0158] It should be noted that Figure 8 The computer system 800 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0159] like Figure 8 As shown, the computer system 800 includes a central processing unit 801 (CPU), which can perform various appropriate actions and processes according to the program stored in the read-only memory 802 (ROM) or the program loaded from the storage part 808 to the random access memory 803 (RAM). Various programs and data required for system operation are also stored in the random access memory 803. The central processing unit 801, the read-only memory 802 and the random access memory 803 are connected to each other through a bus 804. The input / output interface 805 (Input / Output interface, i.e., I / O interface) is also connected to the bus 804.

[0160] The following components are connected to the input / output interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a local area network card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the input / output interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed so that a computer program read therefrom is installed into the storage section 808 as needed.

[0161] In particular, according to an embodiment of the present application, the process described in each method flow chart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program contains a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the central processor 801, various functions defined in the system of the present application are executed.

[0162] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0163] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0164] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0165] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation methods of the present application.

[0166] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary technical means in the art that are not disclosed in the present application.

[0167] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A work order review method, characterized in that: include: For each work order obtained, keep the work order that meets the work order format as the target work order; Using different levels of screening criteria to match the audit level of each target work order in turn, and determining the audit level corresponding to each target work order; wherein, the target work order with the audit level does not need to be screened by the next level of screening criteria; The target work order is manually reviewed or automatically reviewed using the corresponding review level.

2. The method according to claim 1, characterized in that For each work order obtained, keep the work orders that meet the work order format as the target work orders, including: Based on the preset attributes of each filling unit in the work order, if the filled content in the filling unit meets the preset attributes, the work order is taken as the target work order; if the filled content in the filling unit does not meet the preset attributes, the work order is rejected.

3. The method according to claim 1, characterized in that Using different levels of screening criteria for each target work order in turn to match the audit level, and determining the audit level corresponding to each target work order, including: Determine a first-level target work order corresponding to a first-level review level according to the basic attributes of the target work order; For a target work order without a corresponding review level, a secondary target work order corresponding to the secondary review level is determined according to the processing difficulty of the target work order; For a target work order without a corresponding review level, a third-level target work order corresponding to the third-level review level is determined according to the risk attribute of the target work order; Target work orders that do not have corresponding review levels will be treated as manual target work orders.

4. The method according to claim 3, characterized in that Determine the first-level target work order corresponding to the first-level review level according to the basic attributes of the target work order, including: Perform simple rule matching on each of the target work orders to obtain the processing simplicity of each of the target work orders; Performing standard work order matching on each of the target work orders to determine the standardization of each of the target work orders; Calculate the priority of each target work order according to the urgency, customer level, and importance of each target work order; Determining the time limit tension of the target work order according to the remaining processing time of each target work order; The first-level confidence of each target work order is calculated according to the processing simplicity, standardization, priority and time limit tension of each target work order, and the target work order with a first-level confidence greater than a first threshold is taken as the first-level target work order.

5. The method according to claim 3, characterized in that: For target work orders without corresponding review levels, the secondary target work orders corresponding to the secondary review levels are determined according to the difficulty of processing the target work orders, including: For target tickets without corresponding review levels: Extracting information from each of the target work orders to obtain feature information of each of the target work orders; According to the characteristic information of each target work order, processing rule matching, historical case matching, and processing scenario matching are performed to obtain the processing rule matching degree, the historical case matching degree, and the processing scenario matching degree respectively; The secondary confidence is calculated according to the processing rule matching degree, historical case matching degree and processing scenario matching degree, and the target work order with the secondary confidence greater than the second threshold is taken as the secondary target work order. The secondary confidence is inversely proportional to the processing difficulty of the target work order.

6. The method according to claim 3, characterized in that: For a target work order without a corresponding review level, a third-level target work order corresponding to the third-level review level is determined according to the risk attribute of the target work order, including: For target tickets without corresponding review levels: Determine the field to which each target work order belongs according to the filled-in content in each target work order; According to the field to which each target work order belongs, a preset evaluation model is used to evaluate the model score, security score, compliance score, and influence score of each target work order in the field to which it belongs, and the database in the evaluation model includes historical work orders and professional knowledge of the industry to which the work order belongs; According to the model score, security score, compliance score, and influence score of each target work order, the third-level confidence of the target work order is calculated, and the target work order with a third-level confidence greater than a third threshold is taken as a third-level target work order, and the third-level confidence is inversely proportional to the risk attribute of the target work order.

7. A work order review method, characterized in that: The work order review method is configured to run in the review model, and the method further includes: Get feedback on the audit results of target work orders at each audit level; If the proportion of negative feedback in the feedback is greater than the negative review threshold, then the proportion of negative reviews of the target work orders at each review level in the negative feedback is obtained; According to the proportion of negative reviews, the manual review work orders corresponding to each review level are obtained as training work orders, and the proportion of training work orders corresponding to each review level in the overall training work orders is the same as the proportion of negative reviews; The audit model is trained using the training work order.

8. The method according to claim 7, characterized in that The method further comprises: Obtaining a sample work order, where the sample work order is marked with a manual processing result; Inputting the sample work order into the initial audit model, and determining the error of the initial audit model based on the comparison result of the initial audit model output result of the sample work order with the manual processing result; According to the error of the initial audit model, the initial audit model is trained until the error of the initial audit model is less than a set error threshold, thereby obtaining an audit model provided with the audit method of the work order.

9. A work order review device, comprising a memory, a processor, and a readable program stored in the memory, characterized in that: The processor executes the readable program to implement the method according to any one of claims 1 to 7.

10. A readable storage medium having a readable program / instruction stored thereon, characterized in that: When the readable instructions / instructions are executed by a processor, the method described in any one of claims 1 to 7 is implemented.

Images