Training method, device, equipment and storage medium for work order quality inspection model

Abstract

The present application discloses a training method, apparatus, equipment and storage medium for a work order quality inspection model, which belongs to the field of natural language processing. The method includes: obtaining sample work order data and the classification label of the sample work order data; inputting the feature representation of the sample work order data into the work order quality inspection model to obtain the model prediction probability and prediction confidence; calculating the corrected prediction probability obtained after the model prediction probability is corrected; calculating the model prediction error between the corrected prediction probability and the classification label of the sample work order data; performing backward error propagation training on the work order quality inspection model based on the model prediction error to obtain the work order quality inspection model. The present application adds a confidence prediction network, which describes the credibility of the model prediction probability through the prediction confidence, and corrects the model prediction probability, thereby reducing the impact of the classification label labeling on the work order quality inspection model and improving the stability of the work order quality inspection model in performing work order quality inspection.

Description

Technical Field

[0001] The present application relates to the field of natural language processing, and in particular to a training method, apparatus, equipment, and storage medium for a work order quality inspection model. Background Art

[0002] Work order quality inspection is the process of determining whether work order data is abnormal, and is used to screen out work order data that contains errors or risks.

[0003] Currently, the method commonly used is to extract a large number of features from work order data and mark the error types. Machine learning models are used to perform quality inspection on the error types of the marked work order data to screen out work order data with errors or risks.

[0004] Since it is impossible to label all error types that appear in work order data, and the work order quality inspection process relies on data with labeled error types, the work order quality inspection may not be able to accurately determine abnormal work order data. Summary of the Invention

[0005] This application provides a training method, apparatus, device, and storage medium for a work order quality inspection model. By adding a confidence prediction network and adding prediction confidence as an indicator for evaluating the work order quality inspection model, the stability of the work order quality inspection model in performing work order quality inspection is improved. The technical solution is as follows:

[0006] According to one aspect of the present application, a method for training a work order quality inspection model is provided, the method comprising:

[0007] Obtaining sample work order data and a classification label for the sample work order data, wherein the classification label is used to indicate whether the sample work order data is abnormal work order data;

[0008] Inputting the feature representation of the sample work order data into the work order quality inspection model to obtain the model prediction probability and prediction confidence, wherein the prediction confidence is used to describe the credibility of the model prediction probability obtained by prediction;

[0009] Calculating a revised prediction probability using the model prediction probability, the prediction confidence, and the classification label of the sample work order data;

[0010] Calculating a model prediction error between the corrected prediction probability and the classification label of the sample work order data;

[0011] Backward error propagation training is performed on the work order quality inspection model based on the model prediction error to obtain the trained work order quality inspection model.

[0012] According to another aspect of the present application, a training device for a work order quality inspection model is provided, the device comprising:

[0013] A first acquisition module is used to acquire sample work order data and a classification label of the sample work order data, wherein the classification label is used to indicate whether the sample work order data is abnormal work order data;

[0014] A first processing module is configured to input the feature representation of the sample work order data into a work order quality inspection model to obtain a model prediction probability and a prediction confidence, wherein the prediction confidence is used to describe the credibility of the model prediction probability obtained by the prediction;

[0015] A correction module, configured to calculate a corrected prediction probability using the model prediction probability, the prediction confidence, and the classification label of the sample work order data;

[0016] a calculation module, configured to calculate a model prediction error between the corrected prediction probability and the classification label of the sample work order data;

[0017] A training module is used to perform backward error propagation training on the work order quality inspection model based on the model prediction error to obtain the trained work order quality inspection model.

[0018] In an optional design of the present application, the work order quality inspection model includes a feature extraction network, a quality inspection prediction network, and a confidence prediction network;

[0019] The first processing module is further configured to:

[0020] Inputting the feature representation of the sample work order data into the feature extraction network to obtain a hidden layer representation of the feature representation of the sample work order data;

[0021] Inputting the hidden layer representation into the quality inspection prediction network to obtain the model prediction probability;

[0022] The hidden layer representation is input into the confidence prediction network to obtain the prediction confidence.

[0023] In an optional design of this application,

[0024] The feature extraction network includes a cascaded n1-layer feedforward neural network and an n2-layer fully connected neural network;

[0025] The quality inspection prediction network includes an n3-layer feedforward neural network;

[0026] The confidence prediction network includes an n4-layer feedforward neural network;

[0027] Wherein, n1, n2, n3 and n4 are all integers greater than 1.

[0028] In an optional design of the present application, the computing module includes:

[0029] an error calculation unit, configured to calculate a first error based on the corrected prediction probability and the classification label of the sample work order data;

[0030] An error correction unit is used to correct the first error based on the prediction confidence to obtain the model prediction error.

[0031] In an optional design of the present application, the error correction unit is further configured to:

[0032] Calculating a second error based on the prediction confidence;

[0033] Based on the first error and the second error, a model prediction error between the corrected prediction probability and the classification label of the sample work order data is obtained.

[0034] In an optional design of this application,

[0035] In the model prediction error, the first error and the prediction confidence are positively correlated, and the second error and the prediction confidence are negatively correlated;

[0036] or,

[0037] In the model prediction error, the first error and the prediction confidence show a negative correlation, and the second error and the prediction confidence show a positive correlation.

[0038] In an optional design of the present application, the first error is a negative log-likelihood function of the classification label of the sample work order data with respect to the corrected prediction probability.

[0039] In an optional design of the present application, the second error is a negative logarithmic function of the prediction confidence.

[0040] In an optional design of the present application, the device further includes:

[0041] The second acquisition module is used to obtain input work order data;

[0042] A second processing module is configured to input the input work order data into the work order quality inspection model to obtain a model prediction probability;

[0043] An output module is used to determine the work order data whose model prediction probability meets the quality inspection conditions as abnormal work order data, and to output the abnormal work order data, wherein the quality inspection conditions are set based on the model prediction probability corresponding to the input work order data, and the abnormal work order data is part or all of the data in the input work order data.

[0044] In an optional design of this application, the quality inspection conditions include:

[0045] The model predicts a probability greater than m1, where m1 is a number greater than or equal to 0 and less than 1;

[0046] or,

[0047] The input work order data are sorted from high to low according to the rule of the model prediction probability, and the first m2 input work order data are sorted, where m2 is an integer greater than 0 and less than or equal to the number of the input work order data.

[0048] According to another aspect of the present application, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the training method of the work order quality inspection model as described above.

[0049] According to another aspect of the present application, a computer-readable storage medium is provided, in which at least one instruction, at least one program, a code set or an instruction set is stored. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the training method of the work order quality inspection model as described above.

[0050] According to another aspect of the present application, a computer program product or computer program is provided, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor reads and executes the computer instructions from the computer-readable storage medium to implement the training method of the work order quality inspection model described above.

[0051] The beneficial effects brought about by the technical solution provided by the present application include at least: adding a confidence prediction network, describing the credibility of the model prediction probability through prediction confidence, and adding prediction confidence as an indicator for evaluating the work order quality inspection model; using prediction confidence to correct the model prediction probability, and using the corrected prediction probability obtained after correction to calculate the error between the corrected prediction probability and the classification label of the sample work order data, thereby increasing the dimension of evaluating the model prediction error during the training process of the work order quality inspection model, reducing the impact of the labeling of the classification labels of the sample work order data on the work order quality inspection model, improving the training effect of the work order quality inspection model, and improving the stability of the work order quality inspection model in performing work order quality inspection. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0053] Figure 1 is a block diagram of a computer system for training and / or using a work order quality inspection model provided by an exemplary embodiment of the present application;

[0054] Figure 2 is a schematic diagram of a method for training and / or using a work order quality inspection model provided by an exemplary embodiment of the present application;

[0055] Figure 3 This is a flowchart of a method for training a work order quality inspection model provided by an exemplary embodiment of the present application;

[0056] Figure 4 This is a flowchart of a method for training a work order quality inspection model provided by an exemplary embodiment of the present application;

[0057] Figure 5 This is a flowchart of a method for training a work order quality inspection model provided by an exemplary embodiment of the present application;

[0058] Figure 6 This is a flowchart of a method for training a work order quality inspection model provided by an exemplary embodiment of the present application;

[0059] Figure 7 This is a flowchart of a method for training a work order quality inspection model provided by an exemplary embodiment of the present application;

[0060] Figure 8 This is a structural block diagram of a training device for a work order quality inspection model provided by an exemplary embodiment of the present application;

[0061] Figure 9 This is a structural block diagram of a server provided by an exemplary embodiment of the present application.

[0062] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application. DETAILED DESCRIPTION

[0063] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0064] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. As used in this disclosure and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0065] Figure 1 A block diagram of a computer system for training and / or using a work order quality inspection model provided by one embodiment of the present application is shown. The system may include: a terminal 110, a client 120, a server 130, a work order quality inspection model 140, and a network 150.

[0066] Client 120 is used to obtain work order data and send it to server 130. The work order data can be sample work order data and its classification labels to train the work order quality inspection model; it can also be input work order data to use the work order quality inspection model. The use and training of the work order quality inspection model can be implemented using a single client or multiple clients.

[0067] The client 120 may be installed on the terminal 110. For example, the terminal 110 includes but is not limited to at least one of the following computer devices: a smart phone, a tablet computer, a laptop computer, and a desktop computer.

[0068] Server 130 is the operator of the work order quality inspection model 140 constructed in this application and the provider of work order quality inspection services. Server 130 can receive input work order data sent by client 120, perform work order quality inspection, and send abnormal work order data within the input work order data to client 120. Server 130 can also receive sample work order data and classification labels sent by client 120, train the work order quality inspection model, update work order quality inspection model 140, and store work order quality inspection model 140 on server 130. Optionally, server 130 can also send work order quality inspection model 140 to terminal 110.

[0069] The server 130 has at least one of the following capabilities: responding to service requests for training and / or using the work order quality inspection model, undertaking the training and / or using the work order quality inspection model, and ensuring the training and / or using of the work order quality inspection model.

[0070] Optionally, during the training and / or use of the work order quality inspection model, the server 130 undertakes the main computing work and the terminal 110 undertakes the secondary computing work; or, the server 130 undertakes the secondary computing work and the terminal 110 undertakes the main computing work; or, a distributed computing architecture is used between the server 130 and the terminal 110 for collaborative computing.

[0071] The communication between the terminal 110 and the server 130 is achieved through the network 150. The communication mode of the network 150 includes but is not limited to achieving direct or indirect communication between the terminal 110 and the server 130 through wired or wireless modes.

[0072] During the training process of the work order quality inspection model, a confidence prediction network was added. The prediction confidence was used to describe the credibility of the model's prediction probability. The work order quality inspection model was trained in combination with the model's prediction probability. The work order quality inspection model was used to more accurately obtain abnormal work order data in the work order data.

[0073] Work order data is data that records the completion status of work. Work order data can be text content or voice data.

[0074] Abnormal work order data refers to work order data that contains errors or risks. In this embodiment, there is no restriction on the error or risk category of the work order data. For example, the error or risk category of the work order data includes, but is not limited to, at least one of the following categories: incorrect service category, incorrect service content, indifferent service attitude, blunt service content, and poor service results.

[0075] like Figure 2 As shown, the training and / or use of work order quality inspection can be divided into training the work order quality inspection model 220 and / or using the trained work order quality inspection model 250.

[0076] The work order quality inspection model 220 includes a feature extraction network 221 , a confidence prediction network 223 , and a quality inspection prediction network 224 . The trained work order quality inspection model 250 includes a feature extraction network 251 and a quality inspection prediction network 253 .

[0077] During the training of the work order quality inspection model 220, sample work order data 211 and its classification label 212 are obtained. The sample work order data 211 is input into the feature extraction network 221 to obtain the hidden layer representation 222 corresponding to the work order data 211. The hidden layer representation 222 is input into the confidence prediction network 223 and the quality inspection prediction network 224, and then passes through the S-shaped growth function 225 and the normalization function 226, respectively, to obtain the prediction confidence 231 and the model prediction probability 232. The prediction confidence 231 is used to correct the model prediction probability 232 to obtain the corrected prediction probability 233. The corrected prediction probability 233 and the classification label 212 of the sample work order data are used to perform backward error propagation training on the work order quality inspection model 220 to obtain the trained work order quality inspection model 250.

[0078] When using the trained work order quality inspection model 250, input work order data 241 is obtained and fed into a feature extraction network 251 to obtain a hidden layer representation 252 corresponding to the input work order data 241. The hidden layer representation 252 is fed into a quality inspection prediction network 253 and subjected to a normalization function 254 to obtain a model prediction probability 261. Based on the model prediction probability 261 and the quality inspection conditions 262, it is determined whether the input work order data 241 is abnormal work order data 263, and the abnormal work order data 263 is output.

[0079] Figure 3 The flowchart of the training method of a work order quality inspection model of the present application is shown. The method can be executed by a computer device. The method includes:

[0080] Step 310: Obtain sample work order data and classification labels of the sample work order data;

[0081] Work order data is data that records the completion status of work. In this embodiment, there are no restrictions on the recording method of work order data. For example, the work order data can be text content or voice data. There are no restrictions on the data source of work order data. For example, the data source of work order data includes but is not limited to at least one of the following sources: voice telephone customer service and text customer service.

[0082] The sample work order data is labeled work order data with classification labels. In this embodiment, there is no restriction on the labeling method of the sample work order data. Exemplarily, the method of labeling the sample work order data includes but is not limited to at least one of the following methods: manual labeling and machine learning model labeling.

[0083] The classification label of the sample work order data is used to indicate whether the sample work order data is abnormal work order data. In this embodiment, there is no restriction on the number of classification labels of the sample work order data.

[0084] Abnormal work order data is work order data that contains errors or risks. In this embodiment, there is no restriction on the error or risk category of the work order data. For example, the error or risk category of the work order data includes but is not limited to at least one of the following categories:

[0085] Wrong service category;

[0086] For example: Select the individual consumer business service type to provide services to group customers.

[0087] Errors in service content;

[0088] For example: when the service object needs an account number, provide the service object with an account nickname.

[0089] · Indifferent service attitude;

[0090] For example: not replying to messages when the service required by the service recipient cannot be provided.

[0091] · Service content is stiff;

[0092] For example: when the service required by the service recipient cannot be provided, only a welcome statement is replied.

[0093] Poor service performance.

[0094] For example: the service recipient sends a message of dissatisfaction, stupidity, or unresolved issues.

[0095] Step 320: Input the feature representation of the sample work order data into the work order quality inspection model to obtain the model prediction probability and prediction confidence;

[0096] The feature representation of the sample work order data is a set of feature vectors corresponding to the sample work order data. The feature representation of the sample work order data is obtained by mapping with the sample work order data.

[0097] Optionally, before obtaining the feature representation of the sample work order data through mapping, the sample work order data may be preprocessed, and the preprocessed work order data may be used to perform mapping to obtain the feature representation of the sample work order data. In this embodiment, there is no limitation on the preprocessing method of the sample work order data. Exemplary preprocessing methods for the sample work order data include, but are not limited to, at least one of the following methods:

[0098] Perform data processing on sample work order data;

[0099] For example, the second sample work order is subjected to time series transformation, frequency domain transformation, feature spectrum extraction, smoothing processing, and convolution operation.

[0100] Feed sample ticket data into the machine learning model.

[0101] For example, sample work order data is input into an encoding network to obtain preprocessed sample work order data. The encoding network is a machine learning model.

[0102] The work order quality inspection model is a machine learning model used to convert the feature representation of sample work order data into model prediction probability and prediction confidence. The sample work order data is used as sample data and input into the work order quality inspection model, which outputs the model prediction probability and prediction confidence:

[0103] p,c=f(x,θ)

[0104] Where c represents the prediction confidence, p represents the model prediction probability, f represents the work order quality inspection model, x represents the sample work order data, and θ represents the classification label of the sample work order data.

[0105] Prediction confidence describes the degree of confidence in the model's predicted probability, which is obtained using a machine learning model. For example, a higher prediction confidence indicates greater confidence that the model's predicted probability output by the work order quality inspection model can accurately represent the classification label of the sample work order data.

[0106] The prediction confidence is related to the labeling of the classification labels of the sample work order data. Exemplarily, the relationship between the prediction confidence and the labeling of the classification labels of the sample work order data includes, but is not limited to, at least one of the following relationships:

[0107] The prediction confidence is positively correlated with the number of times the classification label is annotated in all sample work order data;

[0108] For example, among 3,000 sample work order data, 1,000 are labeled with the first label and 1,500 are labeled with the second label. When the work order quality inspection model predicts the work order data to be tested as the first label, the prediction confidence is lower than when the work order data to be tested is predicted as the second label.

[0109] The prediction confidence is positively correlated with the number of occurrences of the classification label in the model prediction probability obtained by the work order quality inspection model;

[0110] For example: the first prediction confidence is lower than the second prediction confidence; the first prediction confidence is obtained when the work order quality inspection model predicts the classification label of the work order data to be tested as the first label for the 300th time; the second prediction confidence is obtained when the work order quality inspection model predicts the classification label of the work order data to be tested as the first label for the 500th time.

[0111] The model prediction probability is used to describe the probability that the sample work order data predicted by the work order quality inspection model belongs to the classification label.

[0112] Step 330: Calculate the revised prediction probability using the model prediction probability and prediction confidence and the classification label of the sample work order data;

[0113] The model prediction probability is corrected using the prediction confidence, the model prediction probability, and the classification labels of the sample work order data. Exemplarily, the higher the prediction confidence, the less correction is made to the model prediction probability. In this embodiment, there is no limitation on the method for correcting the model prediction probability using the prediction confidence. Exemplary methods for correcting the model prediction probability include, but are not limited to, the following methods: using the prediction confidence directly or indirectly as a correction factor to correct the model prediction probability.

[0114] For example, the relationship between the corrected prediction probability, the model prediction probability, the prediction confidence, and the classification label of the sample work order data is:

[0115] p i ′=c·p i +(1-c)·y i

[0116] Among them, p i ′ represents the corrected predicted probability of the i-th classification label, c represents the prediction confidence, p i represents the model prediction probability of the i-th classification label, y i Indicates the true probability that the sample work order data is labeled as the i-th classification label. i is obtained based on the classification label of the sample work order data. When the classification label of the sample work order data shows that the sample work order data belongs to the i-th classification label, y i = 1. When the classification label of the sample work order data shows that the sample work order data does not belong to the i-th classification label, y i =0.

[0117] Step 340: Calculate the model prediction error between the corrected prediction probability and the classification label of the sample work order data;

[0118] Compare the difference between the corrected predicted probability and the classification label of the sample work order data to obtain the model prediction error.

[0119] In this embodiment, there are no restrictions on the choice of model prediction error. For example, the model prediction error between the corrected prediction probability and the classification label of the sample work order data is typically obtained based on the corrected prediction probability and the classification label of the sample work order data, but this does not rule out the use of prediction confidence when calculating the model prediction error.

[0120] For example: the 0-1 loss between the corrected prediction probability and the classification label of the sample work order data is used as the model prediction error, the logarithmic loss between the corrected prediction probability and the classification label of the sample work order data is used as the model prediction error, and the cross entropy loss between the corrected prediction probability and the classification label of the sample work order data is used as the model prediction error.

[0121] Step 350: Perform backward error propagation training on the work order quality inspection model based on the model prediction error to obtain a trained work order quality inspection model.

[0122] Based on the model prediction error, the backpropagation algorithm is used to update the parameters of the work order quality inspection model. Using multiple groups of information including sample work order data and classification labels of sample work order data, the model prediction error is compared multiple times and the parameters of the work order quality inspection model are updated to improve the accuracy of the prediction probability of the prediction model.

[0123] To summarize, the method provided in this embodiment adds a confidence prediction network, describes the credibility of the model prediction probability through prediction confidence, and adds prediction confidence as an indicator for evaluating the work order quality inspection model; uses prediction confidence to correct the model prediction probability, and uses the corrected prediction probability obtained after correction to calculate the error between the corrected prediction probability and the classification label of the sample work order data, thereby increasing the dimension of evaluating the model prediction error during the work order quality inspection model training process, reducing the impact of the labeling of the classification labels of the sample work order data on the work order quality inspection model, improving the training effect of the work order quality inspection model, and improving the stability of the work order quality inspection model in performing work order quality inspection.

[0124] Figure 4 The flowchart of the training method of a work order quality inspection model of the present application is shown. The method can be executed by a computer device. The method includes:

[0125] Step 310, step 330, step 340, step 350, refer to the above Figure 3 The steps in the illustrated embodiment will not be repeated in this embodiment.

[0126] Step 322: Input the feature representation of the sample work order data into the feature extraction network to obtain the hidden layer representation of the feature representation of the sample work order data;

[0127] The feature representation of the sample work order data is a set of feature vectors corresponding to the sample work order data. The feature representation of the sample work order data is obtained by mapping with the sample work order data.

[0128] The hidden layer representation of the feature representation of the sample work order data includes the feature information of the sample work order data.

[0129] In this embodiment, there is no restriction on the structure of the feature extraction network. Exemplarily, the structure of the feature extraction network includes but is not limited to at least one of the following structures: a network in network (NIN) structure, a convolutional neural network (CNN) structure, and a recurrent neural network (RNN) structure. Those skilled in the art will appreciate that the above structures can be used independently to construct a feature extraction network to extract feature information from sample work order data; the above structures can also be used in combination to construct a feature extraction network.

[0130] Optionally, the feature extraction network includes a cascade of n1-layer feedforward neural network and n2-layer fully connected neural network, wherein n1 and n2 are both integers greater than 1. Exemplarily, the feedforward neural network and the fully connected neural network are cascaded.

[0131] The n1-layer feedforward neural network is used to extract feature vectors of the sample work order data, and the n2-layer fully connected neural network is used to process the feature vectors of the sample work order data into high-order feature vectors. In this embodiment, there is no restriction on the number of feature vectors of the sample work order data, that is, the n1-layer feedforward neural network can extract one or more feature information of the sample work order data, and the features of the sample work order data are represented by feature vectors. When the n1-layer feedforward neural network can extract multiple feature information of the sample work order data, the features of the sample work order data can be extracted separately using independent feedforward neural networks or extracted using feedforward neural networks with a connected relationship.

[0132] Step 324: Input the hidden layer representation into the quality inspection prediction network to obtain the model prediction probability;

[0133] In this embodiment, no restriction is imposed on the structure of the quality inspection prediction network.

[0134] The model prediction probability is used to describe the probability that the sample work order data predicted by the work order quality inspection model belongs to the classification label.

[0135] Optionally, the model prediction probability is normalized (Softmax) to a number greater than or equal to 0 and less than or equal to 1. The sum of the model prediction probabilities of all classification labels is equal to 1.

[0136] Optionally, the quality inspection prediction network includes an n3-layer feedforward neural network, where n3 is an integer greater than 1.

[0137] Step 326: Input the hidden layer representation into the confidence prediction network to obtain the prediction confidence.

[0138] In this embodiment, no restriction is imposed on the structure of the confidence prediction network.

[0139] Prediction confidence describes the degree of confidence in the model's predicted probability, which is obtained using a machine learning model. For example, a higher prediction confidence indicates greater confidence that the model's predicted probability output by the work order quality inspection model can accurately represent the classification label of the sample work order data.

[0140] The prediction confidence is related to the labeling of the classification labels of the sample work order data.

[0141] Optionally, the prediction confidence is processed using a Sigmoid growth curve, and the prediction confidence is a number greater than or equal to 0 and less than or equal to 1. For example, when there is complete confidence that the model prediction probability obtained by the work order quality inspection model is the same as the classification label of the sample work order data, the prediction confidence is 1; the prediction confidence is positively correlated with the degree of confidence that the model prediction probability obtained by the work order quality inspection model is the same as the classification label of the sample work order data.

[0142] When the model prediction probability is normalized and the prediction confidence is processed using an S-shaped growth curve, the model prediction probability and prediction confidence output by the work order quality inspection model are both numbers greater than or equal to 0 and less than or equal to 1:

[0143] p,c=f(x,θ)

[0144]

[0145] Among them, c represents the prediction confidence, p represents the model prediction probability, and p i represents the model prediction probability of the i-th classification label, f represents the work order quality inspection model, x represents the sample work order data, θ represents the classification label of the sample work order data, and M represents the number of classification labels of the sample work order data.

[0146] Optionally, the confidence prediction network includes an n4-layer feedforward neural network, where n4 is an integer greater than 1.

[0147] It should be noted that in this embodiment, there is no restriction on the relationship between the quality inspection prediction network and the confidence prediction network. For example, the relationship between the quality inspection prediction network and the confidence prediction network includes, but is not limited to, the following relationships: two independent and parallel network structures, or two network structures with a cascade relationship, or two independent network structures that share some network layers and are separated by a network layer.

[0148] It should be noted that, in this embodiment, there is no limitation on the timing relationship between step 324 and step 326 . For example, step 324 can be performed before, after, or simultaneously with step 326 .

[0149] In summary, the method provided in this embodiment uses the latent layer representation of sample work order data, employs a quality inspection prediction network and a confidence prediction network to predict the model's prediction probability and prediction confidence, respectively. This adds prediction confidence as a metric for evaluating the work order quality inspection model. This method fully utilizes the feature information of the sample work order data extracted by the feature extraction network.

[0150] Figure 5 The flowchart of the training method of a work order quality inspection model of the present application is shown. The method can be executed by a computer device. The method includes:

[0151] Step 310, step 320, step 330, step 350, refer to the above Figure 3 The steps in the illustrated embodiment will not be repeated in this embodiment.

[0152] Step 342: Calculate a first error based on the corrected prediction probability and the classification label of the sample work order data;

[0153] The first error is obtained based on the corrected prediction probability and the classification label of the sample work order data. In this embodiment, there is no restriction on the selection of the first error.

[0154] Exemplarily, the selection of the first error includes but is not limited to at least one of the following errors: the negative log-likelihood function of the classification label of the sample work order data with respect to the corrected predicted probability, the 0-1 loss between the corrected predicted probability and the classification label of the sample work order data, the logarithmic loss between the corrected predicted probability and the classification label of the sample work order data, and the cross-entropy loss between the corrected predicted probability and the classification label of the sample work order data.

[0155] Exemplarily, the negative log-likelihood function of the classification label of the sample work order data with respect to the corrected prediction probability is selected as the first error:

[0156]

[0157] Among them, L t represents the first error, M represents the number of classification labels of the sample work order data, and p i ′ represents the corrected predicted probability of the i-th classification label, y i Represents the true probability that the sample work order data is labeled as the i-th classification label.

[0158] Step 344: Based on the prediction confidence, correct the first error to obtain the model prediction error.

[0159] Prediction confidence describes the reliability of the model's predicted probability, which is obtained using a machine learning model. Prediction confidence is related to the classification labels of the sample work order data. Prediction confidence is used as an evaluation metric to correct the first error.

[0160] In this embodiment, there is no limitation on the method for correcting the first error based on the prediction confidence. Exemplarily, the method for correcting the first error includes, but is not limited to, at least one of the following methods: using the prediction confidence directly or indirectly as a correction coefficient to correct the first error, constructing a correction error function based on the prediction confidence, or constructing a correction error function based on the prediction confidence, the correction prediction probability, and the classification label of the sample work order data.

[0161] For example, Figure 6 As shown, step 344 includes the following sub-steps:

[0162] Step 344a: Calculate a second error based on the prediction confidence;

[0163] The second error is a function used to correct the first error. In this embodiment, there is no restriction on the selection of the second error. For example, the prediction confidence can be used directly or indirectly as the second error.

[0164] Exemplarily, the negative logarithmic function of the prediction confidence is selected as the second error:

[0165] L c = -log(c)

[0166] Among them, L c represents the second error, and c represents the prediction confidence.

[0167] Step 344b: Based on the first error and the second error, obtain the model prediction error between the corrected prediction probability and the classification label of the sample work order data.

[0168] The model prediction error is obtained based on the first error and the second error, and is used to represent the error between the corrected prediction probability and the classification label of the sample work order data. In this embodiment, there is no limitation on the method for obtaining the model prediction error. Exemplary methods for obtaining the model prediction error include but are not limited to at least one of the following methods:

[0169] Using mathematical operations to obtain the model prediction error based on the first error and the second error;

[0170] Exemplarily, the mathematical operation method includes but is not limited to at least one of the following methods: summing, subtracting, calculating relative error, and calculating cross entropy error.

[0171] For example, the model prediction error is the sum of the first error and the second error, and the balance factor is used to balance the first error and the second error:

[0172] L=L t +λL c

[0173] Among them, L represents the model prediction error, L t Represents the first error, L c represents the second error, λ represents the balance factor, λ∈[0,1], and λ is a number greater than or equal to 0 and less than or equal to 1.

[0174] Input the first error and the second error into the machine learning model to obtain the model prediction error.

[0175] Exemplarily, the machine learning model includes but is not limited to at least one of the following methods: Convolutional Neural Networks (CNN) and Transformer.

[0176] Optionally, in the model prediction error, the first error and the prediction confidence are positively correlated, and the second error and the prediction confidence are negatively correlated;

[0177] or,

[0178] In the model prediction error, the first error and prediction confidence show a negative correlation, and the second error and prediction confidence show a positive correlation.

[0179] Taking the example where the first error is the negative log-likelihood function of the classification label of the sample work order data with respect to the corrected prediction probability, and the second error is the negative logarithmic function of the prediction confidence: the first error and the prediction confidence show a positive correlation, and the second error and the prediction confidence show a negative correlation.

[0180] When the prediction confidence indicates that the relationship between the model prediction probability and the classification label tends to be similar, the revised prediction probability tends to the model prediction probability, the first error tends to the error between the model prediction probability and the classification label, and the first error tends to 0; when the prediction confidence indicates that the relationship between the model prediction probability and the classification label tends to be different, the revised prediction probability tends to the true probability of the classification label of the sample work order data, the first error tends to 0, and the second error tends to infinity.

[0181] In summary, the method provided in this embodiment calculates the second error based on prediction confidence, using prediction confidence as a metric for evaluating model prediction error during work order quality inspection model training. Using prediction confidence as a factor influencing model prediction error improves the training effectiveness of the work order quality inspection model.

[0182] Those skilled in the art will appreciate that the above embodiments may be implemented independently, or the above embodiments may be freely combined to form new embodiments to implement the training method of the work order quality inspection model of the present application.

[0183] Figure 6 The flowchart of the training method of a work order quality inspection model of the present application is shown. The method can be executed by a computer device. The method includes:

[0184] Step 310, step 320, step 330, step 340, step 350, refer to the above Figure 3 The steps in the illustrated embodiment will not be repeated in this embodiment.

[0185] Step 360: Obtain input work order data;

[0186] The input work order data is the work order data input when using the work order quality inspection model. In this embodiment, there is no restriction on the recording method of the input work order data. For example, the input work order data can be text content or voice data.

[0187] Step 370: Input the input work order data into the work order quality inspection model to obtain the model prediction probability;

[0188] The work order quality inspection model is obtained after backward error propagation training.

[0189] It should be noted that the input work order data is input into the work order quality inspection model to obtain the model prediction probability, that is, when the trained work order quality inspection model is used, the prediction confidence is not output.

[0190] Optionally, when the work order quality inspection model includes a feature extraction network, a quality inspection prediction network, and a confidence prediction network, after the work order quality inspection model is obtained after backward error propagation training, if the confidence prediction network does not affect the quality inspection prediction network, the confidence prediction network in the work order quality inspection model is deleted from the work order quality inspection model. Exemplary situations where the confidence prediction network does not affect the quality inspection prediction network include but are not limited to the following situations:

[0191] The quality inspection prediction network and the confidence prediction network are two independent and parallel network structures;

[0192] Alternatively, the quality inspection prediction network and the confidence prediction network share some network layers, and the two network structures are independent of each other after one network layer, and only the independent part of the confidence prediction network after the network layer is deleted.

[0193] Step 380: Determine the work order data whose model prediction probability meets the quality inspection conditions as abnormal work order data, and output the abnormal work order data.

[0194] Abnormal work order data refers to work order data that contains errors or risks. In this embodiment, there are no restrictions on the error or risk categories of work order data. Exemplary error or risk categories include, but are not limited to, at least one of the following: incorrect service category, incorrect service content, indifferent service attitude, stilted service content, and poor service performance. Abnormal work order data refers to part or all of the input work order data.

[0195] The quality inspection conditions are used to determine whether the input work order data is abnormal work order data. The quality inspection conditions are set based on the model prediction probability corresponding to the input work order data. In this embodiment, there is no restriction on the method of setting the quality inspection conditions. For example, the quality inspection conditions include but are not limited to the following conditions:

[0196] The model predicts a probability greater than m1;

[0197] Here, m1 is a number greater than or equal to 0 and less than 1.

[0198] Exemplarily, methods for determining m1 include but are not limited to the following methods: a fixed value, or the median of the input work order data in the model prediction probability, or the average of the input work order data in the model prediction probability.

[0199] For example: the quality inspection condition is that the model prediction probability is greater than 0.84;

[0200] Alternatively, the input work order data is sorted from high to low according to the model prediction probability, and the first m2 input work order data are sorted.

[0201] Here, m2 is an integer greater than 0 and less than or equal to the number of input work order data.

[0202] Exemplarily, methods for determining m2 include but are not limited to the following methods: a fixed value, a fixed ratio of the quantity of input work order data.

[0203] For example: the quality inspection condition is to sort the input work order data from high to low according to the model prediction probability, and the top 1,500 input work order data are sorted.

[0204] To sum up, the method provided in this embodiment provides a method for using a trained work order quality inspection model. By setting quality inspection conditions, abnormal work order data is determined based on the quality inspection results of the work order quality inspection model and the amount of input work order data, thereby improving the quality inspection efficiency of the work order quality inspection model and improving the stability of the work order quality inspection model in performing work order quality inspection.

[0205] Table 1

[0206]

[0207]

[0208] The performance of the work order quality inspection model was verified using a labeled dataset consisting of annotated work order data and a live dataset consisting of unlabeled work order data. The quality inspection condition was set to a prediction probability greater than 0.84. The experimental results, shown in Table 1, show that the accuracy and stability of the work order quality inspection model were significantly improved after adding the confidence prediction network.

[0209] Figure 8 A block diagram of a training device for a work order quality inspection model provided by an exemplary embodiment of the present application is shown. The device includes:

[0210] A first acquisition module 810 is configured to acquire sample work order data and a classification label of the sample work order data, wherein the classification label is used to indicate whether the sample work order data is abnormal work order data;

[0211] A first processing module 820 is configured to input the feature representation of the sample work order data into a work order quality inspection model to obtain a model prediction probability and a prediction confidence, wherein the prediction confidence is used to describe the credibility of the model prediction probability obtained by the prediction;

[0212] A correction module 830 is configured to calculate a corrected prediction probability using the model prediction probability, the prediction confidence, and the classification label of the sample work order data;

[0213] A calculation module 840 is configured to calculate a model prediction error between the corrected prediction probability and the classification label of the sample work order data;

[0214] The training module 850 is used to perform backward error propagation training on the work order quality inspection model based on the model prediction error to obtain the trained work order quality inspection model.

[0215] In an optional design of this embodiment, the work order quality inspection model includes a feature extraction network, a quality inspection prediction network, and a confidence prediction network;

[0216] The first processing module 820 is further configured to:

[0217] Inputting the feature representation of the sample work order data into the feature extraction network to obtain a hidden layer representation of the feature representation of the sample work order data;

[0218] Inputting the hidden layer representation into the quality inspection prediction network to obtain the model prediction probability;

[0219] The hidden layer representation is input into the confidence prediction network to obtain the prediction confidence.

[0220] In an optional design of this embodiment,

[0221] The feature extraction network includes an n1-layer feedforward neural network and an n2-layer fully connected neural network;

[0222] The quality inspection prediction network includes an n3-layer feedforward neural network;

[0223] The confidence prediction network comprises an n4-layer feedforward neural network;

[0224] Wherein, n1, n2, n3 and n4 are all integers greater than 1.

[0225] In an optional design of this embodiment, the calculation module 840 includes:

[0226] an error calculation unit 841, configured to calculate a first error based on the corrected prediction probability and the classification label of the sample work order data;

[0227] The error correction unit 842 is used to correct the first error based on the prediction confidence to obtain the model prediction error.

[0228] In an optional design of this embodiment, the error correction unit 842 is further configured to:

[0229] Calculating a second error based on the prediction confidence;

[0230] Based on the first error and the second error, a model prediction error between the corrected prediction probability and the classification label of the sample work order data is obtained.

[0231] In an optional design of this embodiment,

[0232] In the model prediction error, the first error and the prediction confidence are positively correlated, and the second error and the prediction confidence are negatively correlated;

[0233] or,

[0234] In the model prediction error, the first error and the prediction confidence show a negative correlation, and the second error and the prediction confidence show a positive correlation.

[0235] In an optional design of this embodiment, the first error is a negative log-likelihood function of the classification label of the sample work order data with respect to the corrected predicted probability.

[0236] In an optional design of this embodiment, the second error is a negative logarithmic function of the prediction confidence.

[0237] In an optional design of this embodiment, the device further includes:

[0238] The second acquisition module 860 is used to obtain input work order data;

[0239] The second processing module 870 is configured to input the input work order data into the work order quality inspection model to obtain a model prediction probability;

[0240] Output module 880 is used to determine the work order data whose model prediction probability meets the quality inspection conditions as abnormal work order data, and output the abnormal work order data, wherein the quality inspection conditions are set based on the model prediction probability corresponding to the input work order data, and the abnormal work order data is part or all of the data in the input work order data.

[0241] In an optional design of this embodiment, the quality inspection conditions include:

[0242] The model predicts a probability greater than m1, where m1 is a number greater than or equal to 0 and less than 1;

[0243] or,

[0244] The input work order data are sorted from high to low according to the rule of the model prediction probability, and the first m2 input work order data are sorted, where m2 is an integer greater than 0 and less than or equal to the number of the input work order data.

[0245] It should be noted that the device provided in the above embodiment only uses the division of the above-mentioned functional modules as an example to implement its functions. In actual applications, the above-mentioned functions can be assigned to different functional modules according to actual needs, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0246] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0247] An embodiment of the present application also provides a computer device, which includes: a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the training method of the work order quality inspection model provided by the above-mentioned method embodiments.

[0248] Optionally, the computer device is a server. For example, Figure 9 This is a structural block diagram of a server provided by an exemplary embodiment of the present application.

[0249] Typically, the server 900 includes a processor 901 and a memory 902 .

[0250] The processor 901 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 901 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 901 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 901 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 901 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0251] The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 902 is used to store at least one instruction, which is executed by the processor 901 to implement the training method of the work order quality inspection model provided in the method embodiment of the present application.

[0252] In some embodiments, the server 900 may further optionally include: an input interface 903 and an output interface 904. The processor 901, the memory 902 and the input interface 903 and the output interface 904 may be connected via a bus or a signal line. Each peripheral device may be connected to the input interface 903 and the output interface 904 via a bus, a signal line or a circuit board. The input interface 903 and the output interface 904 may be used to connect at least one I / O (Input / Output) related peripheral device to the processor 901 and the memory 902. In some embodiments, the processor 901, the memory 902 and the input interface 903 and the output interface 904 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 901, the memory 902 and the input interface 903 and the output interface 904 may be implemented on a separate chip or circuit board, which is not limited in the embodiments of the present application.

[0253] Those skilled in the art will understand that Figure 9 The structure shown in the figure does not constitute a limitation on the server 900, and the server 900 may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0254] In an exemplary embodiment, a chip is also provided, which includes a programmable logic circuit and / or program instructions. When the chip runs on a computer device, it is used to implement the training method of the work order quality inspection model described in the above aspects.

[0255] In an exemplary embodiment, a computer program product or computer program is also provided. The computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the training method for the work order quality inspection model provided in each of the above method embodiments.

[0256] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores at least one program code. When the program code is loaded and executed by a processor of a computer device, the training method of the work order quality inspection model provided by the above-mentioned method embodiments is implemented.

[0257] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0258] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A training method for a work order quality inspection model, characterized in that: The work order quality inspection model includes a feature extraction network, a quality inspection prediction network, and a confidence prediction network. The method includes: Obtaining sample work order data and a classification label for the sample work order data, wherein the classification label is used to indicate whether the sample work order data is abnormal work order data; Inputting the feature representation of the sample work order data into the feature extraction network to obtain a hidden layer representation of the feature representation of the sample work order data; Inputting the hidden layer representation into the quality inspection prediction network to obtain a model prediction probability; and inputting the hidden layer representation into the confidence prediction network to obtain a prediction confidence, wherein the prediction confidence is used to describe the credibility of the predicted model prediction probability; Calculating a revised prediction probability using the model prediction probability, the prediction confidence, and the classification label of the sample work order data; Calculating a model prediction error between the corrected prediction probability and the classification label of the sample work order data; Backward error propagation training is performed on the work order quality inspection model based on the model prediction error to obtain the trained work order quality inspection model.

2. The method according to claim 1, characterized in that The feature extraction network includes a cascaded n1-layer feedforward neural network and an n2-layer fully connected neural network; The quality inspection prediction network includes an n3-layer feedforward neural network; The confidence prediction network includes an n4-layer feedforward neural network; Wherein, n1, n2, n3 and n4 are all integers greater than 1.

3. The method according to claim 1 or 2, characterized in that Calculating the model prediction error between the corrected prediction probability and the classification label of the sample work order data includes: Calculating a first error based on the corrected prediction probability and the classification label of the sample work order data; Based on the prediction confidence, the first error is corrected to obtain the model prediction error.

4. The method according to claim 3, characterized in that The correcting the first error based on the prediction confidence to obtain the model prediction error includes: Calculating a second error based on the prediction confidence; Based on the first error and the second error, a model prediction error between the corrected prediction probability and the classification label of the sample work order data is obtained.

5. The method according to claim 4, characterized in that In the model prediction error, the first error and the prediction confidence are positively correlated, and the second error and the prediction confidence are negatively correlated; or, In the model prediction error, the first error and the prediction confidence show a negative correlation, and the second error and the prediction confidence show a positive correlation.

6. The method according to claim 4, characterized in that The first error is a negative log-likelihood function of the classification label of the sample work order data with respect to the corrected prediction probability.

7. The method according to claim 4, characterized in that The second error is a negative logarithmic function of the prediction confidence.

8. The method according to claim 1 or 2, characterized in that The method further comprises: Get input work order data; Inputting the input work order data into the work order quality inspection model to obtain the model prediction probability; Determine the work order data whose model prediction probability meets the quality inspection condition as abnormal work order data, and output the abnormal work order data, wherein the quality inspection condition is set based on the model prediction probability corresponding to the input work order data, and the abnormal work order data is part or all of the data in the input work order data.

9. The method according to claim 8, characterized in that The quality inspection conditions include: The model prediction probability is greater than a probability threshold m1, where m1 is a number greater than or equal to 0 and less than 1; or, The input work order data are sorted from high to low according to the rule of the model prediction probability, and the first m2 input work order data are sorted, where m2 is an integer greater than 0 and less than or equal to the number of the input work order data.

10. A training device for a work order quality inspection model, characterized in that: The work order quality inspection model includes a feature extraction network, a quality inspection prediction network and a confidence prediction network, and the device includes: A first acquisition module is used to acquire sample work order data and a classification label of the sample work order data, wherein the classification label is used to indicate whether the sample work order data is abnormal work order data; A first processing module, configured to input the feature representation of the sample work order data into the feature extraction network to obtain a hidden layer representation of the feature representation of the sample work order data; Inputting the hidden layer representation into the quality inspection prediction network to obtain a model prediction probability; and inputting the hidden layer representation into the confidence prediction network to obtain a prediction confidence, wherein the prediction confidence is used to describe the credibility of the predicted model prediction probability; A correction module, configured to calculate a corrected prediction probability using the model prediction probability, the prediction confidence, and the classification label of the sample work order data; a calculation module, configured to calculate a model prediction error between the corrected prediction probability and the classification label of the sample work order data; A training module is used to perform backward error propagation training on the work order quality inspection model based on the model prediction error to obtain the trained work order quality inspection model.

11. The device according to claim 10, characterized in that The device further comprises: The second acquisition module is used to obtain input work order data; A second processing module is configured to input the input work order data into the work order quality inspection model to obtain a model prediction probability; An output module is used to determine the work order data whose model prediction probability meets the quality inspection conditions as abnormal work order data, and to output the abnormal work order data, wherein the quality inspection conditions are set based on the model prediction probability corresponding to the input work order data, and the abnormal work order data is part or all of the data in the input work order data.

12. A computer device, characterized in that: The computer device includes: a processor and a memory, wherein the memory stores at least one program; the processor is used to execute the at least one program in the memory to implement the training method of the work order quality inspection model as described in any one of claims 1 to 9.

13. A computer-readable storage medium, characterized in that The readable storage medium stores executable instructions, which are loaded and executed by a processor to implement the training method of the work order quality inspection model as described in any one of claims 1 to 9.

14. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. The processor reads and executes the computer instructions from the computer-readable storage medium to implement the training method of the work order quality inspection model as described in any one of claims 1 to 9.

Images