Work order quality evaluation method and device and work order quality evaluation system

Abstract

The invention provides a work order quality evaluation method and device and a work order quality evaluation system, and the method comprises the steps: 1, obtaining a standard score of each work order according to a scoring standard, and obtaining a work order score of each work order according to the standard score of each work order and an artificial score; training an SVR model by using the spatial feature vector and the work order score of each work order to obtain a work order scoring model; thirdly, inputting the spatial feature vectors into a work order scoring model to obtain model scores of all work orders, and adjusting scoring standards according to the model scores of all the work orders and manual scores; a repeating step: sequentially repeating the first step, the second step and the third step at least once to obtain a plurality of work order scoring models; determining the work order scoring model with the minimum average scoring error as a final work order scoring model; and scoring the new work order by adopting the final work order scoring model. According to the method, the problem of relatively low efficiency caused by the fact that work order scoring depends on manual scoring in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent control evaluation, and in particular, to a work order quality evaluation method, device, computer-readable storage medium, computer program product, and work order quality evaluation system. Background Art

[0002] As a scoring tool based on natural language processing (NLP) technology, the Automated Essay Scoring system (AES) was initially mainly designed for scoring essays in large-scale standardized tests, such as English writing tests or academic paper evaluations. Its core lies in automatically evaluating the grammatical correctness, lexical richness, logical coherence, and structural integrity of texts through algorithms. The advantage of this system is that it can quickly process a large amount of text, provide consistent and objective scoring results, reduce the burden of manual grading, and improve the scoring efficiency.

[0003] However, in the specific field of work order filling and evaluation, the direct application of AES faces challenges. The text content of work orders is essentially different from that of exam essays or academic papers. They pay more attention to the integrity and accuracy of information, as well as the clarity and precision of expression, to ensure that work order requests or problems can be quickly and correctly understood and processed. Traditional AES algorithms tend to focus on the fluency of language and rhetorical splendor, which may not be the most important considerations in work order filling. There are certain differences in their focus dimensions, that is, the current market lacks a system or method for scoring work orders, resulting in the scoring of work orders often relying on manual scoring. Summary of the Invention

[0004] The main purpose of this application is to provide a work order quality evaluation method, device, computer-readable storage medium, computer program product, and work order quality evaluation system to at least solve the problem of low efficiency caused by the excessive dependence on manual scoring in the existing technology for work order scoring.

[0005] To achieve the above object, according to one aspect of the present application, there is provided a work order quality evaluation method, including: a first step of obtaining a plurality of work orders with manual scores, obtaining the standard scores of each work order according to the scoring criteria, and calculating the work order scores of each work order based on the standard scores and the manual scores of each work order. The scoring criteria are to evaluate the integrity, readability, and refinement of the work order; a second step of extracting feature vectors from the text content of each work order to obtain the spatial feature vectors of each work order, and training an SVR model using the spatial feature vectors and the work order scores of each work order to obtain a work order scoring model; a third step of inputting all the spatial feature vectors into the work order scoring model to obtain the model scores of each work order, and adjusting the scoring criteria according to the model scores and the manual scores of each work order; repeating the steps, sequentially repeating the first step, the second step, and the third step at least once to obtain a plurality of work order scoring models; calculating the average value of the scoring errors between the model scores of the plurality of work orders output by each work order scoring model and the corresponding manual scores to obtain a plurality of average scoring errors, and determining the work order scoring model with the smallest average scoring error as the final work order scoring model. The average scoring error corresponds to the work order scoring model one by one; using the final work order scoring model to score a new work order, and triggering an update content notification when the model score of the new work order is lower than a predetermined threshold to trigger an update of the text content of the new work order.

[0006] Optionally, obtaining a plurality of work orders with manual scores and obtaining the standard scores of each work order according to the scoring criteria includes: evaluating the integrity of the work order according to the missing situation of the keywords of the work order to obtain an integrity score. The keywords include the category, problem description, and transfer department; evaluating the readability of the work order according to the number of sentences, the number of words, and the number of punctuation marks of the work order using the Kincaid readability score formula to obtain a readability score. The Kincaid readability score formula is where r is the readability score, w is the number of words, s is the number of sentences, and syl is the number of punctuation marks; calculating the deviation ratio between the text length of the work order and the average text length of the historical work orders to evaluate the refinement of the work order and obtain a refinement score; setting the weights of the integrity score, the readability score, and the refinement score; calculating the standard score S of each work order according to the integrity score and the weight of the integrity score, the readability score and the weight of the readability score, and the refinement score and the weight of the refinement score of each work order sdrad =(S comp *W comp +Sread *W read +S conc *W conc ) / 2, where S comp is the integrity score, W comp is the weight of the integrity score, S read is the readability score, W read is the weight of the readability score, S conc is the refinement score, W conc is the weight of the refinement score.

[0007] Optionally, the work order score of each work order is calculated according to the standard score and the manual score of each work order, including: calculating the work order score of each work order according to the standard score and the manual score of the work order where n is the number of manual scores of the work order, S i is the i-th manual score of the work order.

[0008] Optionally, feature vector extraction is performed on the text content of each work order to obtain the spatial feature vector of each work order, including: processing the text content of each work order using the word2vec algorithm to obtain the spatial feature vector D of each work order new(n*1) =W n*n D old(n*1) where D old(n*1) represents the work order represented by one-hot encoding, W n*n is calculated from word vectors, and W(i,j) is the cosine similarity between word i and word j.

[0009] Optionally, the SVR model is trained using the spatial feature vector and the work order score of each work order to obtain a work order scoring model, including: training the model function y = wx + b according to each spatial feature vector and each work order score, where x is the spatial feature vector, w is the weight parameter, b is the bias parameter, and y is the model score of the work order; adjusting the weight parameter and the bias parameter to minimize the error between the model score and the work order score, and determining the current weight parameter and the current bias parameter as the final weight parameter and the final bias parameter.

[0010] Optionally, all of the spatial feature vectors are input into the work order scoring model to obtain a model score for each of the work orders, and the scoring criteria are adjusted according to the model score and the manual score of each of the work orders, including: arranging the model scores of each of the work orders from large to small to obtain a first scoring set, and arranging the manual scores of each of the work orders from large to small to obtain a second scoring set; calculating the average score of a specific number of the model scores that are ranked high in the first scoring set to obtain a first average score, and calculating the average score of the specific number of the model scores that are ranked high in the second scoring set to obtain a second average score; adjusting the weight of the completeness score, the weight of the readability score, and the weight of the conciseness score according to the first average score and the second average score. Among them, S model is the first average score, S hum is the second average score, W old is one of the weight of the completeness score, the weight of the readability score, and the weight of the conciseness score, W new is the adjusted W old .

[0011] To achieve the above object, according to one aspect of the present application, there is provided a work order quality evaluation device, including: a calculation unit, configured to execute the first step of obtaining a plurality of work orders with manual scores, obtaining the standard scores of each work order according to the scoring criteria, and calculating the work order scores of each work order based on the standard scores and the manual scores of each work order, where the scoring criteria are to evaluate the integrity, readability, and refinement of the work order; a training unit, configured to execute the second step of extracting feature vectors from the text content of each work order to obtain the spatial feature vectors of each work order, and training an SVR model using the spatial feature vectors and the work order scores of each work order to obtain a work order scoring model; an adjustment unit, configured to execute the third step of inputting all the spatial feature vectors into the work order scoring model to obtain the model scores of each work order, and adjusting the scoring criteria according to the model scores and the manual scores of each work order; a repetition unit, configured to execute the repetition step of sequentially repeating the first step, the second step, and the third step at least once to obtain a plurality of work order scoring models; a determination unit, configured to calculate the average value of the scoring errors between the model scores of the plurality of work orders output by each work order scoring model and the corresponding manual scores to obtain a plurality of average scoring errors, and determining the work order scoring model with the smallest average scoring error as the final work order scoring model, where the average scoring error corresponds one-to-one with the work order scoring model; a control unit, configured to score a new work order using the final work order scoring model, and trigger an update content notification to trigger an update of the text content of the new work order when the model score of the new work order is lower than a predetermined threshold.

[0012] According to another aspect of the present application, there is provided a computer-readable storage medium, where the computer-readable storage medium includes a stored program, and where, when the program runs, it controls any one of the methods in the device where the computer-readable storage medium is located.

[0013] According to yet another aspect of the present application, there is provided a computer program product, including a computer program, where the computer program, when executed by a processor, implements any one of the methods.

[0014] According to another aspect of the present application, there is provided a work order quality evaluation system, including: one or more processors, a memory, and one or more programs, where the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include those for executing any one of the methods.

[0015] Applying the technical solution of the present application in the above-mentioned method for evaluating the quality of work orders includes: the first step, obtaining multiple work orders with manual scores, and obtaining the standard scores of each of the above work orders according to the scoring criteria. The work order scores of each of the above work orders are calculated based on the above standard scores and the above manual scores. The above scoring criteria are used to evaluate the integrity, readability, and refinement of the above work orders; the second step, extracting feature vectors from the text content of each of the above work orders to obtain the spatial feature vectors of each of the above work orders, and using the above spatial feature vectors and the above work order scores of each of the above work orders to train the SVR model to obtain a work order scoring model; the third step, inputting all the above spatial feature vectors into the above work order scoring model to obtain the model scores of each of the above work orders, and adjusting the above scoring criteria according to the above model scores and the above manual scores of each of the above work orders; repeating the steps, successively repeating the above first step, the above second step, and the above third step at least once to obtain multiple above work order scoring models; calculating the average value of the scoring errors between the model scores of multiple above work orders output by each of the above work order scoring models and the corresponding manual scores to obtain multiple average scoring errors, and determining the above work order scoring model with the smallest above average scoring error as the final work order scoring model. The above average scoring error corresponds to the above work order scoring model one by one; using the above final work order scoring model to score a new work order. In the case where the above model score of the above new work order is lower than a predetermined threshold, a notification for updating the filled content is triggered to trigger the update of the above text content of the above new work order. The present application calculates the work order score of a work order through the scoring criteria and the manual score, and uses the work order score and the spatial feature vector of each work order to train the SVR model to obtain a work order scoring model. Then, the work order scoring model is used to score all the spatial feature vectors to obtain a model score, and the scoring criteria are adjusted according to the model score and the manual score of each work order. The above method is continuously used to train different multiple work orders to obtain multiple work order scoring models. Finally, the average scoring error between the multiple model scores output by each work order scoring model and the manual score is calculated, and the work order scoring model corresponding to the smallest average scoring error is determined as the final scoring model to score the input work order. Through this model, the text content of the work order can be quickly scored to timely discover the deficiencies in the work order content, solving the problem of low efficiency caused by excessive dependence on manual scoring in the prior art for work order scoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 FIG. shows a hardware structure block diagram of a mobile terminal for implementing a method for evaluating the quality of work orders provided in an embodiment of the present application;

[0017] Figure 2 FIG. shows a flowchart of a method for evaluating the quality of work orders provided in an embodiment of the present application;

[0018] Figure 3 Shows the first-round model training flowchart of a work order quality evaluation system provided according to an embodiment of the present application;

[0019] Figure 4 Shows the second-round model training flowchart of a work order quality evaluation system provided according to an embodiment of the present application;

[0020] Figure 5 Shows the structural block diagram of a work order quality evaluation device provided according to an embodiment of the present application.

[0021] Among them, the above-mentioned drawings include the following reference numerals:

[0022] 102, processor; 104, memory; 106, transmission device; 108, input / output device. Detailed implementation manners

[0023] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments may be combined with each other. The present application will be described in detail below with reference to the drawings and in combination with the embodiments.

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without making creative efforts shall fall within the protection scope of the present application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that such data may be interchanged under appropriate circumstances so that the embodiments of the present application described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these process, method, product or device.

[0026] As introduced in the background art, there is a lack of a system or method for scoring work orders in the prior art, resulting in the scoring of work orders often relying on manual scoring. To solve this technical problem, the embodiments of the present application provide a work order quality evaluation method, device, computer-readable storage medium, computer program product and work order quality evaluation system.

[0027] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0028] The method embodiments provided in the embodiments of the present application can be executed on a mobile terminal, a computer terminal, or a similar computing device. Taking the operation on a mobile terminal as an example, Figure 1 is a hardware structure block diagram of a mobile terminal for a work order quality evaluation method according to an embodiment of the present invention. As Figure 1 shown, the mobile terminal may include one or more ( Figure 1 only one is shown in Figure 1 the processor 102 (the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Among them, the above mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those of ordinary skill in the art can understand that Figure 1 the structure shown in Figure 1 is only schematic and does not limit the structure of the above mobile terminal. For example, the mobile terminal may further include more or fewer components than

[0029] shown in

[0030] In this embodiment, a work order quality evaluation method running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. And although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0031] Figure 2 It is a flowchart of a work order quality evaluation method according to an embodiment of the present application. As Figure 2 shown, the method includes the following steps:

[0032] Step S201, the first step, obtain multiple work orders with manual scores, and obtain the standard scores of each of the above work orders according to the scoring criteria. Calculate the work order scores of each of the above work orders based on the above standard scores and the above manual scores. The above scoring criteria are to evaluate the integrity, readability and refinement of the above work orders;

[0033] Specifically, obtain multiple work order samples that have been manually scored. According to the formulated scoring criteria, quantitatively evaluate the integrity, readability and refinement of these work orders to obtain the standard score of each work order. Then, combine the obtained standard scores with the manual scores to comprehensively score each work order, so as to obtain the work order score reflecting the work order quality. This process realizes the integration of subjective manual scores and objective standardized scoring criteria. By quantifying the integrity, readability and refinement of the work order, a more comprehensive and accurate work order quality evaluation result can be obtained.

[0034] Step S202, the second step, extract feature vectors from the text content of each of the above work orders to obtain the spatial feature vectors of each of the above work orders, and use the above spatial feature vectors and the above work order scores of each of the above work orders to train the SVR model to obtain a work order score model;

[0035] Specifically, extract feature vectors from the text content of each work order and convert it into a series of feature vectors that can capture the essential attributes of the work order. Then, we use these spatial feature vectors of the work orders together with the work order scores calculated before as training data and input them into the support vector regression (SVR) model for learning. The model fits the relationship between the work order features and the scores, and finally constructs a work order score model.

[0036] Step S203, the third step, input all the above spatial feature vectors into the above work order score model to obtain the model scores of each of the above work orders, and adjust the above scoring criteria according to the above model scores and the above manual scores of each of the above work orders;

[0037] Specifically, input all the previously extracted work order space feature vectors into the already trained work order scoring model. Based on the mapping relationship between the work order features it has learned and the scores, the model generates a model score for each work order, and adjusts the scoring criteria according to the model scores of these work orders and the corresponding manual scores to obtain different scoring criteria, preparing for obtaining multiple work order scoring models in the future.

[0038] Step S204: Repeat the steps. Repeat the above first step, the above second step, and the above third step at least once in sequence to obtain multiple above-mentioned work order scoring models.

[0039] Specifically, sequentially repeat the operations of the above first step, second step, and third step to ensure that the entire process is iterated at least once or more, that is, collect different work order samples again, obtain the work order scores of each work order according to different scoring criteria and the corresponding manual scores, and finally generate multiple new work order scoring models.

[0040] Step S205: Calculate the average value of the scoring errors between the model scores of the multiple above-mentioned work orders output by each above-mentioned work order scoring model and the corresponding manual scores to obtain multiple average scoring errors. Determine the above-mentioned work order scoring model with the smallest above-mentioned average scoring error as the final work order scoring model, and the above-mentioned average scoring errors correspond one-to-one with the above-mentioned work order scoring models.

[0041] Specifically, we quantify the gap between the work order model scores output by each different work order scoring model and the corresponding manual scores, calculate the scoring error of each work order, and then take the average value of these scoring errors to obtain the average scoring error corresponding to each model. After that, among multiple work order scoring models, determine the work order scoring model with the lowest average scoring error value and use it as the final work order scoring model. This step provides an objective model performance evaluation index by quantifying the difference between the model score and the manual score, that is, the scoring error.

[0042] Step S206: Use the above final work order scoring model to score new work orders. In the case where the above model score of the above new work order is lower than a predetermined threshold, trigger an update content notification to trigger the update of the above text content of the above new work order.

[0043] Specifically, use the final work order scoring model to score newly received work orders. When the model score result of the new work order fails to reach the preset scoring threshold, the system automatically triggers a notification reminder to update the filled content. The action of triggering the update notification prompts the work order filler to promptly correct the work order content to improve its integrity, readability, and conciseness, and ensure the accuracy of the work order information content and the overall quality.

[0044] Through this embodiment, in the above-mentioned work order quality evaluation method, in the first step, multiple work orders with manual scores are obtained, and the standard scores of each of the above work orders are obtained according to the scoring criteria. The work order scores of each of the above work orders are calculated based on the standard scores and the manual scores of each of the above work orders. The scoring criteria are to evaluate the integrity, readability, and refinement of the above work orders; in the second step, feature vectors are extracted from the text content of each of the above work orders to obtain the spatial feature vectors of each of the above work orders, and the SVR model is trained using the spatial feature vectors and the work order scores of each of the above work orders to obtain a work order scoring model; in the third step, all the above spatial feature vectors are input into the work order scoring model to obtain the model scores of each of the above work orders, and the scoring criteria are adjusted according to the model scores and the manual scores of each of the above work orders; repeat the steps, repeat the above first step, the above second step, and the above third step at least once to obtain multiple above work order scoring models; calculate the average value of the scoring errors between the model scores of multiple above work orders output by each of the above work order scoring models and the corresponding manual scores to obtain multiple average scoring errors, and determine the work order scoring model with the smallest above average scoring error as the final work order scoring model. The above average scoring error corresponds to the above work order scoring model one by one; use the above final work order scoring model to score a new work order. When the model score of the above new work order is lower than a predetermined threshold, a notice to update the filled content is triggered to trigger the update of the above text content of the above new work order. In this application, the work order score of the work order is calculated based on the scoring criteria and the manual score, and the SVR model is trained using the work order score and the spatial feature vector of each work order to obtain a work order scoring model. Then, the work order scoring model is used to score all the spatial feature vectors to obtain a model score, and the scoring criteria are adjusted according to the model score and the manual score of each work order. The above method is continued to train different multiple work orders to obtain multiple work order scoring models. Finally, the average scoring error between the multiple model scores output by each work order scoring model and the manual score is calculated, and the work order scoring model corresponding to the smallest average scoring error is determined as the final scoring model to score the input work order. Through this model, the text content of the work order can be scored quickly to timely discover the deficiencies in the work order content, and solve the problem of low efficiency caused by the excessive dependence on manual scoring in the prior art for work order scoring.

[0045] In an alternative embodiment, to obtain the standard score of the work order, multiple work orders with manual scores are obtained, and the standard scores of each of the above work orders are obtained according to the scoring criteria. The above step S201 includes:

[0046] Step S2011, evaluate the integrity of the above work order based on the missing situation of the keywords in the work order to obtain an integrity score. The above keywords include the category, problem description, and transfer department.

[0047] Specifically, evaluate the integrity of each work order according to the completeness status of the keywords involved in the work order, that is, whether the work order has the preset keywords or whether the preset keywords are missing, and obtain the integrity score according to the missing situation. For example, if all the keywords are available, the full score is obtained; if some keywords are missing, points are deducted; if all are missing, no score is obtained, etc. These keywords include information such as the category of the work order, problem description, and transfer department required for subsequent processing. By systematically checking the completeness of the key information in the work order, the integrity level of the work order is effectively quantified. The specific integrity scoring criteria are formulated based on the keywords required for the corresponding work order.

[0048] In a specific embodiment, taking the extraction of keywords in the work order as an example, taking the three keywords of the category, problem description, and transfer department of the work order as an example, if the current work order has all the above keywords, it gets 100 points; if one keyword is missing, it gets 80 points; if two keywords are missing, it gets 60 points; if all the above keywords are missing, it gets 0 points.

[0049] Step S2012, use the Kincaid readability score formula to evaluate the readability of the above work order based on the number of sentences, number of words, and number of punctuation marks in the work order to obtain a readability score. The above Kincaid readability score formula is where r is the readability score, w is the number of words, s is the number of sentences, and syl is the number of punctuation marks.

[0050] Specifically, use the calculation rule of the Kincaid readability score to evaluate the readability of each work order based on the number of sentences, number of words, and number of punctuation marks in each work order, so as to obtain the specific readability score, that is, to evaluate whether each work order uses too many complex sentence structures, long words, or improper punctuation usage to judge whether the content of the work order is friendly to readers.

[0051] Step S2013, calculate the deviation ratio between the text length of the above work order and the average text length of historical work orders to evaluate the refinement of the above work order and obtain a refinement score.

[0052] Specifically, by measuring the deviation ratio between the text length of each work order and the average text length of past work orders, the refinement of the work order is carefully judged to obtain a refinement score. Specifically, the refinement scores corresponding to different deviation ratios are formulated according to the actual text content of the current work order. Through the acquisition of the refinement score, we can identify those work orders with overly long or short texts that may affect the information transmission efficiency and user experience, prompting the work order writers to optimize the text length and improve the refinement of the work order on the premise of ensuring information integrity.

[0053] In a specific embodiment, work orders with a text length within 20% above or below the average length receive 100 points, those within 20%-40% receive 80 points, those within 40%-60% receive 60 points, and those above 60% receive 40 points.

[0054] Step S2014, set the weights of the above-mentioned integrity score, the above-mentioned readability score, and the above-mentioned refinement score;

[0055] Specifically, allocate specific proportions to the integrity score, the readability score, and the refinement score respectively. This can more precisely reflect the performance of the work order on different quality indicators and avoid the one-sidedness that may be brought about by a single indicator evaluation.

[0056] Step S2015, calculate the above-mentioned standard score S of each of the above-mentioned work orders according to the above-mentioned integrity score of each work order and the weight of the above-mentioned integrity score, the above-mentioned readability score and the weight of the above-mentioned readability score, and the above-mentioned refinement score and the weight of the above-mentioned refinement score sdrad =(S comp *W comp +S read *W read +S conc *W conc ) / 2, where S comp is the above-mentioned integrity score, W comp is the weight of the above-mentioned integrity score, S read is the above-mentioned readability score, W read is the weight of the above-mentioned readability score, S conc is the above-mentioned refinement score, and W conc is the weight of the above-mentioned refinement score.

[0057] Specifically, according to the integrity score of each work order and its corresponding weight, the readability score and its weight, and the refinement score and its weight, a weighted sum is performed to obtain the standard score of each work order. This calculation process integrates the performance of the work order in the three key aspects of integrity, readability, and refinement into a comprehensive index according to the preset weight ratio.

[0058] In an alternative implementation, in order to obtain the work order score of each work order, the work order score of each work order is calculated based on the above-mentioned standard score and the above-mentioned manual score of each work order. The above step S201 further includes:

[0059] Step S2016, calculating the work order score of each work order based on the above-mentioned standard score and the above-mentioned manual score of each work order where n is the number of manual scores of the above work order, and S i is the i-th manual score of the above work order.

[0060] Specifically, the work order score of each work order is obtained by calculating the average of the standard score of each work order and the scores given by n manual scorers.

[0061] In an alternative implementation, in order to convert the work order into a vector representation, the text content of each work order is subjected to feature vector extraction to obtain the spatial feature vector of each work order. The above step S202 includes:

[0062] Step S2021, processing the text content of each work order using the word2vec algorithm to obtain the above-mentioned spatial feature vector D of each work order new(n*1) =W n*n D old(n*1) where D old(n*1) represents the above work order represented by one-hot encoding, and W n*n is calculated from word vectors, and W(i,j) is the cosine similarity between word i and word j.

[0063] Specifically, the word2vec algorithm is used to process the text information of each work order to capture the semantic level and context dependence of each work order, and obtain the spatial feature vector corresponding to each work order.

[0064] In an alternative implementation, in order to obtain a work order scoring model, the SVR model is trained using the above-mentioned spatial feature vectors and the above-mentioned work order scores of each work order to obtain a work order scoring model. The above step S202 further includes:

[0065] Step S2022, training the model function y = wx + b according to each of the above spatial feature vectors and each of the above work order scores, where x is the above spatial feature vector, w is the weight parameter, b is the bias parameter, and y is the model score of the above work order;

[0066] Specifically, the spatial feature vector of each work order, combined with its corresponding work order score, is used as a training data set to calibrate and optimize the parameters in the model function. The essence of this training process is that it continuously adjusts the parameters within the model so that the model can more accurately predict the quality score of the work order from the spatial feature vector.

[0067] Step S2023, adjusting the weight parameters and the bias parameters to minimize the error between the model score and the work order score, and determining the current weight parameters and the current bias parameters as the final weight parameters and the final bias parameters.

[0068] Specifically, the weight parameter w and the bias parameter b are adjusted with the goal of significantly reducing the deviation between the model score y and the actual work order score. If w and b in the current iteration can minimize the score error, they are regarded as the final weight parameter and the final bias parameter, that is, the hyperplane corresponding to the model function in the hyperdimensional space at this time can approximately represent all hyperdimensional point vectors in the space.

[0069] It is understandable that in order to further improve the model performance, the grid search method can be used to adjust the hyperparameters of the SVR model, including the type of kernel function (rbf, poly and sigmod), the penalty coefficient C (1, 10, 100), and the optimal hyperparameter value can be obtained through the grid search results.

[0070] In order to adjust the scoring criteria to obtain different work order scoring models, in an optional implementation, all of the above spatial feature vectors are input into the above work order scoring model to obtain a model score for each of the above work orders, and the above scoring criteria are adjusted according to the above model score and the above manual score for each of the above work orders. The above step S203 includes:

[0071] Step S2031, arranging the model scores of the work orders from large to small to obtain a first score set, and arranging the manual scores of the work orders from large to small to obtain a second score set;

[0072] Specifically, the model score of each work order is sorted and arranged in order from high to low to form an ordered first score set; at the same time, the manual score of the work order is also systematically organized and arranged in order from high to low, where the manual score of the work order refers to the average score given by multiple raters, which constitutes the second score set.

[0073] Step S2032, calculating the average score of a specific number of the model scores ranked at the top in the first score set to obtain a first average score, and calculating the average score of the specific number of the model scores ranked at the top in the second score set to obtain a second average score;

[0074] Specifically, calculate the average value of a certain number of model scores at the forefront in the first score set to obtain a first average score that comprehensively reflects the score level of the high-skill work order model. At the same time, statistically calculate the average value of the manual scores of the work orders in the same position in the second score set, so as to obtain a second average score representing the manual scores of the high-skill work orders.

[0075] Step S2033, adjust the weights of the above-mentioned integrity score, the above-mentioned readability score, and the above-mentioned refinement score according to the above-mentioned first average score and the above-mentioned second average score Among them, S model is the above-mentioned first average score, S hum is the above-mentioned second average score, W old is one of the weights of the above-mentioned integrity score, the above-mentioned readability score, and the above-mentioned refinement score, W new is the adjusted W old .

[0076] Specifically, according to the calculated first average score and second average score, adjust the weights of the integrity score, the readability score, and the refinement score, that is, adjust to obtain the scoring criteria of the standard score.

[0077] Figure 3 Fig. shows the first-round model training flow chart of a work order quality evaluation system provided according to an embodiment of the present application.

[0078] Among them, the storage module is used to store the model configuration after each round of training to avoid the overfitting problem caused by overtraining;

[0079] The work order AI analysis module is used to score the work order model obtained after training and score the work order;

[0080] The manual feedback module is used to calculate the work order score and input the work order score and the spatial feature vector as training samples into the work order AI analysis module for training;

[0081] The weight configuration module is used to adjust the scoring results finally generated by the work order AI analysis module and the manual feedback module to obtain an optimized scoring result;

[0082] The work order module is used to provide work orders for training, testing, and verification.

[0083] The first-round model training process is as Figure 3 shown:

[0084] S1: The work order module obtains multiple work orders with manual scores;

[0085] S2: The weight configuration module obtains the preset weights of the integrity score, the readability score, and the refinement score;

[0086] S3: The manual feedback module calculates the work order score based on the weights configured by the weight configuration module and obtains the spatial feature vector of the work order, and then sends it to the work order AI analysis module;

[0087] S4: The work order AI analysis module trains the input data to obtain a work order scoring model and saves this model to the storage module.

[0088] The second round of model training process is as Figure 4 shown below:

[0089] S5: Adjust the weights of the integrity score, readability score, and refinement score according to the model scores and manual scores of all spatial feature vectors in the previous round;

[0090] S6: Obtain new work order samples;

[0091] S7: Obtain the updated weights in the weight configuration module;

[0092] S8: Calculate the work order score according to the updated weights, and perform vector transformation on the new work order samples to obtain the corresponding spatial feature vectors;

[0093] S9: The work order AI analysis module trains the input new data to obtain the work order scoring model of this round and saves this model to the storage module;

[0094] S10: Determine the best configuration parameters among multiple work order scoring models, that is, determine the final work order scoring model;

[0095] S11: Use the final work order scoring model to score new work orders.

[0096] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0097] The embodiment of the present application also provides a work order quality evaluation device. It should be noted that a work order quality evaluation device in the embodiment of the present application can be used to execute the work order quality evaluation method provided in the embodiment of the present application. This device is used to implement the above-mentioned embodiments and preferred implementation manners, and those that have been described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that can achieve a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0098] The following introduces a work order quality evaluation device provided by an embodiment of the present application.

[0099] Figure 5 It is a structural block diagram of a work order quality evaluation device according to an embodiment of the present application. As Figure 5 shown, the device includes:

[0100] A calculation unit 10, configured to execute a first step of obtaining multiple work orders with manual scores, obtaining the standard scores of each of the above work orders according to a scoring standard, and calculating the work order scores of each of the above work orders based on the standard scores and the manual scores of each of the above work orders. The scoring standard is to evaluate the integrity, readability, and refinement of the above work orders;

[0101] Specifically, obtain multiple work order samples that have been manually scored, quantitatively evaluate the integrity, readability, and refinement of these work orders according to the established scoring standard, obtain the standard score of each work order, and then combine the obtained standard score with the manual score to comprehensively score each work order, so as to obtain the work order score reflecting the work order quality. This process realizes the integration of subjective manual scoring and objective standardized scoring criteria. By quantifying the integrity, readability, and refinement of the work order, a more comprehensive and accurate work order quality evaluation result can be obtained.

[0102] A training unit 20, configured to execute a second step of extracting feature vectors from the text content of each of the above work orders to obtain the spatial feature vectors of each of the above work orders, and training an SVR model using the spatial feature vectors and the work order scores of each of the above work orders to obtain a work order score model;

[0103] Specifically, extract feature vectors from the text content of each work order, convert it into a series of feature vectors that can capture the essential attributes of the work order, and then use the spatial feature vectors of these work orders and the work order scores calculated previously as training data and input them into a support vector regression (SVR) model for learning. By fitting the relationship between the work order features and scores through the model, a work order score model is finally constructed.

[0104] An adjustment unit 30, configured to execute a third step of inputting all the above spatial feature vectors into the above work order score model to obtain the model scores of each of the above work orders, and adjusting the above scoring standard according to the model scores and the manual scores of each of the above work orders;

[0105] Specifically, input all the previously extracted work order space feature vectors into the trained work order scoring model. Based on the mapping relationship between the work order features it has learned and the scores, the model generates a model score for each work order, and adjusts the scoring criteria according to the model scores of these work orders and the corresponding manual scores to obtain different scoring criteria, preparing for obtaining multiple work order scoring models subsequently.

[0106] Repeat unit 40, used to execute the repeating steps, repeating the above first step, the above second step, and the above third step at least once in sequence to obtain multiple above-mentioned work order scoring models;

[0107] Specifically, sequentially repeat the operations of the above first step, second step, and third step to ensure that the entire process is iterated at least once or more, that is, collect different work order samples again, obtain the work order scores of each work order according to different scoring criteria and the corresponding manual scores, and finally generate multiple new work order scoring models.

[0108] Determination unit 50, used to calculate the average value of the scoring errors between the model scores of multiple above-mentioned work orders output by each above-mentioned work order scoring model and the corresponding manual scores to obtain multiple average scoring errors, and determine the above work order scoring model with the smallest above average scoring error as the final work order scoring model, where the above average scoring error corresponds one-to-one with the above work order scoring model;

[0109] Specifically, we quantify the gap between the work order model score output by each different work order scoring model and the corresponding manual score, calculate the scoring error of each work order, and then take the average value of these scoring errors to obtain the average scoring error corresponding to each model. After that, among multiple work order scoring models, determine the work order scoring model with the lowest average scoring error value and use it as the final work order scoring model. This step provides an objective model performance evaluation index by quantifying the difference between the model score and the manual score, that is, the scoring error.

[0110] Control unit 60, used to score new work orders using the above final work order scoring model. When the above model score of the above new work order is lower than a predetermined threshold, trigger an update fill content notification to trigger the update of the above text content of the above new work order.

[0111] Specifically, use the final work order scoring model to score the newly received work orders. When the model score result of the new work order fails to reach the preset scoring threshold, the system automatically triggers a notification reminder to update the fill content. The action of triggering the update notification prompts the work order filler to promptly correct the work order content to improve its integrity, readability, and conciseness, ensuring the accuracy of the work order information content and the overall quality.

[0112] Through this embodiment, in the above-mentioned work order quality evaluation device, a calculation unit is configured to perform the first step of obtaining multiple work orders with manual scores, obtaining the standard scores of each of the above-mentioned work orders according to the scoring criteria, and calculating the work order scores of each of the above-mentioned work orders based on the standard scores and the manual scores of each of the above-mentioned work orders. The scoring criteria are used to evaluate the integrity, readability, and refinement of the above-mentioned work orders; a training unit is configured to perform the second step of extracting feature vectors from the text content of each of the above-mentioned work orders to obtain the spatial feature vectors of each of the above-mentioned work orders, and training an SVR model using the spatial feature vectors and the work order scores of each of the above-mentioned work orders to obtain a work order scoring model; an adjustment unit is configured to perform the third step of inputting all the above-mentioned spatial feature vectors into the above-mentioned work order scoring model to obtain the model scores of each of the above-mentioned work orders, and adjusting the above-mentioned scoring criteria according to the model scores and the manual scores of each of the above-mentioned work orders; a repetition unit is configured to perform a repetition step of sequentially repeating the above-mentioned first step, the above-mentioned second step, and the above-mentioned third step at least once to obtain multiple above-mentioned work order scoring models; a determination unit is configured to calculate the average value of the scoring errors between the model scores of multiple above-mentioned work orders output by each of the above-mentioned work order scoring models and the corresponding manual scores to obtain multiple average scoring errors, and determining the above-mentioned work order scoring model with the smallest above-mentioned average scoring error as the final work order scoring model. The above-mentioned average scoring error corresponds one-to-one with the above-mentioned work order scoring model; a control unit is configured to score a new work order using the above-mentioned final work order scoring model, and trigger an update content notification in the case where the above-mentioned model score of the above-mentioned new work order is lower than a predetermined threshold to trigger an update of the above-mentioned text content of the above-mentioned new work order. In this application, the work order score of a work order is calculated based on the scoring criteria and the manual score, and the SVR model is trained using the work order scores and spatial feature vectors of each work order to obtain a work order scoring model. Then, the work order scoring model is used to score all the spatial feature vectors to obtain model scores, and the scoring criteria are adjusted according to the model scores and manual scores of each work order. The above-mentioned method is continuously used to train different multiple work orders to obtain multiple work order scoring models. Finally, the average scoring errors between the multiple model scores output by each work order scoring model and the manual scores are calculated, and the work order scoring model corresponding to the smallest average scoring error is determined as the final scoring model to score the input work order. Through this model, the text content of the work order can be quickly scored to timely discover the deficiencies in the work order content, solving the problem of low efficiency caused by the over-reliance on manual scoring in the prior art for work order scoring.

[0113] In an alternative embodiment, to obtain the standard score of a work order, multiple work orders with manual scores are obtained, and the standard scores of each of the above-mentioned work orders are obtained according to the scoring criteria. The above-mentioned calculation unit includes:

[0114] The first calculation module is used to evaluate the integrity of the above work order based on the missing situation of the keywords in the work order, and obtain an integrity score. The keywords include the category to which it belongs, the problem description, and the transfer department;

[0115] Specifically, according to the completeness status of the keywords involved in each work order, that is, whether the preset keywords are present in the work order or whether the preset keywords are missing, its integrity is evaluated, and an integrity score is obtained according to the missing situation. For example, if all keywords are available, the full score is obtained; if some keywords are missing, points are deducted; if all are missing, no score is obtained, etc. These keywords include information such as the category to which the work order belongs, the problem description, and the transfer department required for subsequent processing. By systematically checking the completeness of the key information in the work order, the integrity level of the work order is effectively quantified. The specific integrity scoring criteria are formulated based on the keywords required for the corresponding work order.

[0116] The second calculation module is used to evaluate the readability of the above work order according to the number of sentences, the number of words, and the number of punctuation marks in the work order by using the Kincaid readability score formula, and obtain a readability score. The above Kincaid readability score formula is where r is the readability score, w is the number of words, s is the number of sentences, and syl is the number of punctuation marks;

[0117] Specifically, using the calculation rules of the Kincaid readability score, the readability of each work order is evaluated according to the number of sentences, the number of words, and the number of punctuation marks in the work order, so as to obtain a specific readability score, that is, to evaluate whether each work order uses overly complex sentence structures, lengthy words, or improper punctuation usage to judge whether the content of the work order is user-friendly to readers.

[0118] The third calculation module is used to calculate the deviation ratio between the text length of the above work order and the average text length of historical work orders, evaluate the refinement of the above work order, and obtain a refinement score;

[0119] Specifically, by measuring the deviation ratio between the text length of each work order and the average text length of past work orders, the refinement of the work order is carefully judged, so as to obtain a refinement score. The specific refinement scores corresponding to different deviation ratios are formulated according to the actual text content of the current work order. By obtaining the refinement score, we can identify those work orders with overly long or short texts that may affect the information transmission efficiency and user experience, and prompt the work order writer to optimize the text length and improve the refinement of the work order on the premise of ensuring information integrity.

[0120] The fourth calculation module is used to set the weights of the above integrity score, the above readability score, and the above refinement score;

[0121] Specifically, specific weights are assigned to the integrity score, readability score, and conciseness score respectively, which can more precisely reflect the performance of the work orders on different quality indicators and avoid the one-sidedness that may be brought about by a single indicator evaluation.

[0122] The fifth calculation module is used to calculate the standard score S of each of the above work orders based on the above integrity score and the weight of the integrity score, the above readability score and the weight of the readability score, and the above conciseness score and the weight of the conciseness score. sdrad =(S comp *W comp +S read *W read +S conc *W conc ) / 2, where S comp is the above integrity score, W comp is the weight of the above integrity score, S read is the above readability score, W read is the weight of the above readability score, S conc is the above conciseness score, and W conc is the weight of the above conciseness score.

[0123] Specifically, based on the integrity score of each work order and its corresponding weight, the readability score and its weight, and the conciseness score and its weight, a weighted sum is performed to obtain the standard score of each work order. This calculation process integrates the performance of the work order in three key aspects of integrity, readability, and conciseness into a comprehensive indicator according to the preset weight ratio.

[0124] In order to obtain the work order score of the work order, in an optional implementation manner, the work order score of each of the above work orders is calculated based on the above standard score and the above manual score of the work order. The above calculation unit further includes:

[0125] The sixth calculation module is used to calculate the work order score of each of the above work orders based on the above standard score and the above manual score of the work order. where n is the number of manual scores of the above work order, and S i is the i-th manual score of the above work order.

[0126] Specifically, by calculating the average of the standard score of each work order and the scores given by n manual scorers, the work order score of each work order is obtained.

[0127] In order to transform the work order into a vector representation, in an optional implementation manner, feature vector extraction is performed on the text content of each of the above work orders to obtain the spatial feature vector of each of the above work orders. The above training unit includes:

[0128] The first training module is used to process the text content of each of the above work orders using the word2vec algorithm to obtain the above spatial feature vector D of each of the above work orders new(n*1) = W n*n D old(n*1) , where D old(n*1) represents the above work order represented by one-hot encoding, and W n*n is calculated from word vectors, and W(i,j) is the cosine similarity between word i and word j.

[0129] Specifically, the word2vec algorithm is used to process the text information of each work order to capture the semantic level and context dependency relationship of each work order, and obtain the spatial feature vector corresponding to each work order.

[0130] In order to obtain a work order scoring model, in an optional implementation, the above spatial feature vectors of each of the above work orders and the above work order scores are used to train an SVR model to obtain a work order scoring model. The above training unit further includes:[[]]

[0131] The second training module is used to train the model function y = wx + b according to each of the above spatial feature vectors and each of the above work order scores, where x is the above spatial feature vector, w is the weight parameter, b is the bias parameter, and y is the model score of the above work order;

[0132] Specifically, the spatial feature vector of each work order, combined with its corresponding work order score, is used as a training data set to correct and optimize the parameters in the model function. The essence of this training process is that it continuously adjusts the parameters inside the model so that the model can more accurately predict the quality score of the work order from the spatial feature vector.

[0133] The third training module is used to adjust the above weight parameter and the above bias parameter to minimize the error between the above model score and the above work order score, and determine the current weight parameter and the current bias parameter as the final weight parameter and the final bias parameter.

[0134] Specifically, the weight parameter w and the bias parameter b are adjusted. The goal is to significantly reduce the deviation between the model score y and the actual work order score. If w and b in the current iteration can minimize the scoring error, they are regarded as the final weight parameter and the final bias parameter, that is, the hyperplane corresponding to the model function in the hyperdimensional space can approximately represent all the point vectors in the space.

[0135] In order to adjust the scoring criteria to obtain different work order scoring models, in an optional implementation, all of the above spatial feature vectors are input into the above work order scoring model to obtain a model score for each of the above work orders, and the above scoring criteria are adjusted according to the above model score and the above manual score for each of the above work orders, and the above adjustment unit includes:

[0136] A first adjustment module is used to arrange the model scores of the work orders from large to small to obtain a first score set, and to arrange the manual scores of the work orders from large to small to obtain a second score set;

[0137] Specifically, the model score of each work order is sorted and arranged in order from high to low to form an ordered first score set; at the same time, the manual score of the work order is also systematically organized and arranged in order from high to low, where the manual score of the work order refers to the average score given by multiple raters, which constitutes the second score set.

[0138] The second adjustment module is used to calculate the average score of a specific number of the model scores ranked at the top in the first score set to obtain a first average score, and calculate the average score of the specific number of the model scores ranked at the top in the second score set to obtain a second average score;

[0139] Specifically, the average value of a certain number of model scores in the top column of the first scoring set is calculated to obtain a first average score that comprehensively reflects the model score level of high-scoring work orders. At the same time, the average value of the manual scores of work orders in the same position in the second scoring set is calculated to obtain a second average score representing the manual scores of high-scoring work orders.

[0140] A third adjustment module is used to adjust the weight of the completeness score, the weight of the readability score, and the weight of the conciseness score according to the first average score and the second average score. Among them, S model is the first average score mentioned above, S hum is the second average score mentioned above, W old is one of the weight of the completeness score, the weight of the readability score, and the weight of the conciseness score, W new is the adjusted W old .

[0141] Specifically, according to the calculated first average score and second average score, the weight of the completeness score, the weight of the readability score and the weight of the conciseness score are adjusted, that is, the scoring criteria for the standard score are adjusted.

[0142] The above-mentioned work order quality evaluation device includes a processor and a memory. The above-mentioned calculation unit, training unit, adjustment unit, repetition unit, determination unit, control unit, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to implement corresponding functions. The above-mentioned modules are all located in the same processor; alternatively, the above-mentioned each module is located in different processors in any combined form.

[0143] The processor contains a kernel, and the kernel retrieves the corresponding program units from the memory. One or more kernels can be set, and the efficiency of scoring work orders can be improved by adjusting the kernel parameters.

[0144] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or forms such as non-volatile memory, such as read-only memory (ROM) or flash memory (flash RAM), and the memory includes at least one memory chip.

[0145] An embodiment of the present invention provides a computer-readable storage medium. The above-mentioned computer-readable storage medium includes a stored program, wherein when the above-mentioned program runs, it controls the device where the above-mentioned computer-readable storage medium is located to execute the above-mentioned work order quality evaluation method.

[0146] Specifically, a work order quality evaluation method includes:

[0147] Step S201, the first step, obtain multiple work orders with manual scores, and obtain the standard scores of each of the above work orders according to the scoring criteria. Calculate the work order scores of each of the above work orders based on the above standard scores and the above manual scores. The above scoring criteria are to evaluate the integrity, readability, and refinement of the above work orders;

[0148] Step S202, the second step, extract feature vectors from the text content of each of the above work orders to obtain the spatial feature vectors of each of the above work orders, and use the above spatial feature vectors and the above work order scores of each of the above work orders to train the SVR model to obtain a work order scoring model;

[0149] Step S203, the third step, input all the above spatial feature vectors into the above work order scoring model to obtain the model scores of each of the above work orders, and adjust the above scoring criteria according to the above model scores and the above manual scores of each of the above work orders;

[0150] Step S204, the repetition step, repeat the above first step, the above second step, and the above third step at least once in sequence to obtain multiple above work order scoring models;

[0151] Step S205, calculate the average of the scoring errors between the model scores of the multiple work orders output by each of the above work order scoring models and the corresponding manual scores, obtaining multiple average scoring errors. Determine the work order scoring model with the smallest of the above average scoring errors as the final work order scoring model, and the above average scoring errors correspond one-to-one with the above work order scoring models;

[0152] Step S206, use the above final work order scoring model to score a new work order. In the case where the model score of the above new work order is lower than a predetermined threshold, trigger a notification to update the filled content to trigger an update of the above text content of the above new work order.

[0153] An embodiment of the present invention provides a processor, and the above processor is used to run a program. Among them, when the above program runs, it executes the above method for evaluating the quality of work orders.

[0154] Specifically, a method for evaluating the quality of work orders includes:

[0155] Step S201, the first step, obtain multiple work orders with manual scores, and obtain the standard scores of each of the above work orders according to the scoring criteria. Calculate the work order scores of each of the above work orders based on the above standard scores and the above manual scores. The above scoring criteria are to evaluate the integrity, readability, and refinement of the above work orders;

[0156] Step S202, the second step, extract feature vectors from the text content of each of the above work orders to obtain the spatial feature vectors of each of the above work orders, and use the above spatial feature vectors and the above work order scores of each of the above work orders to train an SVR model to obtain a work order scoring model;

[0157] Step S203, the third step, input all the above spatial feature vectors into the above work order scoring model to obtain the model scores of each of the above work orders, and adjust the above scoring criteria according to the above model scores and the above manual scores of each of the above work orders;

[0158] Step S204, repeat steps, repeat the above first step, the above second step, and the above third step at least once in sequence to obtain multiple above work order scoring models;

[0159] Step S205, calculate the average of the scoring errors between the model scores of the multiple work orders output by each of the above work order scoring models and the corresponding manual scores, obtaining multiple average scoring errors. Determine the work order scoring model with the smallest of the above average scoring errors as the final work order scoring model, and the above average scoring errors correspond one-to-one with the above work order scoring models;

[0160] Step S206, use the above-mentioned final work order scoring model to score the new work order. If the model score of the new work order is lower than the predetermined threshold, trigger a notification for updating the filled content to trigger the update of the text content of the new work order.

[0161] The present application also provides a computer program product, which, when executed on a data processing device, is adapted to execute a program initialized with at least the following method steps:

[0162] Step S201, the first step, obtain multiple work orders with manual scores, and obtain the standard scores of each work order according to the scoring criteria. Calculate the work order scores of each work order based on the standard scores and the manual scores of each work order. The scoring criteria are to evaluate the integrity, readability, and refinement of the work order.

[0163] Step S202, the second step, extract feature vectors from the text content of each work order to obtain the spatial feature vectors of each work order, and use the spatial feature vectors and the work order scores of each work order to train the SVR model to obtain a work order scoring model.

[0164] Step S203, the third step, input all the above-mentioned spatial feature vectors into the work order scoring model to obtain the model scores of each work order, and adjust the scoring criteria according to the model scores and the manual scores of each work order.

[0165] Step S204, repeat steps, repeat the above-mentioned first step, the second step, and the third step at least once in sequence to obtain multiple work order scoring models.

[0166] Step S205, calculate the average value of the scoring errors between the model scores of multiple work orders output by each work order scoring model and the corresponding manual scores to obtain multiple average scoring errors, and determine the work order scoring model with the smallest average scoring error as the final work order scoring model. The average scoring errors correspond to the work order scoring models one by one.

[0167] Step S206, use the above-mentioned final work order scoring model to score the new work order. If the model score of the new work order is lower than the predetermined threshold, trigger a notification for updating the filled content to trigger the update of the text content of the new work order.

[0168] The embodiment of the present application also provides a work order quality evaluation system, including: one or more processors, a memory, and one or more programs, where the one or more programs are stored in the memory and are configured to be executed by the one or more processors, including executing any one of the above-mentioned methods in the work order quality evaluation method.

[0169] Specifically, a work order quality evaluation method includes:

[0170] Step S201, the first step, obtain multiple work orders with manual scores, and obtain the standard scores of each of the above work orders according to the scoring criteria. Calculate the work order scores of each of the above work orders based on the standard scores and the manual scores of each of the above work orders. The above scoring criteria are for evaluating the integrity, readability, and refinement of the above work orders;

[0171] Step S202, the second step, extract feature vectors from the text content of each of the above work orders to obtain the spatial feature vectors of each of the above work orders, and use the spatial feature vectors and the work order scores of each of the above work orders to train the SVR model to obtain a work order scoring model;

[0172] Step S203, the third step, input all the above spatial feature vectors into the above work order scoring model to obtain the model scores of each of the above work orders, and adjust the above scoring criteria according to the model scores and the manual scores of each of the above work orders;

[0173] Step S204, repeat steps, repeat the above first step, the above second step, and the above third step at least once in sequence to obtain multiple above work order scoring models;

[0174] Step S205, calculate the average value of the scoring errors between the model scores of multiple above work orders output by each of the above work order scoring models and the corresponding manual scores to obtain multiple average scoring errors, and determine the above work order scoring model with the smallest above average scoring error as the final work order scoring model. The above average scoring errors correspond one-to-one with the above work order scoring models;

[0175] Step S206, use the above final work order scoring model to score a new work order. When the model score of the above new work order is lower than a predetermined threshold, trigger a notification to update the filled content to trigger the update of the above text content of the above new work order.

[0176] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general-purpose computing device. They can be concentrated on a single computing device or distributed on a network composed of multiple computing devices. They can be implemented by program codes executable by the computing device. Thus, they can be stored in a storage device and executed by the computing device. And in some cases, the steps shown or described can be executed in a different order than here, or they can be separately made into individual integrated circuit modules, or multiple of them can be made into a single integrated circuit module to implement. In this way, the present invention is not limited to any specific combination of hardware and software.

[0177] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0178] The present application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one or more of the processes or Figure 1 blocks or combinations of blocks.

[0179] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in Figure 1 one or more of the processes or Figure 1 blocks or combinations of blocks.

[0180] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one or more of the processes or Figure 1 blocks or combinations of blocks.

[0181] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and memory.

[0182] The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM), and / or non-volatile memory such as read-only memory (ROM) or flash memory (flash RAM). The memory is an example of computer-readable media.

[0183] A computer-readable medium includes permanent and non-permanent, removable and non-removable media and can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, magnetic disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media do not include transitory computer-readable media, such as modulated data signals and carrier waves.

[0184] It should also be noted that the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, commodity or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, commodity or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the existence of additional identical elements in the process, method, commodity or device comprising the element.

[0185] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0186] 1), A work order quality evaluation method of the present application calculates the work order score of a work order through a scoring standard and manual scoring, and uses the work order scores and spatial feature vectors of each work order to train an SVR model to obtain a work order scoring model. Then, the work order scoring model is used to score all spatial feature vectors to obtain a model score, and the scoring standard is adjusted according to the model scores and manual scores of each work order. The above method is continued to train different multiple work orders to obtain multiple work order scoring models. Finally, the average scoring error between the multiple model scores output by each work order scoring model and the manual score is calculated, and the work order scoring model corresponding to the smallest average scoring error is determined as the final scoring model to score the input work order. Through this model, the text content of the work order can be quickly scored to timely discover the deficiencies in the work order content, solving the problem of low efficiency caused by over-reliance on manual scoring in the prior art for work order scoring.

[0187] 2) A work order quality evaluation device of the present application calculates the work order score of a work order through a scoring standard and manual scoring, and uses the work order score and spatial feature vector of each work order to train an SVR model to obtain a work order scoring model. Then, the work order scoring model is used to score all spatial feature vectors to obtain a model score, and the scoring standard is adjusted according to the model score and manual score of each work order. The above method is continued to train multiple different work orders to obtain multiple work order scoring models. Finally, the average scoring error between the multiple model scores output by each work order scoring model and the manual score is calculated, and the work order scoring model corresponding to the smallest average scoring error is determined as the final scoring model to score the input work order. Through this model, the text content of the work order can be quickly scored to timely discover the deficiencies in the work order content, solving the problem of low efficiency caused by the over-reliance on manual scoring in the existing work order scoring technology.

[0188] The above are only the preferred embodiments of the present application and are not used to limit the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the protection scope of the present application.

Claims

1. A work order quality evaluation method, characterized in that: include: The first step is to obtain a plurality of work orders with manual scores, and obtain a standard score for each of the work orders according to a scoring standard, and obtain a work order score for each of the work orders according to the standard score and the manual score of each of the work orders, wherein the scoring standard is to evaluate the completeness, readability and refinement of the work order; The second step is to extract feature vectors from the text content of each work order to obtain a spatial feature vector of each work order, and train an SVR model using the spatial feature vector of each work order and the work order score to obtain a work order scoring model; The third step is to input all the spatial feature vectors into the work order scoring model to obtain a model score for each work order, and adjust the scoring criteria according to the model score and the manual score for each work order; Repeating step, repeating the first step, the second step and the third step at least once in sequence, to obtain a plurality of the work order scoring models; Calculate the average of the scoring errors between the model scores of the multiple work orders output by each of the work order scoring models and the corresponding manual scores to obtain multiple average scoring errors, and determine the work order scoring model with the smallest average scoring error as the final work order scoring model, wherein the average scoring error corresponds to the work order scoring model one by one; The final work order scoring model is used to score the new work order, and when the model score of the new work order is lower than a predetermined threshold, a notification of updating the filled content is triggered to trigger the update of the text content of the new work order.

2. The method according to claim 1, characterized in that A plurality of work orders with manual ratings are obtained, and a standard rating of each of the work orders is obtained according to the rating criteria, including: The integrity of the work order is evaluated according to the missing of keywords of the work order to obtain a completeness score, wherein the keywords include the category, the problem description and the transfer department; The readability of the work order is evaluated according to the number of sentences, words and punctuation marks in the work order using the Kincaid readability score formula to obtain a readability score. The Kincaid readability score formula is: Among them, r is the readability score, w is the number of words, s is the number of sentences, and syl is the number of punctuation marks; Calculating the deviation ratio between the text length of the work order and the average text length of historical work orders, evaluating the conciseness of the work order, and obtaining a conciseness score; Setting the weight of the completeness score, the weight of the readability score, and the weight of the conciseness score; The standard score S of each work order is calculated according to the completeness score and the weight of the completeness score, the readability score and the weight of the readability score, and the conciseness score and the weight of the conciseness score of each work order. sdrad =(S comp *W comp +S read *W read +S conc *W conc ) / 2, where S comp For the completeness score, W comp is the weight of the integrity score, S read Score the readability, W read The weight of the readability score, S conc The refinement score is W conc A weight for the refinement score.

3. The method according to claim 2, characterized in that The work order score of each work order is calculated according to the standard score and the manual score of each work order, including: The work order score of each work order is calculated based on the standard score of the work order and the manual score Where n is the number of manual ratings for the work order, S i Give the i-th manual score for the work order.

4. The method according to claim 1, characterized in that Extracting feature vectors from the text content of each work order to obtain spatial feature vectors of each work order includes: The word2vec algorithm is used to process the text content of each work order to obtain the spatial feature vector D of each work order. new(n*1) =W n*n D old(n*1) , where D old(n*1) represents the work order represented by one-hot encoding, W n*n Calculated from the word vector, W(i,j) is the cosine similarity between word i and word j.

5. The method according to claim 1, characterized in that The SVR model is trained using the spatial feature vectors and the work order scores of each work order to obtain a work order scoring model, including: Training a model function y=wx+b according to each of the spatial feature vectors and each of the work order scores, wherein x is the spatial feature vector, w is a weight parameter, b is a bias parameter, and y is the model score of the work order; The weight parameter and the bias parameter are adjusted to minimize the error between the model score and the work order score, and the current weight parameter and the current bias parameter are determined as the final weight parameter and the final bias parameter.

6. The method according to claim 2, characterized in that Inputting all of the spatial feature vectors into the work order scoring model to obtain a model score for each of the work orders, and adjusting the scoring criteria according to the model score and the manual score for each of the work orders, including: Arrange the model scores of the work orders from large to small to obtain a first score set, and arrange the manual scores of the work orders from large to small to obtain a second score set; Calculate the average score of a specific number of model scores ranked high in the first score set to obtain a first average score, and calculate the average score of the specific number of model scores ranked high in the second score set to obtain a second average score; The weight of the completeness score, the weight of the readability score, and the weight of the conciseness score are adjusted according to the first average score and the second average score. Among them, S model is the first average score, S hum is the second average score, W old is one of the weight of the completeness score, the weight of the readability score, and the weight of the conciseness score, W new is the adjusted W old .

7. A work order quality evaluation device, characterized in that: include: A calculation unit, configured to execute the first step, obtain a plurality of work orders with manual scores, and obtain a standard score for each of the work orders according to a scoring standard, and obtain a work order score for each of the work orders according to the standard score and the manual score of each of the work orders, wherein the scoring standard is to evaluate the completeness, readability, and refinement of the work order; A training unit, used to execute the second step, extract feature vectors from the text content of each work order to obtain a spatial feature vector of each work order, and train an SVR model using the spatial feature vectors of each work order and the work order score to obtain a work order scoring model; an adjustment unit, configured to execute the third step, input all of the spatial feature vectors into the work order scoring model, obtain a model score for each of the work orders, and adjust the scoring criteria according to the model score and the manual score for each of the work orders; A repeating unit, used for executing the repeating step, repeating the first step, the second step and the third step in sequence at least once, to obtain a plurality of the work order scoring models; A determination unit, configured to calculate an average of the scoring errors between the model scores of the plurality of work orders output by each of the work order scoring models and the corresponding manual scores, to obtain a plurality of average scoring errors, and to determine the work order scoring model with the smallest average scoring error as the final work order scoring model, wherein the average scoring error corresponds to the work order scoring model in a one-to-one manner; A control unit is used to score a new work order using the final work order scoring model, and trigger an update content notification when the model score of the new work order is lower than a predetermined threshold, so as to trigger an update of the text content of the new work order.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 6.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A work order quality evaluation system, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of claims 1 to 6.

Images