Work order evaluation method, electronic device, and program product

Abstract

The method comprises the following steps: obtaining structured information and unstructured information of a work order; extracting a structured feature vector of each type of data information in the structured information; extracting an unstructured feature vector of the unstructured information; taking the structured feature vector as an input of an attention mechanism layer in an evaluation model, obtaining an attention weighted feature vector based on attention weights of each structured feature vector in the attention mechanism layer, and training the attention weights of each structured feature vector in the attention mechanism layer through the unstructured feature vector; and obtaining a credibility evaluation result of the work order based on the attention weighted feature vector and the unstructured feature vector through a nonlinear transformation layer in the evaluation model. The technical solution can make the evaluation result of the work order more accurate.

Description

Technical Field

[0001] This disclosure relates to the field of user feedback processing technology, specifically to a work order evaluation method, electronic device, and program product. Background Technology

[0002] With the development of technology, there are more and more applications on terminals. Users often encounter problems while using these applications. At this time, users can submit their problems, forming user feedback tickets. The server can evaluate the credibility of these user feedback tickets and select some tickets with high credibility for automated processing based on the evaluation results. This improves the ticket processing speed, optimizes the user experience, and reduces labor costs.

[0003] Generally, user feedback tickets include structured information such as user identity information and unstructured information such as error text submitted by the user. Current credibility assessment schemes for user feedback tickets directly concatenate these two parts of data together for evaluation, without considering the coupling characteristics between the two parts of data and ignoring the relationship between unstructured and structured information. Therefore, the credibility assessment results obtained will be inaccurate. Summary of the Invention

[0004] This disclosure provides a work order evaluation method, electronic device, and program product.

[0005] Firstly, this disclosure provides a work order evaluation method.

[0006] Specifically, the work order evaluation method includes:

[0007] Obtain structured and unstructured information from work orders;

[0008] The structured feature vectors of various types of data information in the structured information are extracted by the structured feature extraction layer in the evaluation model;

[0009] The unstructured feature vector of the unstructured information is extracted through the unstructured feature extraction layer in the evaluation model;

[0010] The structured feature vectors are used as input to the attention mechanism layer in the evaluation model. An attention-weighted feature vector is obtained based on the attention weights of each structured feature vector in the attention mechanism layer. The attention mechanism layer is trained by evaluating the attention weights of each structured feature vector using unstructured feature vectors.

[0011] Based on the attention-weighted feature vector and the unstructured feature vector, the credibility evaluation result of the work order is obtained through the nonlinear transformation layer in the evaluation model.

[0012] In conjunction with the first aspect, in a first implementation of the first aspect of this disclosure, the input to the structured feature extraction layer is graph data, and the method further includes:

[0013] The various types of data information in the structured information are converted into graph data, where the vertices of the graph are the data information of each data type in the structured information, and the edges of the graph are the relationships between the data information of each data type.

[0014] In conjunction with the first aspect and the first implementation of the first aspect, in the second implementation of the first aspect of this disclosure, wherein the unstructured information includes error text information, the method further includes:

[0015] The error text information is converted into an initial vector using a word vector algorithm;

[0016] The step of extracting unstructured feature vectors of the unstructured information through the unstructured feature extraction layer in the evaluation model includes:

[0017] The unstructured feature vector is obtained by extracting features from the initial vector through the unstructured feature extraction layer composed of convolutional and pooling layers in the evaluation model.

[0018] In conjunction with the first aspect and the above-described implementations of the first aspect, in a third implementation of the first aspect, the method further includes:

[0019] Obtain sample data, which includes structured information, unstructured information, and credibility of sample work orders;

[0020] The initial evaluation model is trained based on the sample data, and the parameters in the initial evaluation model are adjusted until the accuracy of the evaluation result reaches a preset threshold to obtain the evaluation model. The evaluation model includes a structured feature extraction layer, an unstructured feature extraction layer, an attention mechanism layer, and a nonlinear transformation layer.

[0021] In conjunction with the first aspect and the above-described implementations of the first aspect, in the fourth implementation of the first aspect of this disclosure, the step of obtaining the credibility assessment result of the work order through the nonlinear transformation layer in the evaluation model based on the attention-weighted feature vector and the unstructured feature vector includes:

[0022] Based on the historical error acceptance rate of the reporting object corresponding to the work order, the unstructured feature vector is corrected to obtain the unstructured corrected feature vector of the work order.

[0023] Based on the attention-weighted feature vector and the unstructured modified feature vector, the credibility evaluation result of the work order is obtained through the nonlinear transformation layer in the evaluation model.

[0024] In conjunction with the first aspect and the above-described implementations of the first aspect, in the fifth implementation of the first aspect of this disclosure, the nonlinear transformation layer includes a sigmoid layer or a softmax layer.

[0025] In conjunction with the first aspect and the above-mentioned implementation methods of the first aspect, in the sixth implementation method of the first aspect of this disclosure, the various types of data information in the structured information include reporting object data information, work order data information, reporting object historical operation information and error entity information.

[0026] Secondly, this disclosure provides a work order evaluation device.

[0027] Specifically, the work order evaluation device includes:

[0028] The first acquisition module is configured to acquire structured and unstructured information of work orders;

[0029] The first extraction module is configured to extract structured feature vectors of various types of data information in the structured information through the structured feature extraction layer in the evaluation model;

[0030] The second extraction module is configured to extract the unstructured feature vector of the unstructured information through the unstructured feature extraction layer in the evaluation model;

[0031] The attention weighting module is configured to take the structured feature vector as input to the attention mechanism layer in the evaluation model, and obtain the attention weighted feature vector based on the attention weight of each structured feature vector in the attention mechanism layer. The attention mechanism layer is trained by evaluating the attention weight of each structured feature vector through unstructured feature vectors.

[0032] The transformation module is configured to obtain the credibility evaluation result of the work order through the nonlinear transformation layer in the evaluation model based on the attention-weighted feature vector and the unstructured feature vector.

[0033] In conjunction with the second aspect, in a first implementation of the second aspect of this disclosure, wherein the input to the structured feature extraction layer is graph data, the apparatus further includes:

[0034] The conversion module is configured to convert various types of data information in the structured information into graph data, wherein the vertices of the graph are the data information of various data types in the structured information, and the edges of the graph are the relationships between the data information of various data types.

[0035] In conjunction with the second aspect and the first implementation of the second aspect, in the second implementation of the present disclosure, the unstructured information includes error text information, and the method further includes:

[0036] The error text information is converted into an initial vector using a word vector algorithm;

[0037] The second extraction module is configured as follows:

[0038] The unstructured feature vector is obtained by extracting features from the initial vector through the unstructured feature extraction layer composed of convolutional and pooling layers in the evaluation model.

[0039] In conjunction with the second aspect and the above-described implementations of the second aspect, in a third implementation of the second aspect of this disclosure, the apparatus further includes:

[0040] The second acquisition module is configured to acquire sample data, which includes structured information, unstructured information and credibility of sample work orders.

[0041] The training module is configured to train an initial evaluation model based on the sample data, adjust the parameters in the initial evaluation model until the accuracy of the evaluation result reaches a preset threshold, and obtain an evaluation model. The evaluation model includes a structured feature extraction layer, an unstructured feature extraction layer, an attention mechanism layer, and a nonlinear transformation layer.

[0042] In conjunction with the second aspect and the above-described implementations of the second aspect, in the fourth implementation of the second aspect of this disclosure, the transformation module is configured as follows:

[0043] Based on the historical error acceptance rate of the reporting object corresponding to the work order, the unstructured feature vector is corrected to obtain the unstructured corrected feature vector of the work order.

[0044] Based on the attention-weighted feature vector and the unstructured modified feature vector, the credibility evaluation result of the work order is obtained through the nonlinear transformation layer in the evaluation model.

[0045] Thirdly, embodiments of this disclosure provide an electronic device, including a memory and a processor. The memory stores one or more computer instructions that support a work order evaluation device in executing the aforementioned work order evaluation method. The processor is configured to execute the computer instructions stored in the memory. The work order evaluation device may also include a communication interface for communicating with other devices or communication networks.

[0046] Fourthly, embodiments of this disclosure provide a computer-readable storage medium for storing computer instructions used by a work order evaluation device, which includes computer instructions for executing the aforementioned work order evaluation method for the work order evaluation device.

[0047] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program / instructions, wherein the computer program / instructions, when executed by a processor, implement the steps in the above-described work order evaluation method.

[0048] Sixthly, embodiments of this disclosure provide a navigation method, wherein a navigation route calculated based at least on a starting point, an ending point, a work order's credibility assessment result, and road conditions is obtained, and navigation guidance is performed based on the navigation route, wherein the work order's credibility assessment result is implemented based on any of the methods described above.

[0049] The technical solutions provided in this disclosure may have the following beneficial effects:

[0050] The above technical solution uses an attention mechanism to evaluate the importance of each structured feature in the structured information based on the unstructured information in the work order. In this way, the features of the two types of information are better integrated to evaluate the credibility of the work order. Different structured features can be assigned different importance at the work order level, avoiding the output of the same importance for different structured features of the work order. It also takes into account the impact of unstructured information on structured information, making the evaluation results of the work order more accurate and closer to the service reality.

[0051] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the embodiments of this disclosure. Attached Figure Description

[0052] Other features, objects, and advantages of embodiments of this disclosure will become more apparent from the following detailed description of non-limiting implementations, taken in conjunction with the accompanying drawings. In the drawings:

[0053] Figure 1 A flowchart illustrating a work order evaluation method according to an embodiment of the present disclosure is shown;

[0054] Figure 2 A structural block diagram of a work order evaluation device according to an embodiment of the present disclosure is shown;

[0055] Figure 3 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown;

[0056] Figure 4 This is a schematic diagram of the structure of a computer system suitable for implementing a work order evaluation method according to an embodiment of the present disclosure. Detailed Implementation

[0057] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings to enable those skilled in the art to readily implement them. Furthermore, for clarity, portions unrelated to the description of the exemplary embodiments have been omitted from the drawings.

[0058] In embodiments disclosed herein, it should be understood that terms such as “comprising” or “having” are intended to indicate the presence of features, numbers, steps, behaviors, components, portions or combinations thereof disclosed herein, and are not intended to exclude the possibility that one or more other features, numbers, steps, behaviors, components, portions or combinations thereof are present or added.

[0059] It should also be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings and examples.

[0060] In this disclosure, the acquisition of user information or user data is done with the user's authorization or confirmation, or by the user's active choice.

[0061] The technical solution provided in this disclosure adopts an attention mechanism to evaluate the importance of each structured feature in the structured information based on the unstructured information in the work order. In this way, the features of the two types of information are better integrated to evaluate the credibility of the work order. Different structured features can be given different importance at the work order level, avoiding the output of the same importance for different structured features of the work order. It also takes into account the impact of unstructured information on structured information, making the evaluation results of the work order more accurate and closer to the service reality.

[0062] Figure 1 A flowchart of a work order evaluation method according to an embodiment of the present disclosure is shown, such as... Figure 1 As shown, the work order evaluation method includes the following steps S101-S105:

[0063] In step S101, the structured and unstructured information of the work order is obtained;

[0064] In step S102, the structured feature vectors of various types of data information in the structured information are extracted by the structured feature extraction layer in the evaluation model;

[0065] In step S103, the unstructured feature vector of the unstructured information is extracted through the unstructured feature extraction layer in the evaluation model;

[0066] In step S104, the structured feature vector is used as the input to the attention mechanism layer in the evaluation model. An attention-weighted feature vector is obtained based on the attention weights of each structured feature vector in the attention mechanism layer. The attention mechanism layer is trained by evaluating the attention weights of each structured feature vector through unstructured feature vectors.

[0067] In step S105, based on the attention-weighted feature vector and the unstructured feature vector, the credibility evaluation result of the work order is obtained through the nonlinear transformation layer in the evaluation model.

[0068] As mentioned above, with the development of technology, there are more and more applications on terminals. Users often encounter problems while using these applications, and in such cases, they can report these problems, forming user feedback tickets. The server can evaluate the credibility of these user feedback tickets and select those with high credibility for automated processing based on the evaluation results. This improves ticket processing speed, optimizes user experience, and reduces manual costs. Generally, user feedback tickets include structured information such as identity information and unstructured information such as the error text submitted by the user. Current credibility evaluation schemes for user feedback tickets directly concatenate these two parts of data for evaluation, without considering the coupling characteristics between the two parts and ignoring the connection between unstructured and structured information. Therefore, the resulting credibility evaluation results are inaccurate.

[0069] In view of the above problems, this embodiment proposes a work order evaluation method. This method uses an attention mechanism to evaluate the importance of each structured feature in the structured information based on the unstructured information in the work order. In this way, the features of the two types of information are better integrated to evaluate the credibility of the work order. Different importance can be assigned to different structured features at the work order level, avoiding the output of the same importance for different structured features of the work order. Moreover, the influence of unstructured information on structured information is taken into account, making the evaluation results of the work order more accurate and closer to reality.

[0070] In one embodiment of this disclosure, the work order evaluation method can be applied to computers, computing devices, electronic devices, servers, service clusters, etc., that can perform work order evaluations.

[0071] In one embodiment of this disclosure, a work order refers to a work order from a reporting entity, such as a user, regarding various issues reported by the application client, and can be called a user feedback work order. This work order can be a feedback work order reported by the reporting entity when encountering a problem while using the application client, or it can be a feedback work order submitted on the application client after the reporting entity proactively discovers a problem.

[0072] In one embodiment of this disclosure, the structured information refers to quantitative data information, which is information that can be represented by data or a unified structure, such as the identity information and submission time of the reporting object who submitted the work order. The structured information in this embodiment can be divided into multiple types of data information, including reporting object data information, work order data information, reporting object historical operation information, and error entity information. The reported object data information can include the reported object's identity information, the duration of the reported object's use of the application client, and other reported object profile information. The work order data information includes the work order submission time, the classification of the entity involved in the work order, such as the classification of the problem involved in the work order for a navigation application client as the city where the work order is located, or the classification of the problem involved in the work order for a short video application client as the category of the short video involved in the work order, and so on. The reported object's historical operation information refers to the various operation records of the reported object on the application client for which the work order is targeted within a historical period. The error entity information refers to the relevant information of the problem entity reflected in the work order. For example, if a navigation application client has three types of feedback work orders: POI, public transport, and road, and this work order is a road feedback work order for a navigation application client, the error entity is the road, and the error entity information is the relevant information of the road, such as the road ID, road location, etc.

[0073] In one embodiment of this disclosure, the unstructured information is data information with variable fields, including all information other than structured information, and can be text information or non-text information. For example, when the application client targeted by the work order is a navigation client, the unstructured information of the work order can be the error text information submitted by the reporting object, such as "This road is blocked".

[0074] In one embodiment of this disclosure, the evaluation model is a machine learning model. The input of the evaluation model is the structured and unstructured information of the work order, and the output of the evaluation model is the credibility evaluation result of the work order.

[0075] In one embodiment of this disclosure, the evaluation model includes a structured feature extraction layer, an unstructured feature extraction layer, an attention mechanism layer, and a nonlinear transformation layer. The structured feature extraction layer is used to extract structured feature vectors for various types of data information from the structured information, with each type of data information corresponding to a specific structured feature vector. The unstructured feature extraction layer is used to extract unstructured feature vectors from the unstructured information. This attention mechanism layer utilizes the attention weights of each structured feature vector in the layer to convert the structured feature vectors of various data types into attention-weighted feature vectors. These attention weights can be obtained through machine learning training. In this embodiment, the attention mechanism layer uses unstructured feature vectors as basis vectors and evaluates the attention weights of each structured feature vector using these basis vectors. For example, assuming the structured feature vectors of various data types are part1, part2, part3, ..., partN, and the corresponding attention weights for each structured feature vector are w1, w2, w3, ..., wN, the initial values ​​of these attention weights can be randomly generated. As the evaluation model is trained end-to-end with sample data, these attention weights will be continuously updated until convergence, resulting in w1, w2, w3, ..., wN. Multiplying part1, part2, part3, ..., partN by the corresponding attention weights w1, w2, w3, ..., wN yields the attention-weighted feature vectors st1, st2, st3, ..., stN.

[0076] In one embodiment of this disclosure, the nonlinear transformation layer can map attention-weighted feature vectors and unstructured feature vectors to a probability value or other data as a credibility evaluation result.

[0077] In the above implementation, firstly, the structured and unstructured information of the work order is acquired. A structured feature extraction layer is used to extract structured feature vectors for various data types within the structured information, and an unstructured feature extraction layer is used to extract unstructured feature vectors from the unstructured information. Then, the attention weights of each structured feature vector in the attention mechanism layer (these attention weights are obtained through training and evaluation of each structured feature vector using unstructured feature vectors) are used to convert the structured feature vector into an attention-weighted feature vector. Finally, the attention-weighted feature vector and the unstructured feature vector are concatenated and transformed through a nonlinear transformation layer to obtain the credibility assessment result of the work order. In this way, different attention weights can be assigned to different structured feature vectors at the work order level, avoiding the output of the same attention weights for different structured feature vectors of the work order. Furthermore, the influence of unstructured information on structured information is taken into account, making the credibility assessment result of the work order more accurate and closer to reality. After obtaining accurate work order credibility assessment results, the processing priority, manual processing procedures, and even handling strategies for the work order can be determined to achieve the goal of production line optimization. In some simple and non-critical scenarios, automated processing can be directly performed based on the credibility assessment results of the work order. For example, when the credibility assessment result is very high, manual review is not required, and the work order can be automatically processed according to the preset processing strategy. Automating the processing of a portion of work orders will improve the work order processing speed and optimize the experience of the reporting recipients; in addition, it can also reduce labor costs.

[0078] In one embodiment of this disclosure, the input to the structured feature extraction layer is graph data, and the method may further include the following steps:

[0079] The various types of data information in the structured information are converted into graph data, where the vertices of the graph are the data information of each data type in the structured information, and the edges of the graph are the relationships between the data information of each data type.

[0080] In this embodiment, the graph data is a data graph composed of vertices and edges connecting the vertices. In this embodiment, the vertices of the graph are four types of data information in the structured information: data information of reporting objects, work order data information, historical operation information of reporting objects, and error entity information. The edges of the graph are the relationship quantities between these four types of data information.

[0081] In this implementation, the graph feature extraction layer can predict and output new vertex feature vectors, i.e., structured feature vectors of various data types, for each vertex in the graph based on the data information of each vertex and the relationship between the data information. For example, the graph feature extraction layer can be a GAT (Graph Attention Network) layer.

[0082] In this implementation, using graphs for graph feature extraction can more accurately express the information contained in various types of structured data. This is because the properties of graphs ensure that the influence of other types of data on this type of data can be absorbed when extracting the structured feature vectors of various types of data, thus having the possibility of achieving global optimization. In this way, the extracted structured feature vectors represent the structured information more accurately, thereby making the credibility assessment results obtained from them more accurate.

[0083] In one embodiment of this disclosure, the unstructured information includes error text information, and the method further includes:

[0084] The error text information is converted into an initial vector using a word vector algorithm;

[0085] The step of extracting unstructured feature vectors of the unstructured information through the unstructured feature extraction layer in the evaluation model includes:

[0086] The unstructured feature vector is obtained by extracting features from the initial vector through the unstructured feature extraction layer composed of convolutional and pooling layers in the evaluation model.

[0087] In this implementation, when the unstructured information includes error text, it is necessary to first convert the error text into an initial vector using a word vector algorithm. There are many word vector algorithms, all based on the idea that the meaning of any word can be represented by its surrounding words. Word vector algorithms can be divided into statistical algorithms and language model-based algorithms.

[0088] In this embodiment, the unstructured feature extraction layer in the evaluation model includes a convolutional layer and a pooling layer. The convolutional layer can be multiple convolutional layers with different kernel sizes. The convolutional layer is used to extract features, while the pooling layer can reduce the number of parameters.

[0089] In one embodiment of this disclosure, the method may further include the following steps:

[0090] Obtain sample data, which includes structured information, unstructured information, and credibility of sample work orders;

[0091] The initial evaluation model is trained based on the sample data, and the parameters in the initial evaluation model are adjusted until the accuracy of the evaluation result reaches a preset threshold to obtain the evaluation model. The evaluation model includes a structured feature extraction layer, an unstructured feature extraction layer, an attention mechanism layer, and a nonlinear transformation layer.

[0092] In this implementation, the sample work order can be a historical work order with accurate credibility. The structured and unstructured information from each sample work order can be input into the initial evaluation model. The initial evaluation model then performs feature extraction on the structured information by the structured feature extraction layer, feature extraction on the unstructured information by the unstructured feature extraction layer, and transformation of the extracted structured feature vector by the attention mechanism layer to obtain the attention-weighted feature vector. Finally, the attention-weighted feature vector and the unstructured feature vector are nonlinearly transformed by the nonlinear transformation layer to obtain the credibility value output by the initial evaluation model. The parameters in the initial evaluation model are continuously adjusted until the accuracy of the output credibility value compared with the credibility of the sample work order reaches a preset threshold, such as 97% or 99%. Then the training is completed, and the evaluation model is obtained.

[0093] It should be noted that the evaluation model can be trained by the device executing the work order evaluation method, or it can be trained by other devices and sent to the device executing the work order evaluation method. There are no restrictions here.

[0094] In one embodiment of this disclosure, step S105, which involves obtaining the credibility assessment result of the work order through the nonlinear transformation layer in the evaluation model based on the attention-weighted feature vector and the unstructured feature vector, includes:

[0095] Based on the historical error acceptance rate of the reporting object corresponding to the work order, the unstructured feature vector is corrected to obtain the unstructured corrected feature vector of the work order.

[0096] Based on the attention-weighted feature vector and the unstructured modified feature vector, the credibility evaluation result of the work order is obtained through the nonlinear transformation layer in the evaluation model.

[0097] In this implementation, different historical error acceptance ratios correspond to different weight values. Based on these weight values, the unstructured feature vector can be corrected to obtain the corrected unstructured feature vector of the work order.

[0098] In this implementation, the historical error acceptance rate of the reporting object helps to make a more accurate judgment on the credibility of the work order at the spatiotemporal level. In addition, the introduction of the historical error acceptance rate of the reporting object also plays a weighting role on unstructured information, thereby making the final feature vector representation of unstructured information more accurate.

[0099] In one embodiment of this disclosure, the aforementioned nonlinear transformation layer includes a sigmoid layer or a softmax layer.

[0100] In this embodiment, the sigmoid layer includes a sigmoid function that can map variables within [0, 1]. The variables in the sigmoid function can be attention-weighted feature vectors and unstructured modified feature vectors.

[0101] In this implementation, the softmax layer includes a softmax function (normalized exponential function), which can "compress" a K-dimensional vector z into another K-dimensional real vector σ(z), such that the range of each element is between (0, 1) and the sum of all elements is 1. In this embodiment, the work order credibility evaluation result output by the softmax function is the credibility probability and the untrustworthiness probability of the work order, and the sum of the two probabilities is 1.

[0102] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein.

[0103] Figure 2 This diagram illustrates a structural block diagram of a work order evaluation apparatus according to an embodiment of the present disclosure. This apparatus can be implemented as part or all of an electronic device through software, hardware, or a combination of both. Figure 2 As shown, the work order evaluation device includes:

[0104] The first acquisition module 201 is configured to acquire structured and unstructured information of work orders;

[0105] The first extraction module 202 is configured to extract structured feature vectors of various types of data information in the structured information through the structured feature extraction layer in the evaluation model;

[0106] The second extraction module 203 is configured to extract the unstructured feature vector of the unstructured information through the unstructured feature extraction layer in the evaluation model;

[0107] The attention weighting module 204 is configured to take the structured feature vector as input to the attention mechanism layer in the evaluation model, obtain the attention weighted feature vector based on the attention weight of each structured feature vector in the attention mechanism layer, and train the attention mechanism layer by evaluating the attention weight of each structured feature vector through unstructured feature vectors.

[0108] The transformation module 205 is configured to obtain the credibility evaluation result of the work order through the nonlinear transformation layer in the evaluation model based on the attention-weighted feature vector and the unstructured feature vector.

[0109] As mentioned above, with the development of technology, there are more and more applications on terminals. Users often encounter problems while using these applications. In such cases, users can submit their problems, forming user feedback tickets. The server can evaluate the credibility of these user feedback tickets and select those with high credibility for automated processing based on the evaluation results. This improves ticket processing speed, optimizes user experience, and reduces manual costs. These user feedback tickets include structured information such as user identity information and unstructured information such as the error text submitted by the user. Current credibility evaluation schemes for user feedback tickets directly concatenate these two parts of data, failing to consider the coupling characteristics between the two parts and ignoring the connection between unstructured and structured information. Therefore, the resulting credibility evaluation results are inaccurate.

[0110] In view of the above problems, this embodiment proposes a work order evaluation method. This method uses an attention mechanism to evaluate the importance of each structured feature in the structured information based on the unstructured information in the work order. In this way, the features of the two types of information are better integrated to evaluate the credibility of the work order. Different importance can be assigned to different structured features at the work order level, avoiding the output of the same importance for different structured features of the work order. Moreover, the influence of unstructured information on structured information is taken into account, making the evaluation results of the work order more accurate and closer to reality.

[0111] In one embodiment of this disclosure, the work order evaluation method can be applied to computers, computing devices, electronic devices, servers, service clusters, etc., that can perform work order evaluations.

[0112] In one embodiment of this disclosure, a work order refers to a work order from a reporting entity, such as a user, regarding various issues reported by the application client, and can be called a user feedback work order. This work order can be a feedback work order reported by the reporting entity when encountering a problem while using the application client, or it can be a feedback work order submitted on the application client after the reporting entity proactively discovers a problem.

[0113] In one embodiment of this disclosure, the structured information refers to quantitative data information, which is information that can be represented by data or a unified structure, such as the identity information and submission time of the reporting object who submitted the work order. The structured information in this embodiment can be divided into multiple types of data information, including reporting object data information, work order data information, reporting object historical operation information, and error entity information. The reported object data information can include the reported object's identity information, the duration of the reported object's use of the application client, and other reported object profile information. The work order data information includes the work order submission time, the classification of the entity involved in the work order, such as the classification of the problem involved in the work order for a navigation application client as the city where the work order is located, or the classification of the problem involved in the work order for a short video application client as the category of the short video involved in the work order, and so on. The reported object's historical operation information refers to the various operation records of the reported object on the application client for which the work order is targeted within a historical period. The error entity information refers to the relevant information of the problem entity reflected in the work order. For example, if a navigation application client has three types of feedback work orders: POI, public transport, and road, and this work order is a road feedback work order for a navigation application client, the error entity is the road, and the error entity information is the relevant information of the road, such as the road ID, road location, etc.

[0114] In one embodiment of this disclosure, the unstructured information is data information with variable fields, including all information other than structured information, and can be text information or non-text information. For example, when the application client targeted by the work order is a navigation client, the unstructured information of the work order can be the error text information submitted by the reporting object, such as "This road is blocked".

[0115] In one embodiment of this disclosure, the evaluation model is a machine learning model. The input of the evaluation model is the structured and unstructured information of the work order, and the output of the evaluation model is the credibility evaluation result of the work order.

[0116] In one embodiment of this disclosure, the evaluation model includes a structured feature extraction layer, an unstructured feature extraction layer, an attention mechanism layer, and a nonlinear transformation layer. The structured feature extraction layer is used to extract structured feature vectors for various types of data information from the structured information, with each type of data information corresponding to a specific structured feature vector. The unstructured feature extraction layer is used to extract unstructured feature vectors from the unstructured information. This attention mechanism layer utilizes the attention weights of each structured feature vector in the layer to convert the structured feature vectors of various data types into attention-weighted feature vectors. These attention weights can be obtained through machine learning training. In this embodiment, the attention mechanism layer uses unstructured feature vectors as basis vectors and evaluates the attention weights of each structured feature vector using these basis vectors. For example, assuming the structured feature vectors of various data types are part1, part2, part3, ..., partN, and the corresponding attention weights for each structured feature vector are w1, w2, w3, ..., wN, the initial values ​​of these attention weights can be randomly generated. As the evaluation model is trained end-to-end with sample data, these attention weights will be continuously updated until convergence, resulting in w1, w2, w3, ..., wN. Multiplying part1, part2, part3, ..., partN by the corresponding attention weights w1, w2, w3, ..., wN yields the attention-weighted feature vectors st1, st2, st3, ..., stN.

[0117] In one embodiment of this disclosure, the nonlinear transformation layer can map attention-weighted feature vectors and unstructured feature vectors to a probability value or other data as a credibility evaluation result.

[0118] In the above implementation, firstly, the structured and unstructured information of the work order is acquired. A structured feature extraction layer is used to extract structured feature vectors for various data types within the structured information, and an unstructured feature extraction layer is used to extract unstructured feature vectors from the unstructured information. Then, the attention weights of each structured feature vector in the attention mechanism layer (these attention weights are obtained through training and evaluation of each structured feature vector using unstructured feature vectors) are used to convert the structured feature vector into an attention-weighted feature vector. Finally, the attention-weighted feature vector and the unstructured feature vector are concatenated and transformed through a nonlinear transformation layer to obtain the credibility assessment result of the work order. In this way, different attention weights can be assigned to different structured feature vectors at the work order level, avoiding the output of the same attention weights for different structured feature vectors of the work order. Furthermore, the influence of unstructured information on structured information is taken into account, making the credibility assessment result of the work order more accurate and closer to reality. After obtaining accurate work order credibility assessment results, the processing priority, manual processing procedures, and even handling strategies for the work order can be determined to achieve the goal of production line optimization. In some simple and non-critical scenarios, automated processing can be directly performed based on the credibility assessment results of the work order. For example, when the credibility assessment result is very high, manual review is not required, and the work order can be automatically processed according to the preset processing strategy. Automating the processing of a portion of work orders will improve the work order processing speed and optimize the experience of the reporting recipients; in addition, it can also reduce labor costs.

[0119] In one embodiment of this disclosure, the input to the structured feature extraction layer is graph data, and the apparatus may further include:

[0120] The conversion module is configured to convert various types of data information in the structured information into graph data, wherein the vertices of the graph are the data information of various data types in the structured information, and the edges of the graph are the relationships between the data information of various data types.

[0121] In this embodiment, the graph data is a data graph composed of vertices and edges connecting the vertices. In this embodiment, the vertices of the graph are four types of data information in the structured information: data information of reporting objects, work order data information, historical operation information of reporting objects, and error entity information. The edges of the graph are the relationship quantities between these four types of data information.

[0122] In this implementation, the graph feature extraction layer can predict and output new vertex feature vectors, i.e., structured feature vectors of various data types, for each vertex in the graph based on the data information of each vertex and the relationship between the data information. For example, the graph feature extraction layer can be a GAT (Graph Attention Network) layer.

[0123] In this implementation, using graphs for graph feature extraction can more accurately express the information contained in various types of structured data. This is because the properties of graphs ensure that the influence of other types of data on this type of data can be absorbed when extracting the structured feature vectors of various types of data, thus having the possibility of achieving global optimization. In this way, the extracted structured feature vectors represent the structured information more accurately, thereby making the credibility assessment results obtained from them more accurate.

[0124] In one embodiment of this disclosure, where the unstructured information includes error text information, the apparatus further includes:

[0125] The vector conversion module is configured to convert the error text information into an initial vector using a word vector algorithm;

[0126] The second extraction module 203 is configured as follows:

[0127] The unstructured feature vector is obtained by extracting features from the initial vector through the unstructured feature extraction layer composed of convolutional and pooling layers in the evaluation model.

[0128] In this implementation, when the unstructured information includes error text, it is necessary to first convert the error text into an initial vector using a word vector algorithm. There are many word vector algorithms, all based on the idea that the meaning of any word can be represented by its surrounding words. Word vector algorithms can be divided into statistical algorithms and language model-based algorithms.

[0129] In this embodiment, the unstructured feature extraction layer in the evaluation model includes a convolutional layer and a pooling layer. The convolutional layer can be multiple convolutional layers with different kernel sizes. The convolutional layer is used to extract features, while the pooling layer can reduce the number of parameters.

[0130] In one embodiment of this disclosure, the apparatus further includes:

[0131] The second acquisition module is configured to acquire sample data, which includes structured information, unstructured information and credibility of sample work orders.

[0132] The training module is configured to train an initial evaluation model based on the sample data, adjust the parameters in the initial evaluation model until the accuracy of the evaluation result reaches a preset threshold, and obtain an evaluation model. The evaluation model includes a structured feature extraction layer, an unstructured feature extraction layer, an attention mechanism layer, and a nonlinear transformation layer.

[0133] In this implementation, the sample work order can be a historical work order with accurate credibility. The structured and unstructured information from each sample work order can be input into the initial evaluation model. The initial evaluation model then performs feature extraction on the structured information by the structured feature extraction layer, feature extraction on the unstructured information by the unstructured feature extraction layer, and transformation of the extracted structured feature vector by the attention mechanism layer to obtain the attention-weighted feature vector. Finally, the attention-weighted feature vector and the unstructured feature vector are nonlinearly transformed by the nonlinear transformation layer to obtain the credibility value output by the initial evaluation model. The parameters in the initial evaluation model are continuously adjusted until the accuracy of the output credibility value compared with the credibility of the sample work order reaches a preset threshold, such as 97% or 99%. Then the training is completed, and the evaluation model is obtained.

[0134] It should be noted that the evaluation model can be trained by the device executing the work order evaluation method, or it can be trained by other devices and sent to the device executing the work order evaluation method. There are no restrictions here.

[0135] In one embodiment of this disclosure, the transformation module 205 is configured as follows:

[0136] Based on the historical error acceptance rate of the reporting object corresponding to the work order, the unstructured feature vector is corrected to obtain the unstructured corrected feature vector of the work order.

[0137] Based on the attention-weighted feature vector and the unstructured modified feature vector, the credibility evaluation result of the work order is obtained through the nonlinear transformation layer in the evaluation model.

[0138] In this implementation, different historical error acceptance ratios correspond to different weight values. Based on these weight values, the unstructured feature vector can be corrected to obtain the corrected unstructured feature vector of the work order.

[0139] In this implementation, the historical error acceptance rate of the reporting object helps to make a more accurate judgment on the credibility of the work order at the spatiotemporal level. In addition, the introduction of the historical error acceptance rate of the reporting object also plays a weighting role on unstructured information, thereby making the final feature vector representation of unstructured information more accurate.

[0140] In one embodiment of this disclosure, the aforementioned nonlinear transformation layer includes a sigmoid layer or a softmax layer.

[0141] In this embodiment, the sigmoid layer includes a sigmoid function that can map variables within [0, 1]. The variables in the sigmoid function can be attention-weighted feature vectors and unstructured modified feature vectors.

[0142] In this implementation, the softmax layer includes a softmax function (normalized exponential function), which can "compress" a K-dimensional vector z into another K-dimensional real vector σ(z), such that the range of each element is between (0, 1) and the sum of all elements is 1. In this embodiment, the work order credibility evaluation result output by the softmax function is the credibility probability and the untrustworthiness probability of the work order, and the sum of the two probabilities is 1.

[0143] This disclosure also discloses a navigation service, wherein, based on the above-described parking lot status determination method, a work order evaluation result of the navigated object is obtained, and guidance services for the navigated object in a corresponding scenario are provided based on the work order evaluation result. The corresponding scenario is one or more combinations of AR navigation, elevated road navigation, or main and auxiliary road navigation.

[0144] This disclosure also discloses a navigation method, wherein a navigation route is obtained based on at least the start point, the end point, the credibility assessment result of the work order, and the road conditions; navigation guidance is performed based on the navigation route; and the credibility assessment result of the work order is implemented based on any of the methods described above.

[0145] This disclosure also discloses an electronic device. Figure 3 This diagram illustrates a structural block diagram of an electronic device according to an embodiment of the present disclosure, such as... Figure 3 As shown, the electronic device 300 includes a memory 301 and a processor 302; wherein,

[0146] The memory 301 is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor 302 to implement the above method steps.

[0147] Figure 4 This is a schematic diagram of the structure of a computer system suitable for implementing a work order evaluation method according to an embodiment of the present disclosure.

[0148] like Figure 4As shown, the computer system 400 includes a processing unit 401, which can execute various processes described above based on a program stored in a read-only memory (ROM) 402 or a program loaded from a storage section 408 into a random access memory (RAM) 403. The RAM 403 also stores various programs and data required for the operation of the system 400. The processing unit 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0149] The following components are connected to I / O interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to I / O interface 405 as needed. A removable medium 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 410 as needed so that computer programs read from it can be installed into storage section 408 as needed. The processing unit 401 can be implemented as a CPU, GPU, TPU, FPGA, NPU, etc.

[0150] In particular, according to embodiments of this disclosure, the methods described above can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program tangibly contained on a readable medium thereof, the computer program containing program code for performing the work order evaluation method. In such embodiments, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411.

[0151] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0152] The units or modules described in the embodiments of this disclosure can be implemented in software or hardware. The described units or modules can also be located in a processor, and the names of these units or modules do not necessarily constitute a limitation on the unit or module itself.

[0153] In another aspect, embodiments of this disclosure also provide a computer-readable storage medium, which may be a computer-readable storage medium included in the apparatus described in the above embodiments; or it may be a standalone computer-readable storage medium not assembled into a device. The computer-readable storage medium stores one or more programs, which are used by one or more processors to perform the methods described in embodiments of this disclosure.

[0154] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A work order evaluation method, comprising: Obtain structured and unstructured information of work orders. The structured information refers to quantitative data information, which is information that can be represented by data or a unified structure. The structured feature vectors of various types of data information in the structured information are extracted by the structured feature extraction layer in the evaluation model; The unstructured feature vector of the unstructured information is extracted through the unstructured feature extraction layer in the evaluation model; The structured feature vectors are used as input to the attention mechanism layer in the evaluation model. An attention-weighted feature vector is obtained based on the attention weights of each structured feature vector in the attention mechanism layer. The attention mechanism layer is trained by evaluating the attention weights of each structured feature vector using unstructured feature vectors. Based on the attention-weighted feature vector and the unstructured feature vector, the credibility evaluation result of the work order is obtained through the nonlinear transformation layer in the evaluation model.

2. The method according to claim 1, wherein, The input to the structured feature extraction layer is graph data, and the method further includes: The various types of data information in the structured information are converted into graph data, where the vertices of the graph are the data information of each data type in the structured information, and the edges of the graph are the relationships between the data information of each data type.

3. The method according to claim 1, wherein, The unstructured information includes error text information, and the method further includes: The error text information is converted into an initial vector using a word vector algorithm; The step of extracting unstructured feature vectors of the unstructured information through the unstructured feature extraction layer in the evaluation model includes: The unstructured feature vector is obtained by extracting features from the initial vector through the unstructured feature extraction layer composed of convolutional and pooling layers in the evaluation model.

4. The method according to claim 1, wherein, The method further includes: Obtain sample data, which includes structured information, unstructured information, and credibility of sample work orders; The initial evaluation model is trained based on the sample data, and the parameters in the initial evaluation model are adjusted until the accuracy of the evaluation result reaches a preset threshold to obtain the evaluation model. The evaluation model includes a structured feature extraction layer, an unstructured feature extraction layer, an attention mechanism layer, and a nonlinear transformation layer.

5. The method according to claim 1, wherein, The process of obtaining the credibility assessment result of the work order based on the attention-weighted feature vector and the unstructured feature vector through the nonlinear transformation layer in the evaluation model includes: Based on the historical error acceptance rate of the reporting object corresponding to the work order, the unstructured feature vector is corrected to obtain the unstructured corrected feature vector of the work order. Based on the attention-weighted feature vector and the unstructured modified feature vector, the credibility evaluation result of the work order is obtained through the nonlinear transformation layer in the evaluation model.

6. The method according to claim 1 or 5, wherein, The nonlinear transformation layer includes a sigmoid layer or a softmax layer.

7. The method according to claim 1, wherein, The structured information includes various types of data, such as reporting object data, work order data, historical operation information of the reporting object, and error entity information.

8. An electronic device comprising a memory and at least one processor; wherein, The memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the at least one processor to implement the steps of the method according to any one of claims 1-7.

9. A computer program product comprising a computer program / instructions, wherein, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-7.

10. A navigation method, wherein, Obtain a navigation route calculated based at least on the origin, destination, work order credibility assessment results, and road conditions; provide navigation guidance based on the navigation route; wherein the work order credibility assessment results are implemented based on any one of the methods described in claims 1-7.

Images