Digital employee robot performance intelligent evaluation method and system

Abstract

The invention provides a digital employee robot performance intelligent evaluation method and system, and relates to the technical field of data processing, and the method comprises the steps: carrying out the feature extraction of user interaction contents in all scenes; performing fluctuation detection, identifying a time period of abnormal fluctuation, and extracting user request content and robot response content in the time period; restoring the current session path based on a preset business process template, performing path consistency comparison in combination with a response rule of the robot under the path, and generating an abnormal node set if the path is interrupted or the response is offset; extracting a user request format, a robot response behavior and an external data calling state corresponding to each abnormal node, performing abnormal source analysis, calculating user input abnormity, and identifying an attribution probability of abnormity and data interface abnormity; according to the invention, autonomy and accuracy of robot performance intelligent evaluation are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a digital employee robot performance intelligent evaluation method and system. BACKGROUND

[0002] In the prior art, the performance evaluation of digital employee robots mainly relies on manually set multi-dimensional evaluation indicators, such as task completion rate, response speed, operation accuracy, etc. The evaluation process usually includes collecting log data of the robot in the actual running process, counting the performance of each indicator, and generating a fixed format analysis report by the background system. This method generally uses static rules to process and score data, does not have the ability to dynamically adapt and adjust, and is difficult to optimize for different business scenarios.

[0003] Taking the enterprise customer service scene as an example, the current commonly used performance evaluation system often only counts the accuracy rate of the robot's answer to the customer's question, and fails to comprehensively consider the complex changes of customer intent. In the face of peak hours, a large number of non-standardized questions, the robot may have short-term performance fluctuations, and the existing system cannot timely capture and distinguish the fluctuation causes during such abnormal periods, resulting in inaccurate evaluation results for judging the actual business impact. Specifically, the evaluation report only reflects the overall accuracy rate decline, but cannot distinguish whether it is caused by robot model recognition failure, external data anomaly, or customer input feature change, affecting the pertinence and effectiveness of subsequent optimization decisions. SUMMARY

[0004] The purpose of the present application is to provide a digital employee robot performance intelligent evaluation method and system, which aims to solve the problems mentioned in the background art.

[0005] To solve the above technical problems, the technical solution of the present application is as follows: In a first aspect, a digital employee robot performance intelligent evaluation method, the method comprising: Obtaining the running log data of the robot in the specified application scene, and classifying it by time period and business category to obtain a scene-specific original data set; According to the scene-specific original data set, the user interaction content in each scene is extracted, including identifying keywords, input methods and time labels in the user request, and generating scene interaction feature data; Performing fluctuation detection on the scene interaction feature data, identifying the time period of abnormal fluctuation, and extracting the user request content and robot response content in the time period to generate abnormal segment data; For abnormal segment data, restore the current session path based on the pre-set business process template, and perform path consistency comparison combined with the response rules of the robot in the path, if the path is interrupted or the response is offset, an abnormal node set is generated; According to the abnormal node set, the user request format, the robot response behavior and the external data calling state corresponding to each abnormal node are extracted, abnormal source analysis is performed, the attribution probability of user input abnormality, identification abnormality and data interface abnormality is calculated, and abnormal cause analysis results are generated; According to the abnormal cause analysis results, the main performance bottlenecks and optimization directions in each scene are summarized, and a comprehensive report containing scene-specific performance evaluation conclusions and cause analysis is generated.

[0006] Preferably, according to the scene-specific original data set, the user interaction content in each scene is extracted, including identifying keywords, input methods and time labels in user requests, to generate scene interaction feature data, including: According to the scene-specific original data set, the semantic segmentation and label mapping of each user request record are performed, the user intent keywords are extracted, and they are mapped to the preset intent classification system, to generate intent label data; According to the scene-specific original data set, the input method of the user request is parsed, and it is judged to be a structured form, a natural language text or a composite instruction type, and the corresponding type label is labeled, to generate input structure label data; According to the scene-specific original data set, the time stamp associated with each request record is sampled and aligned, the stage information in the business process is extracted, and the sequence timing label is generated in combination with the previous request content; The intent label data, input structure label data and sequence timing label are fused to form a complete scene interaction feature data set.

[0007] Preferably, the scene interaction feature data is subjected to fluctuation detection, the time period of abnormal fluctuation is identified, and the user request content and robot response content in the time period are extracted to generate abnormal segment data, including: According to the scene interaction feature data, a continuous time window sequence is constructed, the statistical distribution of the feature values in each time window is analyzed, and the change amplitude between adjacent windows is calculated to obtain fluctuation evaluation parameters; According to the fluctuation evaluation parameters, the time windows with a change amplitude exceeding a preset amplitude threshold are labeled, and the user request and robot response content in the time period are extracted to form candidate abnormal segments; The semantic matching degree between each request and response content in the candidate abnormal segment is analyzed, and by comparing the semantic matching degree of each request and response content with a preset fluctuation interval threshold, the request and response content pair with a semantic matching degree exceeding the preset fluctuation interval threshold is taken as a response offset segment, to generate a response offset segment; The response offset segment is sorted according to the conversation time sequence to construct abnormal segment data that maintains the continuity of the context.

[0008] Preferably, for the abnormal segmented data, the current session path is restored based on the preset business process template, and the path consistency comparison is performed in combination with the response rules of the robot at the path. If the path is interrupted or the response is deviated, an abnormal node set is generated, including: According to the time sequence and request content of the user request in the abnormal segmented data, a task keyword is extracted, and a session task node sequence is constructed according to the process node information corresponding to the task keyword; Based on the preset business process template, a standard task execution sequence in the same application scenario as the session task node sequence is obtained, and the session task node sequence and the standard task execution sequence are compared node by node to identify the breakpoint, jump point or repeated node in the node sequence, and a node comparison result is generated; According to the node comparison result, the robot response content at the breakpoint, jump point or repeated node is matched and verified with the standard response rule. When there is a response content inconsistent with the node rule, the abnormal node and its corresponding user request content, robot response content and external data calling state are encapsulated as abnormal node data, and an abnormal node set is constructed.

[0009] Preferably, according to the scene-based original data set, each user request record is semantically segmented and labeled to extract user intent keywords, which are mapped to a preset intent classification system to generate intent labeling data, including: According to the scene-based original data set, the user request record content is morphologically segmented and semantic phrase extraction is performed to obtain a candidate keyword group containing a verb and an object structure; According to the candidate keyword group, the semantic similarity value of each standard label in the preset intent classification system is calculated, and the label exceeding the preset semantic similarity threshold is identified to form a preliminary intent mapping result; According to the preliminary intent mapping result, the context semantics of the current user request and adjacent historical requests are combined for sequence consistency analysis and label weight adjustment to obtain a final intent label; The final intent label is bound to the current user identifier to generate structured intent labeling data.

[0010] Preferably, the semantic matching degree between each request and response content in the candidate abnormal segment is analyzed piece by piece. By comparing the semantic matching degree of each request and response content with the preset fluctuation interval threshold, the request and response content pair with the semantic matching degree exceeding the preset fluctuation interval threshold is taken as the response deviation segment to generate the response deviation segment, including: The text content of each request and response content in the candidate abnormal segment is extracted, a semantic vector is generated through a semantic feature conversion method, and a corresponding request vector and response vector group are constructed; A semantic distance is calculated between each set of request vectors and their response vectors, and compared with a preset fluctuation interval threshold to obtain a semantic shift value; According to the semantic shift value, the request vector and the response vector set exceeding the preset shift threshold are identified, and a similarity clustering process is performed to extract a response drift segment that meets a preset clustering similarity threshold; The response drift segment that meets the preset clustering similarity threshold is arranged in chronological order, and combined with the corresponding user interaction record to generate a response drift segment dataset with coherent context.

[0011] Preferably, according to the abnormal node set, the user request format, the robot response behavior and the external data calling state corresponding to each abnormal node are extracted for abnormal source analysis, the attribution probability of user input abnormality, identification abnormality and data interface abnormality is calculated, and abnormal cause analysis results are generated, including: According to the abnormal node set, each abnormal node is extracted and the following processing is performed: The user request record corresponding to each abnormal node is extracted, the structural length, semantic organization form and input timing stability of the request text are feature encoded, and an input feature vector is generated; The response content made by the robot in each abnormal node is extracted, the semantic correspondence degree of the response content and the flow node and the response delay time are calculated, and a response feature vector is generated; The data interface calling information triggered by each abnormal node is extracted, including the interface calling return state, whether the response data is empty, and the interface calling delay interval, and a data interface feature vector is generated; The input feature vector, the response feature vector and the data interface feature vector are combined to form a unified attribution feature set; The attribution feature set is input into a preset attribution processing model, and the attribution probability value of the attribution probability distribution of the user input abnormality, the identification abnormality and the data interface abnormality is calculated through multi-dimensional feature proportion; The abnormal type with the maximum attribution probability value is taken as the determination result, and abnormal cause analysis data containing an abnormal type label is generated.

[0012] In a second aspect, a digital employee robot performance intelligent evaluation system is provided, and the system comprises: A log classification module is configured to obtain running log data of the robot in a specified application scenario, and classify the log data by time period and business category to obtain a scene-specific original dataset; An interaction feature extraction module is configured to extract features of user interaction content in each scene based on the scene-specific original dataset, identify keywords, input methods and time labels in user requests, and generate scene interaction feature data; The fluctuation detection module is configured to perform fluctuation detection on the scene interaction feature data, identify an abnormal fluctuation time period, and extract user request content and robot response content in the time period to generate abnormal segmented data; The path consistency analysis module is configured to restore a path based on a preset business process template, and perform consistency comparison on the abnormal segmented data and a response rule of the robot in the path. The cause attribution module is configured to extract user request formats, robot response behaviors and external data calling states corresponding to each abnormal node based on the abnormal node set, perform abnormal source analysis, calculate attribution probabilities of user input abnormalities, identification abnormalities and data interface abnormalities, and generate abnormal cause analysis results. The performance evaluation module is configured to induce main performance bottlenecks and optimization directions in each scene based on the abnormal cause analysis results, and generate a comprehensive report including scene-specific performance evaluation conclusions and cause analysis.

[0013] The above scheme of the present application at least has the following beneficial effects: Firstly, the running log data of the robot in the specified application scene is automatically acquired, and multi-dimensional classification is performed in combination with time periods and business categories, so that the structured management of the original data is realized. This method effectively improves the pertinence of data organization and the refinement degree of analysis, and can provide accurate and traceable basic data for performance analysis in different business scenes, overcoming the problem of extensive data classification in the prior art, which cannot meet the business diversity requirements.

[0014] Secondly, the present application not only performs in-depth feature extraction on user interaction content, including systematic analysis of keywords, input methods and time labels of each request, but also further uses a fluctuation detection algorithm to dynamically identify abnormal fluctuation periods, and automatically extracts related request and response content when an abnormality occurs. By structurally organizing the abnormal segmented data, the performance fluctuation nodes and specific influencing factors can be located in time during business peak periods, scene switching or complex customer intention changes, which makes up for the deficiency of the prior art that cannot dynamically respond to actual business changes.

[0015] Thirdly, the present application combines abnormal segmented data with a preset business process template, automatically restores the actual conversation path, and performs consistency comparison on each node and response rule, so as to accurately locate abnormal nodes such as business process interruption and response deviation. This mechanism can distinguish whether the performance fluctuation is caused by user input mode change, robot recognition mechanism problem or external data interface abnormality, thereby providing an operable basis for subsequent attribution analysis and performance optimization.

[0016] Further, by inputting the multi-source features such as the user request format of the abnormal node, the robot response behavior, the external data calling state and the like into the attribution analysis process, the system can automatically calculate the attribution probability of each type of abnormality, and realize the multi-dimensional and multi-factor fusion of abnormal cause discrimination. Finally, the application can induce the performance bottleneck and optimization direction in each scene according to the attribution analysis result, generate a comprehensive evaluation report containing detailed cause analysis and business improvement suggestions, and significantly improve the intelligent performance evaluation and business guidance value.

[0017] Overall, the application breaks through the defects of static, single, extensive and passive in traditional digital employee robot performance evaluation, realizes the full-process automation, scene refinement and multi-dimensional dynamic attribution of the evaluation process, greatly enhances the adaptability of the system to the complexity of the actual business, and provides efficient, intelligent and fine technical support for the continuous optimization of enterprise digital operation. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is a flow chart of a digital employee robot performance intelligent evaluation method provided by an embodiment of the application. DETAILED DESCRIPTION

[0019] Exemplary embodiments of the present disclosure will be described in greater detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be accurately conveyed to those skilled in the art.

[0020] As shown in Figure 1 , an embodiment of the application proposes a digital employee robot performance intelligent evaluation method, which comprises: obtaining the running log data of the robot in the specified application scene, and classifying it by time period and business category to obtain a scene-specific original data set; According to the scene-specific original data set, the user interaction content in each scene is extracted, including identifying the keywords, input method and time label in the user request, to generate scene interaction feature data; Performing fluctuation detection on the scene interaction feature data, identifying the time period of abnormal fluctuation, and extracting the user request content and robot response content in the time period to generate abnormal segment data; For abnormal segment data, restore the current session path based on the pre-set business process template, and perform path consistency comparison combined with the response rule of the robot in the path. If the path is interrupted or the response is offset, an abnormal node set is generated; According to the abnormal node set, the user request format, the robot response behavior and the external data calling state corresponding to each abnormal node are extracted, abnormal source analysis is performed, the attribution probability of user input abnormality, identification abnormality and data interface abnormality is calculated, and abnormal cause analysis results are generated; According to the abnormal cause analysis results, the main performance bottlenecks and optimization directions in each scene are summarized, and a comprehensive report containing scene-specific performance evaluation conclusions and cause analysis is generated.

[0021] In the embodiment of the application, the system first automatically obtains all running log data of the robot in a specified application scene. This data contains all interaction information of users and robots in multiple time periods and different business categories. After obtaining, the system classifies the original log data according to the time period and business category to which each piece of data belongs, and then constructs original data sets for different sub-scenes. Taking a customer service robot as an example, in the financial industry scene, the interaction logs generated in different working days, holidays and different business lines (such as financial consulting, insurance reporting, etc.) will be sorted out to form scene-specific original data sets, which are convenient for subsequent targeted analysis.

[0022] Subsequently, the system uses a pre-set feature extraction mechanism to extract key interaction information from each scene-specific original data set. Specifically, for each user request, the system will automatically identify the keywords in the request text, identify the input method of the user (such as text input, voice input, form click, etc.), and record the time label corresponding to the request. Through these steps, scene interaction feature data reflecting the interaction process of each user and the robot can be generated, ensuring that the subsequent performance analysis of the system has accurate and comprehensive basic data support.

[0023] On the basis of the scene interaction feature data, the system implements fluctuation detection analysis to timely discover the fluctuation of the robot performance. The method includes continuously monitoring the feature data, and automatically identifying the case where the performance significantly fluctuates in a specific time period by setting reasonable detection indicators. For example, during the holiday peak period, if it is found that the response accuracy of the robot has decreased significantly, the system can locate the specific abnormal fluctuation period. For these fluctuation periods, the system will further deeply mine the specific content of user requests and robot responses in the time period, and organize these contents into abnormal segment data to provide materials for subsequent problem root cause analysis.

[0024] The system also combines a preset business process template to restore the path of the above abnormal segmented data. By comparing the response rules of the robot in the process path, it checks whether the current session path is consistent with the standard process. When detecting path interruption or response deviation, the system will record these nodes to form an abnormal node set. Next, the system extracts the user request format, robot response behavior, external data call state, and other multi-dimensional features for each abnormal node, and uses attribution analysis model to analyze the source of the abnormality, and calculates the probability distribution of different causes such as user input abnormality, identification mechanism abnormality, and data interface abnormality. Finally, the system can automatically summarize the main performance bottlenecks of digital employee robots in actual operation based on these analysis results, propose optimization suggestions, and automatically generate a comprehensive report covering performance conclusions and cause analysis in different scenarios. Such an implementation ensures that the performance of digital employee robots in actual business deployment can be scientifically and objectively evaluated, which helps operation and maintenance personnel to timely discover and solve problems.

[0025] Among them, the running log data of the robot in the specified application scenario is obtained, and it is classified according to time period and business category to obtain a scene-specific original data set, which specifically includes: In the actual deployment of digital employee robots in a business environment, the system first needs to specify the target application scenario, such as online customer service, financial approval, or intelligent outbound call, etc. According to the scene setting, the system automatically connects to the running log collection port of the robot platform, and pulls the required log data in real time or periodically. Log data generally includes user request content, robot response text, request occurrence time, business type identifier, session identifier, and data interface call record, etc.

[0026] After obtaining the original log, the system divides all log data according to the time stamp field in the record according to the established time period, and the common division method includes day, hour or minute, to ensure effective monitoring of fluctuations in various time periods. On the basis of time division, the system further analyzes the business category label in the log, and attributes each log to the corresponding business process board, such as product consultation, complaint acceptance, process approval, information verification, etc. Through the above two steps, the system can realize multi-dimensional classification of the original log, and each type of log has a clear time attribute and business attribute.

[0027] Further, the system aggregates log data of the same time period and the same business category together to form structured scenario-specific raw data sets. Each scenario-specific raw data set can be regarded as the full interaction trajectory of the robot under a specific time window and business demand, providing an efficient data basis for subsequent feature extraction, behavior analysis, and anomaly detection. This classification and aggregation method can ensure that each analysis reflects the actual performance of the robot under business peaks, off-peak periods, and different business branches, improving data processing efficiency and the accuracy of evaluation results. This step combines existing log management and grouping techniques to achieve systematic archiving of robot behavior in a multi-business, multi-period operating environment.

[0028] According to the anomaly cause analysis results, the main performance bottlenecks and optimization directions in each scenario are summarized to generate a comprehensive report containing scenario-specific performance evaluation conclusions and cause analysis, including: After the system completes cause analysis of each type of abnormal node and outputs the abnormal cause analysis results, the system automatically summarizes the performance shortfalls in each business scenario based on the analysis results. Specifically, the system first counts all abnormal cause analysis data and sorts the distribution quantity and proportion of each abnormal type (such as user input abnormality, robot recognition abnormality, and data interface abnormality) in each scenario. The system further aggregates the high-frequency nodes, time periods, and trigger conditions of abnormal occurrences based on scenario attributes and historical running trends to form a summary of the main performance bottlenecks.

[0029] The system automatically sorts and lists the abnormal data, lists the abnormal types and specific trigger conditions that occur frequently, and points out the most likely problem points in the current business process. For example, if a large number of recognition abnormalities are found in a multi-round complex dialogue link in a certain business process, the system defines this process segment as a performance bottleneck. The system proposes targeted optimization directions based on the cause analysis results, such as simplifying the input process, enhancing the robot semantic understanding module, or optimizing the fault tolerance processing logic of the data interface. Each optimization suggestion is data-driven.

[0030] Finally, the system structures and organizes all scenario-specific bottleneck summaries, abnormal type distributions, specific trigger conditions, and optimization suggestions and outputs them as a comprehensive evaluation report. The report content includes performance evaluation conclusions, abnormal reason details, and improvement scheme descriptions for each business scenario, and auxiliary displays the abnormal distribution characteristics in the form of charts or text explanations. The entire processing flow combines existing big data analysis and business performance evaluation methods, enabling operation and business management personnel to intuitively understand the actual performance and improvement space of the robot in different scenarios, effectively supporting continuous business optimization and intelligent operation and maintenance decisions.

[0031] In a preferred embodiment of the present application, according to the scene-based original data set, the user interaction content in each scene is extracted, including identifying keywords in the user request, input method and time label, generating scene interaction feature data, including: According to the scene-based original data set, the semantic segmentation and label mapping of each user request record are performed, the user intention keywords are extracted, and the user intention keywords are corresponded to the preset intention classification system to generate intention label data; According to the scene-based original data set, the input method of the user request is analyzed, and is judged to be a structured form, a natural language text or a composite instruction type, and a corresponding type label is labeled to generate input structure label data; According to the scene-based original data set, the time stamp associated with each request record is sampled and aligned, the stage information thereof in the business process is extracted, and the sequence timing label is generated in combination with the previous request content; The intention label data, the input structure label data and the sequence timing label are fused to form a complete scene interaction feature data set.

[0032] In the embodiment of the present application, for the interaction content of the digital employee robot in the scene, a hierarchical and multi-feature fusion interaction feature extraction method is adopted. The system first extracts each user request from the scene-based original data set. The system applies a language processing engine to perform semantic segmentation on the request text, identifies the main verbs, objects and other core components in the text through intelligent algorithms, and maps these components to intention keywords. For example, when the user inputs “I want to claim travel expenses”, the system can accurately identify “claim” and “travel expenses” as intention keywords. These keywords are matched with the preset intention classification system to generate intention label data for the request, and all subsequent related analysis is based on this.

[0033] The system further analyzes the input method of each request to determine whether the request is a structured form input (such as a drop-down menu selection), a natural language text input, or a composite instruction containing multiple operations. Each input method will be labeled with a specific type label by the system to ensure that subsequent processing logic can be targeted for different input methods. For example, a structured form request can be directly converted into structured data, and a text input needs to be further analyzed by natural language.

[0034] Meanwhile, the system also normalizes the timestamp of each request record to ensure that all time labels can be sampled and aligned according to a unified standard. For scenarios involving complex business processes, the system also identifies the correspondence between the current request and the business process stage in combination with the content of the previous user request, and then generates a sequence timing label for each request. This mechanism can accurately reflect the evolution trajectory of different stages in the multi-round dialogue process between the user and the robot. For example, when the user switches intent multiple times in the business handling process, the system can track the entire business handling path through the timing label to avoid missing important context information.

[0035] Finally, the system fuses all extracted intent label data, input structure label data, and sequence timing labels to form a complete and structured scenario interaction feature dataset. Through this method, both the comprehensiveness of feature extraction and the rich data foundation for subsequent volatility detection and anomaly analysis are ensured.

[0036] Among them, according to the scenario-based original dataset, the input method of the user request is parsed, judged as a structured form, natural language text, or composite instruction type, and the corresponding type label is labeled to generate input structure label data, which specifically includes: In actual digital employee robot systems, the input methods used by users when interacting with robots differ significantly, directly affecting subsequent information processing and model judgment. To achieve efficient data structuring and subsequent semantic processing, the system first calls the input parsing engine for each user request in the scenario-based original dataset to identify the underlying data format of the request. Specifically, the system analyzes the fields and content form of the user request data: if the input content is a system-provided option selection, single or multiple selection, drop-down list, table filling, etc., and the data is composed of explicit structure fields, the request is determined to be of the structured form type; if the input is a coherent natural language sentence, question, or statement (such as the user directly inputting "Please help me check this month's salary"), it is determined to be of the natural language text type; if a request contains both structured fields and natural language expressions, or has multiple instruction logic associations (such as "first reimbursement, then query the approval progress"), it is identified as a composite instruction type.

[0037] For each type, the system automatically assigns and labels the corresponding input type label, which can be "form", "text", or "composite instruction". After labeling, the system binds the input type label to the request data in a structured manner to form the input structure label data. This data not only facilitates subsequent automatic classification and semantic analysis, but also provides support for multi-input type fusion processing and intelligent adaptation. By combining existing text classification and form recognition technologies, this process can operate stably in a large-scale concurrent business environment.

[0038] Among them, according to the sub-scene original data set, the time stamp associated with each request record is sampled and aligned, the stage information in the business process is extracted, and the sequence time label is generated combined with the previous request content, which specifically includes: In order to ensure the relevance of the robot processing history record and the business context of the current request, the system first extracts the original timestamp information of each request record in the sub-scene original data set. For different business scenarios, the system can set a unified time sampling interval, such as every 10 seconds, every minute, etc. All request times are mapped to this sampling interval. In this way, the time error caused by different terminal devices and network delays can be effectively avoided, and the timelines of all requests can be compared in the global range.

[0039] Next, the system uses the business process management module to locate each request to the stage of the corresponding business process according to its time point and business logic sequence. For example, in the approval process, a request can be marked as "initiation", "in review" or "completed" according to the request time and process template.

[0040] Based on the generation of stage labels, the system further analyzes the business logic relationship between each request and the content of the previous request. By comparing the intent keywords and input types of the previous and subsequent requests, it can be identified whether the user's operation belongs to the same business chain, whether there is a business link jump or process switching. Finally, the system generates a comprehensive sequence time label for each request record, which not only contains absolute time attribution, but also contains process stage and context logic position and other multi-dimensional information. This method fully draws on existing business process modeling and event sequence label management technology to ensure that the label structure is stable and easy to analyze retrospectively.

[0041] Among them, the intent label data, input structure label data and sequence time label are fused to form a complete scene interaction feature data set, which specifically includes: After completing the intent recognition, input method classification and time flow positioning of each user request, the system automatically performs a multi-feature fusion process. The specific steps are as follows: the system synchronously retrieves and logically matches three types of data, namely intent label data (i.e. labels reflecting the user's current demand or operation purpose), input structure label data (i.e. labels indicating the input method used by the request), and sequence time label (i.e. labels reflecting the business process stage and context sequence).

[0042] The system newly creates a structured data object for each user request, and assigns the three types of labels to the corresponding attribute fields of the object. Subsequently, the system sorts and collects all request data objects according to the session identifier and time sequence, and outputs a complete scene interaction feature dataset in batches. In this way, each dataset not only comprehensively covers key features such as interaction content, operation mode and business time sequence, but also lays a solid data foundation for subsequent data mining, anomaly detection and performance evaluation processes.

[0043] The fusion method fully utilizes existing data normalization and feature splicing techniques to ensure consistency and scalability of the fusion results. In this way, subsequent intelligent analysis algorithms can carry out quantitative and qualitative analysis based on unified and complete datasets, thereby greatly improving the efficiency and accuracy of digital employee robot performance evaluation and optimization.

[0044] In a preferred embodiment of the present application, the scene interaction feature data is subjected to fluctuation detection, the time period of abnormal fluctuation is identified, and the user request content and robot response content in the time period are extracted to generate abnormal segmented data, including: According to the scene interaction feature data, a continuous time window sequence is constructed, the feature values in each time window are subjected to statistical distribution analysis, and the change amplitude between adjacent windows is calculated to obtain a fluctuation evaluation parameter; According to the fluctuation evaluation parameter, the time windows with a change amplitude exceeding a preset amplitude threshold are marked, and the user request and robot response content in the time period are extracted to form a candidate abnormal segment; The semantic matching degree between each request and response content in the candidate abnormal segment is analyzed, and by comparing the semantic matching degree of each request and response content with a preset fluctuation interval threshold, the request and response content pair with a semantic matching degree exceeding the preset fluctuation interval threshold is taken as a response deviation segment to generate a response deviation segment; The response deviation segment is sorted according to the session time sequence to construct abnormal segmented data that maintains context continuity.

[0045] In the embodiment of the present application, the system carries out high-precision fluctuation detection and abnormal segment generation on the scene interaction feature data. First, the system constructs an ordered continuous time window sequence from all scene interaction feature data according to a preset time window length. For example, the system can set every five minutes or every hour as a window, and collect all user requests and robot response content during the period into the respective windows. Subsequently, the system performs statistical analysis on the feature values (such as the number of requests, the frequency of specific keywords, response time delay, success rate, etc.) in each window, and calculates the feature change amplitude between adjacent windows to obtain a fluctuation evaluation parameter for each time window. Such statistical processing facilitates the timely capture of sudden changes such as business peaks and fault outbreaks.

[0046] When the system monitors that the fluctuation evaluation parameter of a certain window exceeds the pre-set fluctuation threshold, the time window and the user request and robot response content within it are automatically marked as a candidate abnormal segment. This processing method can effectively narrow down the analysis target to the key time period with significant fluctuations, improving the efficiency of subsequent abnormality discrimination.

[0047] For each candidate abnormal segment, the system further analyzes the semantic matching degree between the user request and the robot response content in the segment. Specifically, the system uses a semantic analysis engine to convert the features of each pair of request and response content and quantify the semantic consistency to obtain a set of semantic matching degree data. Then, the system compares each set of semantic matching degree with the set fluctuation interval threshold in the system. If the semantic matching degree of a pair of request and response content exceeds the threshold, it is identified as semantic deviation. All pairs of request and response content identified in this way will be automatically classified by the system as response deviation segments.

[0048] Finally, the system arranges all response deviation segments according to the time sequence of the actual conversation to ensure the continuity of the context of the previous and subsequent conversations, and outputs the abnormal segment data in a structured data form. Through this method, the most valuable abnormal segments can be efficiently and automatically screened, greatly improving the intelligent level of digital employee robot problem positioning and fault tracing, and laying a solid foundation for subsequent abnormal path restoration and cause analysis.

[0049] Among them, the response deviation segments are arranged according to the conversation time sequence to construct abnormal segment data that maintains the continuity of the context, specifically including: To ensure that the abnormal analysis results have business context interpretability, the system will sort and arrange the response deviation segments according to the time sequence of the original conversation after identifying them. In specific operation, the system assigns each response deviation segment its absolute timestamp and relative sequence position in the overall conversation process. Then, the system combines the conversation identifier to which each segment belongs to concatenate all response deviation segments in the same conversation, preserving the user request and robot response content associated before and after them, so that the abnormal segment not only marks the abnormality independently, but also reflects the stage and context it is in in the business conversation.

[0050] This arrangement helps to restore the business process before and after the anomaly, user intention change and system response chain, and facilitate subsequent problem tracing, cause attribution and operation and maintenance decision. Through structured organization and continuity maintenance, the system can output abnormal segmentation data with complete conversation context and abnormal logic chain, providing real and traceable data basis for evaluation report generation and subsequent model optimization. This method combines existing conversation management, time sorting and context merging technology, and realizes efficient and interpretable output of abnormal data in large-scale conversation scenarios.

[0051] In a preferred embodiment of the present application, for abnormal segmentation data, the current conversation path is restored based on the preset business process template, and the path consistency is compared in combination with the response rules of the robot in the path. If the path is interrupted or the response is deviated, an abnormal node set is generated, including: According to the time sequence and request content of the user request in the abnormal segmentation data, the task keywords are extracted, and the conversation task node sequence is constructed according to the process node information corresponding to the task keywords; Based on the preset business process template, the standard task execution sequence in the same application scenario as the conversation task node sequence is obtained, and the conversation task node sequence and the standard task execution sequence are compared node by node to identify the breakpoint, jump point or repeated node in the node sequence, and generate a node comparison result; According to the node comparison result, the robot response content at the breakpoint, jump point or repeated node is matched and verified with the standard response rule. When there is an inconsistent response content and node rule, an answer deviation, the abnormal node and its corresponding user request content, robot response content and external data calling state are encapsulated as abnormal node data, and an abnormal node set is constructed.

[0052] In the embodiment of the present application, after obtaining the abnormal segmentation data, the system first extracts the task keywords that can reflect the actual business target according to the time sequence and request content of each user request. Then, the system restores the user interaction sequence to a complete conversation task node sequence according to the node definition of each task keyword in the preset business process template. For example, in the enterprise financial approval scenario, the user initiates "submit reimbursement" and then initiates "approval progress query". These operations can be restored to the corresponding nodes in the process template.

[0053] On the basis of the restored conversational task node sequence, the system calls the business process template to obtain the standard task execution sequence in the current scenario, and compares it with the user real interaction node sequence node by node. During the comparison process, the system can accurately identify the breakpoints (such as a step not being triggered), the jump points (such as an abnormal user operation sequence), or the repeated nodes (such as the same request being initiated multiple times) in the process nodes. The comparison result can not only show the structural differences between the actual process and the standard process, but also output the specific location of each abnormal situation.

[0054] Then, the system automatically extracts the response content of the robot at the identified breakpoints, jump points, or repeated nodes, and one-by-one matches and verifies it with the standard response rules. For example, if the response content does not cover the node target, or the reply content does not match the process state, the system will mark the node as response deviation, and encapsulate the node and its corresponding user request content, robot response content, and external data calling state together to construct a structured abnormal node data. All abnormal node data are collected by the system into an abnormal node set, which contains all the interaction fragments that are determined to be process structure abnormal and response deviation.

[0055] Through the above implementation, the system can automatically restore multiple rounds of complex conversations to process nodes in a structured manner, accurately identify process abnormalities and response abnormalities, and efficiently organize a complete context of the abnormal node set. This not only improves the accuracy of problem positioning, but also provides detailed evidence for subsequent abnormal cause analysis and performance optimization. Practice shows that in the real digital employee robot operation scenario, the present embodiment can significantly improve the accuracy of abnormal identification and attribution in multiple scenarios, reduce the workload of manual review and problem tracing, and provide a reliable data foundation for continuous optimization of robot business processes.

[0056] Among them, according to the time sequence and request content of the user request in the abnormal segment data, the task keywords are extracted, and according to the process node information corresponding to the task keywords, the conversational task node sequence is constructed, specifically including: In the digital employee robot performance intelligent evaluation system, the system first arranges the original data in order according to the time sequence of each user request in the abnormal segment data. For each user request, the system calls the keyword extraction module to automatically analyze the core verbs, nouns, or proper nouns in the request text to determine the keywords that can reflect the business target or operation intent. These keywords are the task keywords. For example, “submit reimbursement form”, “query balance”, “upload invoice” and the like can be used as task keywords in the business process.

[0057] After extracting the keywords of each task, the system matches them with the pre-set business process node information. The process node information is generally stored in the form of process number, node description, etc., and each task keyword is associated with one or more process nodes. The system assigns the corresponding node number and description to the current user request according to the matching result, and concatenates all nodes in the time sequence of the request to form a complete session task node sequence. Through this sequence, the user's operation path in a business handling process can be truly restored, which helps subsequent process consistency comparison and abnormal node positioning. This method combines existing text mining and business process modeling techniques to achieve efficient mapping from unstructured user requests to structured business nodes.

[0058] The pre-set business process template specifically includes: When evaluating the performance of digital employees, the system usually establishes a standardized business process template in advance for different business scenarios and application requirements. The pre-set business process template is a structured and standardized expression of the actual business operation path, usually consisting of a set of ordered process nodes, each corresponding to a specific business operation, state or checkpoint, such as "user authentication", "information filling", "approval", "process end", etc.

[0059] The process template is usually stored in the system in the form of process description files, database structures or flowcharts, etc., and there is a strict relationship between nodes and jump conditions. For example, only after completing the "submit application" node, the process can enter the "approval" node. The template can also contain complex control logic such as branching, looping, exception handling, etc., to adapt to all changes in the actual business process.

[0060] When evaluating the actual dialogue path, the system compares the user's actual operation nodes with the standard nodes in the template one by one to determine whether there are deviations, omissions or repetitions. By introducing the business process template, the evaluation and analysis can ensure consistency with industry standards and business specifications, providing a reliable basis for robot behavior monitoring, anomaly identification and continuous optimization. This solution combines existing process modeling and business process management (BPM) technology, facilitating seamless integration with the enterprise's original information system.

[0061] According to the node comparison result, the robot response content at the interruption point, jump point or repeated node is matched and verified with the standard response rules, specifically including: In the business process consistency analysis, the system compares the session task node sequence with the standard task execution sequence of the pre-set process template at the node level. If a node is found to be out of order (interruption point), skip some steps and directly enter the subsequent link (jump point), or a node is repeatedly triggered (repeated node), the system determines that the process has structural abnormalities.

[0062] For these abnormal nodes, the system further extracts the corresponding robot response content and calls the standard response rule library for matching verification. In the standard response rule library, the text features, business prompts or data operation requirements of the expected robot response are defined for each process node. The system compares the actual response content with the standard content in the rule library to evaluate the consistency of the two in text expression, data accuracy and business operation. For example, if the reply received by the user at the "submit application" node does not contain "application has been submitted, please wait for approval" type confirmation information, or the reply content contains business terminology confusion, data omission, etc., it is determined that the response is deviated.

[0063] The system structurally records all the abnormal response contents, deviation reasons and node information found by the system, providing high-quality basic data for subsequent abnormal node set packaging and cause attribution analysis. This method combines text similarity matching, business rule verification and multi-link automatic verification technology, effectively ensuring the objectivity and operability of the evaluation conclusion.

[0064] In a preferred embodiment of the present application, according to the scene-specific original data set, the semantic segmentation and label mapping are performed on each user request record, the user intent keywords are extracted, and they are corresponded to the preset intent classification system to generate intent labeled data, including: According to the scene-specific original data set, the user request record content is subjected to morphological segmentation and semantic phrase extraction to obtain a candidate keyword group containing a verb and an object structure; According to the candidate keyword group, the semantic similarity value of each standard label in the preset intent classification system is calculated, and the label that exceeds the preset semantic similarity threshold is identified to form a preliminary intent mapping result; According to the preliminary intent mapping result, the context semantics of the current request and adjacent historical requests are combined for sequence consistency analysis and label weight adjustment to obtain a final intent label; The final intent label is bound with the current user identifier to generate structured intent labeled data.

[0065] In the embodiment of the present application, the system performs multi-level mining on the semantic features of the user request record, improving the accuracy of robot intent recognition and subsequent analysis. First, the system uses word segmentation and phrase extraction algorithm to finely segment the text content for each user request record from the scene-specific original data set. During the processing, the key word group containing the grammar core structure such as verb and object can be accurately identified. For example, for the request "I want to apply for annual leave", the system will automatically extract "apply" and "annual leave" as candidate intent keyword groups.

[0066] Subsequently, the system calculates the semantic similarity value between the candidate keyword group and each standard label in the preset intent classification system. Using vector space analysis, synonym expansion and other semantic analysis mechanisms, the system can determine the degree of correlation between the user's natural expression and the standard business intent. Only when the semantic similarity value exceeds the threshold set by the system, the standard label will be determined as a hit intent label by the system, thereby generating a preliminary intent mapping result. This approach not only ensures the accuracy of intent extraction, but also avoids misjudgment due to differences in expression methods.

[0067] After generating the preliminary intent mapping result, the system further combines the semantic association between the current request and the context history request to analyze the sequence consistency of the intent label and adjust the weight. For example, during a multi-round conversation, the user may issue related requests in sequence, such as first "query balance" and then "transfer". The system can automatically identify the temporal association between the two and optimize the final intent label assignment accordingly. The final intent label is bound to the user's identity marker to form structured intent marker data, facilitating the extraction of subsequent task nodes and response deviation judgment.

[0068] This embodiment ensures the accuracy and context consistency of intent features through multiple processing steps, not only improving the understanding ability of the robot, but also greatly enhancing the data basis of subsequent abnormal analysis and evaluation processes. In practical applications, this method can effectively improve the intent recognition success rate in multiple scenarios and multi-round conversations, providing technical support for performance optimization of digital employee robots in complex business scenarios.

[0069] Among them, according to the scene-based original data set, the user request record content is subjected to lexical segmentation and semantic phrase extraction to obtain candidate keyword groups containing verb and object structures, specifically including: When performing intent recognition on digital employee robot interaction data, the system first calls the word segmentation and phrase recognition tool to perform lexical segmentation on the text content for each user request record in the scene-based original data set. This process can be based on existing natural language processing techniques, such as dictionary-based and statistical model-based word segmentation algorithms, to disassemble continuous text into basic words or word groups. The system further analyzes the syntactic structure of the text to automatically extract verbs and their corresponding objects, and identifies and combines them into verb-object structures. For example, for the request "I want to query the salary details", the system disassembles it into "query-salary details" as a candidate keyword group. If the request contains multiple verb-object pairs, the system extracts them all.

[0070] The generation of candidate keyword groups helps to accurately classify user intent and lays the foundation for business process recognition in complex multi-intent scenarios. This processing flow combines existing word segmentation, phrase recognition and syntactic analysis techniques to ensure high adaptability and universality for various natural language expression methods.

[0071] Among them, based on the candidate keyword group, the semantic similarity value between it and each standard label in the preset intent classification system is calculated, and labels exceeding the preset semantic similarity threshold are identified to form a preliminary intent mapping result, which specifically includes: After obtaining candidate keyword phrases, the system invokes the intent matching engine to compare each verb-object structure against standard labels in a pre-defined intent classification system. This intent classification system consists of intent labels related to business scenarios, such as "check balance," "submit application," and "modify information." The system uses a semantic similarity calculation method to compare the candidate keyword phrases with each standard label. Common methods include cosine similarity in word vector space, synonym expansion, or contextual relevance scoring to quantify the comparison results into similarity values.

[0072] For each set of comparison results, if the semantic similarity value reaches or exceeds the system's preset threshold, the candidate keyword group is determined to have a high correlation with the standard intent tag and is included as the preliminary intent mapping result. If multiple tags meet the threshold at the same time, the system will include all relevant tags in the candidate range.

[0073] This process effectively avoids missing intents due to diverse expressions, ensuring both coverage and accuracy of intent mapping. This method combines existing text similarity calculation and semantic understanding technologies, making it suitable for a variety of natural language scenarios.

[0074] Among them, based on the preliminary intent mapping results, combined with the contextual semantics of the user's current request and adjacent historical requests, sequence consistency analysis and label weight adjustment are performed to obtain the final intent label, which specifically includes: After obtaining preliminary intent mapping results, the system not only considers the content of the current request but also leverages the contextual information generated by the continuous interaction between the user and the bot. The system first extracts the current request and several previous requests before and after it, summarizing and sorting the intent tags associated with all requests. By analyzing the logical order and semantic relevance of these tags within the business process, the system determines whether the current request and previous requests belong to the same session, business chain, or task process.

[0075] If a tag appears repeatedly in historical requests or is highly relevant to the current business phase, the system automatically increases its weight. Conversely, if a tag appears only in isolation within a context, its weight is reduced. The system optimizes and filters the preliminary set of intent tags by combining tag weights with the priorities of business process nodes, ultimately outputting the intent tag that best matches the current business context as the final recognition result.

[0076] The sequence consistency analysis and tag weight adjustment process combines existing conversation management, context modeling and tag weighting filtering technology, and realizes efficient restoration of the real intention of the user in a complex dialogue scene. Through this method, the recognition accuracy of the robot for user demand in a multi-round, multi-intention, and multi-business cross-scene can be effectively improved.

[0077] In a preferred embodiment of the present application, the semantic matching degree between each request and response content in the candidate abnormal segment is analyzed piece by piece, and by comparing the semantic matching degree of each request and response content with the preset fluctuation interval threshold, the request and response content pair with a semantic matching degree exceeding the preset fluctuation interval threshold is taken as the response offset segment, and the response offset segment is generated, including: The text content of each request and response content in the candidate abnormal segment is extracted, a semantic vector is generated through a semantic feature conversion method, and a corresponding request vector and response vector group are constructed; The semantic distance between each request vector and its response vector is calculated, and the difference is compared with the preset fluctuation interval threshold to obtain a semantic offset value; According to the semantic offset value, the request vector and response vector group exceeding the preset offset threshold are identified, and similarity clustering processing is performed thereon to extract a response drift segment satisfying a preset clustering similarity threshold; The response drift segment satisfying the preset clustering similarity threshold is arranged in chronological order, and a response offset segment dataset with coherent context is generated in combination with the corresponding user interaction record.

[0078] In the embodiment of the present application, the system performs fine semantic matching analysis and offset segment clustering on the user request and robot response content in the candidate abnormal segment. Specifically, the system first extracts each pair of user request and robot response text in each candidate abnormal segment, and converts them into semantic vectors by means of a semantic feature conversion tool to construct a request vector and response vector group. This processing ensures that requests and responses in different forms of expression can be uniformly quantitatively described.

[0079] For each request vector and response vector group, the system applies a semantic distance calculation method to obtain the semantic distance value between them. Then, the system compares these distance values with the preset fluctuation interval threshold to calculate the semantic offset value of each pair of content. Any request and response content pair with a semantic offset value exceeding the system preset threshold will be determined as a possible abnormal response behavior.

[0080] The system further performs similarity clustering analysis on all request and response vector groups determined to be abnormal. By comparing the semantic similarity between these abnormal groups, the system can classify content with a similarity reaching a clustering threshold into the same response drift segment. This way, not only can typical abnormal response patterns be automatically identified, but sporadic and isolated abnormalities can also be effectively filtered out, improving the accuracy and generalization ability of the analysis.

[0081] After clustering is complete, the system sorts the response drift segments that meet the clustering similarity threshold in chronological order and combines them with their original user interaction records to form a response drift segment dataset with complete context. This dataset can provide strong support for business personnel, operation engineers, or algorithm models and play a key role in subsequent abnormal attribution, optimization decision-making, and continuous iteration. Through this embodiment, the system exhibits strong abnormal identification and automatic classification capabilities in a high-concurrency and complex business environment, improving the overall intelligent level of digital employee robots.

[0082] Among them, the semantic distance between each group of request vectors and their response vectors is calculated, and the difference is compared with the preset fluctuation interval threshold to obtain the semantic offset value, specifically including: In the implementation process of this method, the system first converts the text semantic features of each group of user request content and corresponding robot response content in the candidate abnormal segment. By calling a semantic embedding engine or a pre-trained language model, the text information is converted into a high-dimensional vector representation. For each request content and its corresponding response content, the system establishes a pair of vector groups in the same semantic space.

[0083] The system then calculates the semantic distance between each group of request vectors and response vectors using distance measurement algorithms such as Euclidean distance, inverse cosine similarity, or Manhattan distance. This semantic distance reflects the difference in semantic expression between user requests and robot responses. Each distance value calculated is considered as the semantic distance score of the current request-response pair.

[0084] The system compares the above semantic distance score with the system's preset fluctuation interval threshold. This threshold is a reference standard set manually or adjusted dynamically by analyzing the semantic distance distribution of normal requests and responses in historical data. If the semantic distance of a pair exceeds this threshold, it is considered that there is a semantic offset between them, and the system calculates the semantic offset value, which is the quantitative indicator of the abnormality of the pair. This method relies on existing technical means such as natural language processing and clustering analysis and can perform batch and automated abnormal detection on a large number of unstructured text interactions.

[0085] According to the semantic offset value, the request vector and the response vector group exceeding the preset offset threshold are identified, similarity clustering processing is performed, and a response drift segment meeting a preset clustering similarity threshold is extracted, specifically including: In the subsequent processing link of the method, the system screens all vector groups exceeding the preset offset threshold of the system based on the semantic offset values of all the request-response pairs calculated in the foregoing, and these groups are considered to have potential abnormal response behaviors. The system further performs similarity clustering analysis on the request vector and the response vector group determined to be abnormal.

[0086] In the clustering processing, the system compares the semantic features of all the abnormal vector groups with each other, analyzes their distance or similarity relationship in the vector space, and classifies abnormal groups with highly consistent semantic expressions or similar behavior patterns into the same cluster. The system sets a clustering similarity threshold to ensure that only abnormal segments with sufficiently similar semantic contents can be classified into the same category. For each cluster category, the system selects a representative response content segment as a typical response drift segment for output.

[0087] Through this clustering method, the system can not only effectively filter occasional and isolated offset points, but also automatically discover batch and patterned response abnormalities, providing high-value data samples for subsequent operation tracing, business optimization and model fine-tuning. The method widely draws on existing mature technologies in the fields of natural language processing and unsupervised clustering, and has engineering feasibility and application value in large-scale data scenarios.

[0088] In a preferred embodiment of the present application, according to the abnormal node set, the user request format, the robot response behavior and the external data calling state corresponding to each abnormal node are extracted for abnormal source analysis, the attribution probability of user input abnormality, identification abnormality and data interface abnormality is calculated, and an abnormal cause analysis result is generated, including: According to the abnormal node set, each abnormal node is extracted and the following processing is performed: The user request record corresponding to each abnormal node is extracted, the structural length, semantic organization form and input timing stability of the request text are feature coded, and an input feature vector is generated; The response content made by the robot in each abnormal node is extracted, the semantic correspondence degree between the response content and the flow node and the response delay time are calculated, and a response feature vector is generated; The data interface calling information triggered by each abnormal node is extracted, including the interface calling return state, whether the response data is empty, and the interface calling delay interval, and a data interface feature vector is generated; The input feature vector, the response feature vector and the data interface feature vector are combined to form a unified attribution feature set; inputting the attribution feature set into a preset attribution processing model, and calculating attribution probability values of attribution probability distributions of user input abnormality, recognition abnormality and data interface abnormality through multi-dimensional feature proportion calculation; taking the abnormal type with the maximum attribution probability value as a judgment result, and generating abnormal cause analysis data containing an abnormal type label.

[0089] In the embodiment of the present application, the system further carries out multi-dimensional feature extraction and attribution analysis for each abnormal node data identified by the process analysis, thereby distinguishing different types of abnormal causes in detail. The system first carries out text structure length statistics, semantic organization form analysis and input timing stability analysis on the user request records contained in the abnormal node data. Based on these analysis results, the system encodes the user input content into an input feature vector, so that various complex expressions, grammar variations and other factors can be quantitatively expressed.

[0090] Meanwhile, the system extracts and analyzes the robot response content in each abnormal node, including response text and expected business process node semantic consistency calculation, response delay measurement and the like, and further generates a response feature vector. This process can reflect whether the robot has problems such as understanding obstacles, delayed responses or output content out of line with the business process in actual operation.

[0091] The system also extracts information of each external data interface call from the abnormal node data, including interface return status code, response data null or not, call delay time and the like, and uniformly encodes them into a data interface feature vector. In this way, whether due to external service fluctuation, interface abnormality or data loss, it can be quantitatively included in subsequent attribution analysis.

[0092] All the above feature vectors will be automatically combined into a unified attribution feature set by the system, which is input into the attribution processing model. The attribution model based on the weighted calculation method of multi-factor features comprehensively evaluates the probability distribution of the three types of causes of user input abnormality, robot recognition abnormality and data interface abnormality. The system automatically determines the abnormal type with the maximum attribution probability, and generates abnormal cause analysis data with an abnormal type label based on it. Through this implementation, the system can accurately and transparently classify the causes of each abnormal node, and provide high-credibility quantitative analysis results for subsequent performance evaluation and problem positioning. In practical application, this scheme greatly reduces the subjectivity of manual tracing, improves the consistency and automation level of abnormal attribution, and provides strong technical support for stable and efficient operation of digital employee robots.

[0093] Among them, the user request records corresponding to each abnormal node are extracted, the structure length, semantic organization form and input timing stability of the request text are encoded for feature, and an input feature vector is generated, which specifically includes: In the process of digital employee robot performance evaluation, for each identified abnormal node, the system first obtains its corresponding user request record. The system analyzes the structure length of the text, including the number of characters, the number of words or the number of sentences in the text, and then quantifies the complexity of the user request expression. Subsequently, the system analyzes the semantic organization form of the text content, judges whether the request text uses standard sentence, declarative sentence, interrogative sentence, list and other expression methods, and whether there are complex structures such as multi-intention and multi-clause.

[0094] In addition, the system analyzes the stability of the user request record in time, that is, checks whether the user repeatedly inputs multiple times in a short time, changes the content quickly, and other phenomena in this session link, to evaluate the consistency of the user input behavior. The system encodes the above various information (structure length, semantic organization, time sequence stability, etc.) into a set of multi-dimensional numerical values or labels to form an input feature vector. The feature vector, as an important basis for input abnormal attribution analysis, can reflect the specific operation behavior and expression characteristics of the user when the abnormal node occurs. This processing combines existing technologies such as text feature extraction, syntax analysis and time series analysis in natural language processing.

[0095] Among them, the response content made by the robot in each abnormal node is extracted, the semantic correspondence degree of the response content and the flow node and the response delay time are calculated, and the response feature vector is generated, which specifically includes: After obtaining the user request record of the abnormal node, the system synchronously extracts the response content made by the robot at the node. The system performs semantic analysis on the response text and compares it with the standard response content of the flow node to which the node belongs, to evaluate the semantic consistency between the actual response and the business expectation. This semantic consistency is usually realized through semantic similarity algorithm, which can be quantified based on word vector, phrase embedding or context analysis.

[0096] The system also records the time consumed by the robot from receiving the user request to actually generating the response, and takes the response delay time as an important indicator reflecting the timeliness of the system. The semantic correspondence degree and the response delay time are integrated into the response feature vector, which comprehensively describes the response behavior of the robot in the abnormal node. These features provide data support for determining response deviation, time delay abnormality and other problems. This method combines existing technologies such as business process modeling, response delay monitoring and semantic consistency discrimination, and is suitable for automated attribution analysis in complex business environment.

[0097] Among them, the data interface call information triggered by each abnormal node is extracted, including interface call return status, whether the response data is empty, and interface call delay interval, to generate a data interface feature vector, which specifically includes: In the digital employee robot performance evaluation process of the application, for each identified abnormal node, the system automatically analyzes whether the business process corresponding to the node involves a call operation to an external data interface. If there is an interface call, the system first extracts the return state information of the interface call. Specifically, the system detects the result status code of the interface call, such as HTTP or custom API response code, and judges whether the interface call is successful, has an error or an abnormal termination according to the result status code. This state information reflects the reliability and availability of the interface and is an important basis for identifying data interface abnormalities.

[0098] Subsequently, the system further analyzes the data content returned by the interface, focusing on whether the returned data is empty or missing. If the interface returns a null data structure, an empty array, an empty object, or the main fields in the returned content are missing, the system will record this as "response data is empty". This judgment helps to identify the abnormal situation that the interface returns successfully in form but the content is invalid or incomplete, and facilitates further positioning of data source problems or business link breakage.

[0099] At the same time, the system also monitors the time performance during the interface call. For each interface call, the system automatically records the actual time consumption from initiating the request to receiving the response. By comparing the current time consumption with the set normal response time interval, the system can determine whether the interface call has performance bottlenecks such as timeout and delay. For example, if the response time of an interface is significantly higher than the historical average or the expected upper limit, the system will mark this delay interval as an abnormal feature.

[0100] Finally, the system quantitatively encodes the above interface call return state, response data emptiness and interface call delay interval, and combines them into a multi-dimensional data interface feature vector. Each feature dimension corresponds to interface state, data validity and call timeliness, etc.

[0101] The method for constructing the preset attribution processing model comprises the following steps: Based on historical attribution sample data, the input feature vector, response feature vector and data interface feature vector of each labeled abnormal node are obtained, all feature data are normalized to generate a unified attribution training sample set; For each type of abnormality, the typical value range of the input feature, response feature and data interface feature in the attribution training sample set under each abnormal type is respectively counted, and the probability distribution of each feature under different abnormal types is calculated; An initial weight parameter is assigned to each feature, and the weight parameter is iteratively adjusted according to the distinguishing ability of the feature to the abnormal type in the attribution training sample to improve the weight of the feature with strong distinguishing ability; For each abnormal type, a corresponding discrimination threshold is set, which is set according to the comprehensive attribution score distribution of the abnormal type in the training sample, and is used for subsequent abnormal discrimination; The above probability distribution, feature weight and discrimination threshold form an attribution score rule, and are solidified into an attribution processing model, which is called in actual abnormal node attribution analysis; In the actual attribution analysis process, the input feature vector, response feature vector and data interface feature vector of the abnormal node to be attributed are input, which are compared with the feature probability distribution of each abnormal type in the model, and the comprehensive attribution score of each abnormal type is calculated according to the comparison result and the feature weight in a weighted cumulative manner, and then compared with the discrimination threshold of each abnormal type, and the abnormal type with the attribution score exceeding the discrimination threshold is output as the attribution result of the node.

[0102] Embodiments of the present application also provide a digital employee robot performance intelligent evaluation system, which comprises: A log classification module is configured to obtain running log data of the robot in a specified application scenario, and classify the log data according to time periods and business categories to obtain a scene-specific original data set. An interaction feature extraction module is configured to extract features of user interaction content in each scene according to the scene-specific original data set, identify keywords, input methods and time labels in user requests, and generate scene interaction feature data. A fluctuation detection module is configured to detect fluctuations in the scene interaction feature data, identify time periods of abnormal fluctuations, and extract user request content and robot response content in the time periods to generate abnormal segment data. A path consistency analysis module is configured to restore a path based on a preset business process template, and perform consistency comparison based on response rules of the robot in the path, and if a path interruption or response deviation is detected, an abnormal node set is generated. A cause attribution module is configured to extract user request formats, robot response behaviors and external data calling states corresponding to each abnormal node in the abnormal node set, perform abnormal source analysis, calculate attribution probabilities of user input abnormalities, identification abnormalities and data interface abnormalities, and generate abnormal cause analysis results. A performance evaluation module is configured to induce main performance bottlenecks and optimization directions in each scene based on the abnormal cause analysis results, and generate a comprehensive report containing scene-specific performance evaluation conclusions and cause analysis.

[0103] It should be noted that the system corresponds to the above method, and all implementation manners in the above method embodiments are applicable to this embodiment and can achieve the same technical effects.

[0104] The embodiment of the present application also provides a computing device, comprising a processor, a memory storing a computer program, the computer program being executed by the processor to perform the method as described above. All implementation manners in the above method embodiment are suitable for this embodiment and can achieve the same technical effects.

[0105] The embodiment of the present application also provides a computer readable storage medium storing instructions, which, when executed on a computer, cause the computer to perform the method as described above. All implementation manners in the above method embodiment are suitable for this embodiment and can achieve the same technical effects.

[0106] The above is the preferred embodiment of the present application, it should be pointed out that, for those skilled in the art, without departing from the principles of the present application, can make a number of improvements and refinements, these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A method for intelligent performance evaluation of a digital employee robot, characterized in that: The method comprises: Obtain the robot's operating log data in a specified application scenario and classify it by time period and business category to obtain the original data set for each scenario; Based on the original scene-by-scene data set, feature extraction is performed on the user interaction content in each scene, including identifying keywords, input methods, and time tags in user requests to generate scene interaction feature data; Perform fluctuation detection on scene interaction feature data, identify time periods with abnormal fluctuations, extract user request content and robot response content within that time period, and generate abnormal segmented data; For abnormal segmented data, the current session path is restored based on the preset business process template, and the path consistency is compared in combination with the robot's response rules under the path. If the path is interrupted or the response is offset, an abnormal node set is generated; Based on the set of abnormal nodes, the user request format, robot response behavior, and external data call status corresponding to each abnormal node are extracted to analyze the source of the abnormality. The attribution probability of user input abnormalities, recognition abnormalities, and data interface abnormalities is calculated to generate the abnormality cause analysis results. Based on the results of the abnormality cause analysis, the main performance bottlenecks and optimization directions in each scenario are summarized, and a comprehensive report containing the performance evaluation conclusions and cause analysis for each scenario is generated.

2. A digital employee robot performance intelligent evaluation method according to claim 1, characterized in that: Based on the original scene-by-scene data set, feature extraction is performed on the user interaction content in each scene, including identifying keywords, input methods, and time tags in user requests, and generating scene interaction feature data, including: Based on the original scenario-based data sets, semantic segmentation and label mapping are performed on each user request record to extract user intent keywords. These keywords are then mapped to the preset intent classification system to generate intent tag data. Based on the original scenario data set, the user's input method is parsed to determine whether it is a structured form, natural language text, or compound instruction type, and the corresponding type label is annotated to generate input structure label data; Based on the original scenario-specific dataset, we sample and align the timestamps associated with each request record, extract information about its stage in the business process, and combine it with the previous request content to generate a sequence time label. The intent tag data, input structure tag data and sequence time series labels are integrated to form a complete scene interaction feature dataset.

3. A digital employee robot performance intelligent evaluation method according to claim 1, characterized in that: Perform fluctuation detection on scene interaction feature data, identify time periods with abnormal fluctuations, extract user request content and robot response content within that time period, and generate abnormal segmented data, including: A continuous time window sequence is constructed based on the scene interaction feature data. The characteristic values ​​in each time window are statistically analyzed for distribution, and the variation between adjacent windows is calculated to obtain the fluctuation assessment parameter. Based on the fluctuation assessment parameters, the time window where the change amplitude exceeds the preset amplitude threshold is marked, and the user request and robot response content within this time period are extracted to form candidate abnormal segments; Analyze the semantic matching degree between each request and response content in the candidate abnormal segment one by one, compare the semantic matching degree of each request and response content with the preset fluctuation interval threshold, and take the request and response content pairs whose semantic matching degree exceeds the preset fluctuation interval threshold as the response offset segment, and generate the response offset segment; The response offset fragments are sorted according to the session time sequence to construct abnormal segmented data that maintains context continuity.

4. A digital employee robot performance intelligent evaluation method according to claim 1, characterized in that: For abnormal segmented data, the current session path is restored based on the preset business process template, and the path consistency is compared in combination with the robot's response rules under the path. If the path is interrupted or the response is offset, an abnormal node set is generated, including: Extract task keywords based on the time sequence and request content of user requests in the abnormal segmentation data, and construct a session task node sequence based on the process node information corresponding to the task keywords; Based on the preset business process template, obtain the standard task execution sequence in the same application scenario as the conversation task node sequence, and compare the conversation task node sequence with the standard task execution sequence node by node, identify the node sequence breakpoints, jump points or repeated nodes, and generate the node comparison results; According to the node comparison results, the robot response content at the interruption point, jump point or repeated node is matched and checked with the standard response rules. When there is a response offset where the response content is inconsistent with the node rules, the abnormal node and its corresponding user request content, robot response content and external data call status are encapsulated as abnormal node data, and an abnormal node set is constructed.

5. A digital employee robot performance intelligent evaluation method according to claim 2, characterized in that: Based on the original scenario-specific data sets, each user request record is semantically segmented and labeled, user intent keywords are extracted, and mapped to the preset intent classification system to generate intent tag data, including: Based on the original scenario-specific data sets, the user request records are lexically segmented and semantic phrases are extracted to obtain candidate keyword groups containing verb and object structures. Based on the candidate keyword group, calculate its semantic similarity value with each standard tag in the preset intent classification system, identify tags that exceed the preset semantic similarity threshold, and form a preliminary intent mapping result; Based on the preliminary intent mapping results, the final intent label is obtained by performing sequence consistency analysis and label weight adjustment based on the contextual semantics of the user's current request and adjacent historical requests. Bind the final intent tag with the current user ID to generate structured intent tag data.

6. A digital employee robot performance intelligent evaluation method according to claim 3, characterized in that: The semantic matching degree between each request and response content in the candidate abnormal segment is analyzed one by one. By comparing the semantic matching degree of each request and response content with the preset fluctuation interval threshold, the request and response content pairs whose semantic matching degree exceeds the preset fluctuation interval threshold are regarded as response offset segments, and the response offset segments are generated, including: Extract the text content of each request and response in the candidate abnormal fragment, generate semantic vectors through semantic feature conversion, and construct corresponding request vector and response vector groups; Calculate the semantic distance between each set of request vectors and their response vectors, and compare the difference with the preset fluctuation interval threshold to obtain the semantic offset value; Based on the semantic offset value, the request vector and response vector groups that exceed the preset offset threshold are identified, similarity clustering is performed on them, and response drift segments that meet the preset clustering similarity threshold are extracted; The response drift segments that meet the preset clustering similarity threshold are arranged in chronological order and combined with their corresponding user interaction records to generate a response drift segment dataset with contextual coherence.

7. A digital employee robot performance intelligent evaluation method according to claim 4, characterized in that: Based on the set of abnormal nodes, the user request format, robot response behavior, and external data call status corresponding to each abnormal node are extracted to analyze the source of the abnormality. The attribution probability of user input abnormalities, recognition abnormalities, and data interface abnormalities is calculated to generate abnormality cause analysis results, including: Based on the abnormal node set, extract each abnormal node and perform the following processing: Extract the user request record corresponding to each abnormal node, perform feature encoding on the structural length, semantic organization form, and input temporal stability of the request text, and generate an input feature vector; Extract the response content of the robot in each abnormal node, calculate the semantic correspondence between the response content and the process node and the response delay time, and generate a response feature vector; Extract the data interface call information triggered by each abnormal node, including the interface call return status, whether the response data is empty, and the interface call delay interval, and generate a data interface feature vector; Combining the input feature vector, the response feature vector, and the data interface feature vector to form a unified attribution feature set; Input the attribution feature set into the preset attribution processing model, and calculate the attribution probability value of the attribution probability distribution of user input anomalies, recognition anomalies, and data interface anomalies through the multi-dimensional feature proportion; The anomaly type with the largest attribution probability value is taken as the judgment result, and anomaly cause analysis data including anomaly type labels is generated.

8. A digital employee robot performance intelligent evaluation system, characterized in that: Applied to the method according to any one of claims 1 to 7, the system comprises: The log classification module is used to obtain the robot's operating log data in a specified application scenario and classify it by time period and business category to obtain the original data set by scenario; The interaction feature extraction module is used to extract features of user interaction content in each scenario based on the original scene data set, identify keywords, input methods and time tags in user requests, and generate scene interaction feature data; The fluctuation detection module is used to detect fluctuations in scene interaction feature data, identify time periods with abnormal fluctuations, extract user request content and robot response content within this time period, and generate abnormal segmented data; The path consistency analysis module is used to restore the path of abnormal segmented data based on the preset business process template and perform consistency comparison based on the robot's response rules under the path. If a path interruption or response deviation is detected, an abnormal node set is generated; The cause attribution module is used to extract the user request format, robot response behavior, and external data call status corresponding to each abnormal node based on the abnormal node set, perform abnormal source analysis, calculate the attribution probability of user input abnormalities, recognition abnormalities, and data interface abnormalities, and generate abnormal cause analysis results; The performance evaluation module is used to summarize the main performance bottlenecks and optimization directions in each scenario based on the results of the abnormal cause analysis, and generate a comprehensive report that includes the performance evaluation conclusions and cause analysis for each scenario.

9. A computing device, characterized in that include: one or more processors; A storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images