Voice call scoring method and device for controller simulation training, equipment and medium
By acquiring speech recognition result data through a simulation training system, calculating the matching degree between the speech recognition text and the corpus instructions, and the instruction execution status, the comprehensive scoring method solves the problems of low evaluation efficiency and insufficient objectivity in controller simulation training, and realizes intelligent and objective evaluation.
Patent Information
- Application Number
- CN202511867815.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-11
- Publication Date
- 2026-02-10
AI Technical Summary
In existing technologies, voice call evaluation during controller simulation training is inefficient and the scoring results are not objective enough.
The speech recognition results data are obtained through a simulation training system. The matching degree between the speech recognition text and the corpus instructions and the status of instruction execution results are calculated to determine the number of executed instructions, the number of redundant instructions, etc. The comprehensive scoring method includes scores for recognition accuracy, effectiveness, process standardization and scenario awareness, and the evaluation level is quantified.
It enables intelligent and objective evaluation of the voice communication process during controller simulation training, improving evaluation efficiency and scoring accuracy, and better reflecting training effectiveness.
Smart Images
Figure CN121506189A_ABST
Abstract
Description
Technical Field
[0001] Embodiments of this disclosure relate to the field of controller simulation training, and more specifically, to a voice call scoring method, apparatus, device, and medium suitable for controller simulation training. Background Technology
[0002] In controller simulation training, the rationality of the instructions issued and their matching degree with the business process during the corresponding call process need to be objectively and timely evaluated.
[0003] In related technologies, the evaluation of controller training voice communication mainly relies on human experience. For example, instructors with relevant professional knowledge manually observe the controller's behavior and response during the training process, and then calculate corresponding scores to evaluate the controller's voice communication.
[0004] However, using existing technologies results in low evaluation efficiency and insufficient objectivity in the scoring results. Summary of the Invention
[0005] The embodiments described herein provide a method, apparatus, device, and medium for scoring voice calls during controller simulation training, overcoming the aforementioned problems.
[0006] Firstly, based on the content of this disclosure, a voice call scoring method for controller simulation training is provided, including: The controllers are trained to conduct voice communication simulations using a simulation training system, and the voice recognition results data output by the voice server in the simulation training system are obtained. The voice recognition results data include: the voice recognition text of the call, the simulated execution instructions, and the status of the instruction execution results. The total number of execution instructions and the number of matching instructions are determined based on the voice recognition text of the call; the number of matching instructions is the number of data entries in the voice recognition text whose matching degree with the instructions in the corpus is greater than a preset matching threshold; The number of successfully executed instructions is determined based on the execution result status of the instructions; the number of successfully executed instructions is the number of instructions in the simulated execution instructions whose execution result status is successful. The number of redundant instructions is determined based on the simulated execution instructions; Based on the voice recognition text of the call and the execution result status of the instruction, the number of invalid instructions is determined; the number of invalid instructions is the number of data entries in the voice recognition text that have a matching relationship with the instruction in the corpus and whose corresponding execution result status is failure. The accuracy score for recognizing the simulated execution instructions is determined based on the number of matched instructions and a preset accuracy weight; the effectiveness score for recognizing the simulated execution instructions is determined based on the number of successfully executed instructions and a preset success weight; the process standardization score is determined based on the number of missing instructions, the number of instructions with incorrect execution order, and the number of redundant instructions; and the scenario awareness score is determined based on the number of invalid state instructions and the total number of executed instructions. The accuracy score, effectiveness score, process standardization score, and scenario awareness score are summed to obtain the simulation training score for the controller; and the corresponding simulation training evaluation level is determined based on the controller's simulation training score.
[0007] Secondly, according to the present disclosure, a voice call scoring device for controller simulation training is provided, comprising: The acquisition module is used to simulate voice communication for controllers through a simulation training system and acquire the voice recognition result data output by the voice server in the simulation training system; the voice recognition result data includes: the voice recognition text of the call, the simulated execution instruction, and the instruction execution result status; The first determining module is used to determine the total number of execution instructions and the number of matching instructions based on the voice recognition text of the call; the number of matching instructions is the number of data entries in the voice recognition text whose matching degree with the instructions in the corpus is greater than a preset matching threshold; The second determining module is used to determine the number of successfully executed instructions based on the instruction execution result status; the number of successfully executed instructions is the number of instructions in the simulated execution instructions whose instruction execution result status is successful. The third determining module is used to determine the number of redundant instructions based on the simulated execution instructions; The fourth determining module is used to determine the number of invalid instructions based on the voice recognition text of the call and the execution result status of the instruction; the number of invalid instructions is the number of data entries in the voice recognition text of the call that have a matching relationship with the instruction in the corpus and whose corresponding execution result status is failure. The fifth determining module is used to determine the recognition accuracy score of the simulated execution instruction based on the number of matched instructions and a preset accuracy weight; determine the recognition validity score of the simulated execution instruction based on the number of successfully executed instructions and a preset success weight; determine the process standardization score based on the number of missing instructions, the number of instructions with incorrect execution order, and the number of redundant instructions; and determine the scene awareness score based on the number of invalid state instructions and the total number of executed instructions. The sixth determining module is used to sum the recognition accuracy score, the recognition effectiveness score, the process standardization score, and the scenario awareness score to obtain the simulation training score corresponding to the controller; and to determine the corresponding simulation training evaluation level based on the simulation training score corresponding to the controller.
[0008] Thirdly, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the voice call scoring method for controller simulation training as described in any of the above embodiments.
[0009] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the voice call scoring method for controller simulation training as described in any of the above embodiments.
[0010] The voice call scoring method for controller simulation training provided in this application embodiment simulates voice calls for controllers through a simulation training system and obtains voice recognition result data output by the voice server in the simulation training system. The voice recognition result data includes: the voice recognition text of the call, simulated execution instructions, and the status of instruction execution results. The total number of execution instructions and the number of matched instructions are determined based on the voice recognition text. The number of matched instructions is the number of data entries in the voice recognition text whose matching degree with corpus instructions is greater than a preset matching threshold. The number of successfully executed instructions is determined based on the status of instruction execution results. The number of successfully executed instructions is the number of instructions in the simulated execution instructions whose execution result status is successful. The number of redundant instructions is determined based on the simulated execution instructions. The method is further determined based on the voice recognition text and the status of instruction execution results. The system calculates the number of invalid commands; these are data entries in the voice recognition text that match commands in the corpus but whose execution result is a failure. The accuracy score for simulated command execution is determined based on the number of matched commands and a preset accuracy weight. The effectiveness score is determined based on the number of successfully executed commands and a preset success weight. The process standardization score is determined based on the number of missing commands, commands with incorrect execution order, and redundant commands. The scenario awareness score is determined based on the number of invalid commands and the total number of executed commands. The accuracy score, effectiveness score, process standardization score, and scenario awareness score are summed to obtain the controller's simulation training score. The corresponding simulation training evaluation level is then determined based on the controller's simulation training score. This system statistically and intelligently evaluates the voice call performance across four dimensions—accuracy, command effectiveness, process standardization, and scenario awareness—to effectively solve the problems of cumbersome statistics, low efficiency, and subjective results in the voice call performance evaluation process for controllers during simulation training.
[0011] The above description is merely an overview of the technical solutions of the embodiments of this application. In order to better understand the technical means of the embodiments of this application and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of this application more obvious and understandable, specific implementation methods of this application are described below. Attached Figure Description
[0012] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. It should be understood that the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure, wherein: Figure 1 This is a flowchart illustrating a voice call scoring method for controller simulation training provided in this publication.
[0013] Figure 2 This is a schematic diagram of the structure of a voice call scoring device for controller simulation training provided in this disclosure.
[0014] Figure 3 This is a schematic diagram of the structure of a computer device provided in this disclosure.
[0015] It should be noted that the elements in the attached diagram are schematic and not drawn to scale. Detailed Implementation
[0016] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are also within the scope of protection of this disclosure.
[0017] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this subject matter pertains. It will be further understood that terms such as those defined in commonly used dictionaries shall be interpreted as having the meaning consistent with their meaning in the context of the specification and in the relevant art, and shall not be interpreted in an idealized or overly formal form unless otherwise explicitly defined herein. As used herein, the statement of “connecting” or “coupling” two or more parts together shall mean that these parts are directly joined together or joined through one or more intermediate components.
[0018] The term "embodiment" as used herein means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of the phrase "embodiment" in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.
[0019] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists, A and B exist simultaneously, or B exists. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Terms such as "first" and "second" are only used to distinguish one component (or part of a component) from another component (or another part of a component).
[0020] In the description of this application, unless otherwise stated, "multiple" means two or more (including two), and similarly, "multiple groups" means two or more (including two groups).
[0021] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.
[0022] Figure 1 This is a flowchart illustrating a voice call scoring method for controller simulation training provided in this disclosure, as shown in the embodiment. Figure 1 As shown, the specific process of the voice call scoring method in controller simulation training includes: S110. Conduct voice call simulations for controllers through a simulation training system and obtain the voice recognition result data output by the voice server in the simulation training system; the voice recognition result data includes: the voice recognition text of the call, the simulated execution instructions, and the status of the instruction execution results.
[0023] The Log collection module can receive speech recognition results data output from the speech server in the training system. The call speech recognition text is the text content converted after the call speech is recognized; the execution result status is used to indicate the execution result of the simulated execution instruction, such as "OK" indicating that the corresponding simulated execution instruction was executed successfully, and "ERROR" indicating that the corresponding simulated execution instruction failed / error occurred.
[0024] S120. Determine the total number of execution instructions and the number of matched instructions based on the voice recognition text of the call; determine the number of successfully executed instructions based on the status of the instruction execution results; determine the number of redundant instructions based on the simulated execution instructions.
[0025] The total number of executed instructions is the sum of the number of non-empty data entries in the call speech recognition text. The number of matched instructions is the number of data entries in the call speech recognition text whose matching degree with the instructions in the corpus is greater than a preset matching threshold. The number of matched instructions can be determined by summing the number of data entries in all data where the matching degree between the call speech recognition text and the instructions in the corpus is greater than or equal to 0.8.
[0026] For example, each data point is evaluated in a loop, and the Jaccard similarity formula is used to calculate the similarity between the speech recognition text in each data point and the corresponding instruction in the corpus. The similarity calculation formula is shown in formula (1) below.
[0027] J(A,B)=∣A∩B∣ / ∣A∪B∣(1) In formula (1), A represents the corresponding instruction in the corpus; B represents the text of the voice recognition call.
[0028] The instruction matching degree S state is shown in the following formula (2).
[0029] (2) When J ≥ 0.8, it means that the speech recognition text matches the instructions in the corpus, denoted as T; when J < 0.8, it means that the speech recognition text does not match the instructions in the corpus, denoted as F.
[0030] The speech recognition text of S is counted and summed according to the following formula (3).
[0031] (3) In formula (3), MI represents the number of matching instructions; n represents the total number of loop iterations; S i This represents the calculation result (F / T) of the i-th iteration; ( ) is an indicator function that indicates when the instruction matches the degree of S and the state is T. =1, otherwise =0.
[0032] The number of successfully executed instructions is the number of instructions whose execution result status is successful in the simulated execution. That is, it is the sum of the number of data entries with an instruction execution result status of OK in all data. As shown in the following formula (4).
[0033] (4) In formula (4), SI represents the number of instructions executed successfully.
[0034] The number of redundant instructions is the sum of the number of all adjacent instructions that are repeated in the simulated execution instructions.
[0035] In some embodiments, determining the number of redundant instructions based on simulated execution instructions includes: obtaining duplicate instructions in the simulated execution instructions and recording the execution order of the duplicate instructions and the corresponding instruction execution result status; if the duplicate instructions are determined to be in an adjacent order based on the execution order of the duplicate instructions, and the instruction execution result status corresponding to the duplicate instructions are all the same, then the duplicate instructions are determined to be redundant instructions; and the number of redundant instructions is determined based on the total value of the redundant instructions.
[0036] The execution order of repeated instructions is the sequence in which the same instructions appear consecutively during the simulation. By analyzing the execution order of repeated instructions, we can determine which instructions are adjacent. If the execution result states of adjacent repeated instructions are the same, it means that these instructions do not have a substantial impact on the final result, and are therefore classified as redundant instructions. This effectively reduces redundant parts in the instruction sequence, improving execution efficiency and resource utilization.
[0037] S130. Determine the number of invalid commands based on the voice recognition text and the command execution result status.
[0038] The number of invalid commands refers to the number of data entries in the call speech recognition text that match commands in the corpus, but whose execution result is an ERROR. In other words, it is the sum of all data entries where the call speech recognition text matches commands in the corpus, but the command execution result is ERROR.
[0039] For example, the instruction matching degree is determined using formulas (1) and (2). Instruction status As shown in formula (5) below.
[0040] (5) When S=T and RS=ERROR, the result is denoted as T; when S=T and RS=OK, the result is denoted as F. According to the following formula (6), Count and sum the data for T.
[0041] (6) In formula (6), IS represents the number of invalid instructions.
[0042] S140. Determine the recognition accuracy score of the simulated execution command based on the number of matched commands and the preset accuracy weight; determine the recognition effectiveness score of the simulated execution command based on the number of successfully executed commands and the preset success weight; determine the process standardization score based on the number of missing commands, the number of commands with incorrect execution order, and the number of redundant commands; determine the scenario awareness score based on the number of invalid state commands and the total number of executed commands.
[0043] The preset accuracy and success rate weights can be set according to actual needs to adjust the importance of each score in the overall evaluation. The process standardization score reflects the degree of standardization of the operation process by comprehensively analyzing the number of missing instructions, instructions with incorrect execution order, and redundant instructions. The scenario awareness score quantifies the system's adaptability and response level to complex scenarios by combining the ratio of invalid state instructions to the total number of executed instructions.
[0044] In some embodiments, the accuracy score for recognizing simulated execution instructions is determined based on the number of matched instructions and a preset accuracy weight, including: calculating the ratio of the number of matched instructions to the total number of executed instructions; and multiplying the ratio of the number of matched instructions to the total number of executed instructions by the preset accuracy weight and normalizing it to obtain the accuracy score for recognizing simulated execution instructions.
[0045] The recognition accuracy score RA can be calculated according to the following formula (7).
[0046] RA = MI / TI RA_W(7) In formula (7), MI represents the number of matched instructions; TI represents the total number of executed instructions; and RA_W represents the preset accuracy weight, which, after multiple tests, is 0.25. For example, if MI = 113 and TI = 125, then RA = 113 / 125. 100 0.25 = 22.6.
[0047] In some embodiments, the identification effectiveness score of the simulated execution instruction is determined based on the number of successfully executed instructions and a preset success rate weight, including: calculating the ratio of the number of successfully executed instructions to the total number of executed instructions; multiplying the ratio of the number of successfully executed instructions to the total number of executed instructions by the preset success rate weight and normalizing it to obtain the identification effectiveness score of the simulated execution instruction.
[0048] The identification validity score (IV) can be calculated according to the following formula (8).
[0049] IV = SI / TI IV_W(8) In formula (8), SI represents the number of successfully executed instructions; IV_W represents the preset success rate weight, which, after multiple tests, is 0.3. For example, if SI=102 and TI=125, then IV=102 / 125. 100 0.3 = 24.48.
[0050] In some embodiments, the process standardization score is determined based on the number of missing instructions, the number of instructions with incorrect execution order, and the number of redundant instructions. This includes: performing an instruction integrity assessment based on the number of missing instructions and the number of standard process instruction sets to obtain an integrity assessment score; performing an order correctness assessment based on the number of instructions with incorrect execution order and the number of standard process instruction sets to obtain a correctness assessment score; performing an instruction redundancy assessment based on the number of redundant instructions and the number of standard process instruction sets to obtain a redundancy assessment score; and determining the process standardization score based on the integrity assessment score, the correctness assessment score, the redundancy assessment score, and a preset standardization weight.
[0051] The standard process instruction set is a set of instructions that follows the standard process of the training subject and its corresponding execution order. The number of standard process instruction sets is the total number of instructions in the standard process instruction set.
[0052] The set of successfully executed instructions is the set of all instructions whose execution status is OK. The number of missing instructions is the number of instructions in the set of successfully executed instructions that are not included in the standard flow instruction set.
[0053] The number of instructions with execution order errors is the number of instructions in the set of instructions that were executed in a different order than the standard flow instruction set.
[0054] The integrity assessment score EI can be calculated according to the following formula (9).
[0055] EI = (1 - MIN / SN) 100 EI_W(9) In formula (9), MIN represents the number of missing instructions; SN represents the number of standard procedure instruction sets; and EI_W represents the integrity assessment weight, which, after multiple tests, is 0.6. For example, if MIN=4 and SN=86, then EI==(86-4) / 86 100 0.6 = 57.21.
[0056] The accuracy assessment score EC can be calculated according to the following formula (10).
[0057] EC = (1 - ESI / SN) 100 EC_W(10) In formula (10), ESI represents the number of instructions with execution sequence errors; EC_W represents the correctness evaluation weight, which, after multiple tests, is 0.3. For example, if ESI=5 and SN=86, then EC=(86-5) / 86 100 0.3 = 28.26.
[0058] The redundancy assessment score ER can be calculated according to the following formula (11).
[0059] ER = (1 - RI / SN) 100 ER_W(11) In formula (11), RI represents the number of redundant instructions; ER_W represents the redundancy evaluation weight, which, after multiple tests, is 0.1. For example, if RI=7 and SN=86, then ER==(86-7) / 86 100 0.1 = 9.19.
[0060] The process standardization score PS can be calculated according to the following formula (12).
[0061] PS = (EI + EC + ER) PS_W = (57.21 + 28.26 + 9.19) 0.25 = 23.67 (12) In formula (12), PS_W represents the preset normalization weight. After multiple tests, PS_W=0.25.
[0062] In some embodiments, the scene awareness score is determined based on the number of invalid state instructions and the total number of executed instructions, including: calculating the proportion of valid state instructions in the total number of executed instructions; multiplying the proportion of valid state instructions in the total number of executed instructions by a preset scene awareness weight and normalizing it to obtain the scene awareness score.
[0063] The scene awareness score CA can be calculated according to the following formula (13).
[0064] CA = (1 - IS / TI) 100 CA_W = (1 - 11 / 125) 100 0.2 = 18.24 (13) In formula (13), (1-IS / TI) represents the proportion of the number of valid state instructions in the total number of executed instructions; CA_W represents the preset scenario awareness weight, which is 0.2 after multiple tests.
[0065] S150. Sum the recognition accuracy score, recognition effectiveness score, process standardization score, and scenario awareness score to obtain the controller's corresponding simulation training score; and determine the corresponding simulation training evaluation level based on the controller's corresponding simulation training score.
[0066] Based on the above examples, the simulation training score for the controller can be obtained as FS=RA+IV+PS+CA=22.6+24.48+23.67+18.24=88.99.
[0067] In some embodiments, before determining the corresponding simulation training evaluation level based on the controller's simulation training score, the method further includes: constructing a score-level mapping table; the score-level mapping table stores the correspondence between the simulation training score range and the model training evaluation level.
[0068] The corresponding simulation training evaluation level is determined based on the controller's simulation training score, including: matching the controller's simulation training score with the simulation training score range corresponding to each model training evaluation level in the score-level mapping table to obtain the controller's simulation training evaluation level.
[0069] The scoring-level mapping table is shown in Table 1 below. The simulation training score corresponding to the controller is matched with the simulation training score range corresponding to each model training evaluation level in the scoring-level mapping table 1, resulting in a simulation training evaluation level of "Good" for the controller.
[0070] Table 1 Rating-Level Mapping Table
[0071] In Table 1, a simulation training evaluation level of "Excellent" indicates that the controller is excellent in all aspects; a simulation training evaluation level of "Good" indicates that the controller is good in most aspects; a simulation training evaluation level of "Meets the Standard" indicates that the controller meets the basic requirements; a simulation training evaluation level of "Basically Meets the Standard" indicates that the controller needs further improvement; and a simulation training evaluation level of "Failes to Meet the Standard" indicates that the controller needs to undergo intensive retraining.
[0072] Therefore, by matching the controller's simulation training score with the simulation training score range corresponding to each model training evaluation level in the score-level mapping table, the training effect of the controllers can be effectively quantified. This allows for more targeted intensive training or advanced courses to be arranged by clearly defining each controller's simulation training evaluation level, thereby comprehensively improving the team's overall quality and response capabilities.
[0073] In this embodiment, a voice call simulation is conducted for the controller using a simulation training system, and the voice recognition result data output by the voice server in the simulation training system is obtained. The voice recognition result data includes: the voice recognition text of the call, the simulated execution instructions, and the status of the instruction execution results. The total number of execution instructions and the number of matching instructions are determined based on the voice recognition text. The number of matching instructions is the number of data entries in the voice recognition text whose matching degree with the instructions in the corpus is greater than a preset matching threshold. The number of successfully executed instructions is determined based on the status of the instruction execution results. The number of successfully executed instructions is the number of instructions in the simulated execution instructions whose status is successful. The number of redundant instructions is determined based on the simulated execution instructions. The number of invalid instructions is determined based on the voice recognition text of the call and the status of the instruction execution results. The number of invalid commands refers to the number of data entries in the voice recognition text that match commands in the corpus, but whose execution result is a failure. The accuracy score for simulated command execution is determined based on the number of matched commands and a preset accuracy weight. The effectiveness score for simulated command execution is determined based on the number of successfully executed commands and a preset success weight. The process standardization score is determined based on the number of missing commands, commands with incorrect execution order, and redundant commands. The scenario awareness score is determined based on the number of invalid commands and the total number of executed commands. The accuracy score, effectiveness score, process standardization score, and scenario awareness score are summed to obtain the controller's simulation training score. The corresponding simulation training evaluation level is then determined based on the controller's simulation training score. In this way, statistical and intelligent evaluation is performed from four dimensions—accuracy, command effectiveness, process standardization, and scenario awareness—to obtain the voice call score for this training, effectively solving the problems of cumbersome statistics, low efficiency, and insufficient objectivity in the process of evaluating controllers' simulated voice call business capabilities.
[0074] Figure 2 This is a schematic diagram of a voice communication scoring device for controller simulation training provided in this embodiment. The voice communication scoring device for controller simulation training may include: The acquisition module 210 is used to simulate voice communication for controllers through the simulation training system and acquire the voice recognition result data output by the voice server in the simulation training system; the voice recognition result data includes: the voice recognition text of the call, the simulated execution instructions, and the status of the instruction execution results.
[0075] The first determining module 220 is used to determine the total number of execution instructions and the number of matching instructions based on the voice recognition text of the call; the number of matching instructions is the number of data entries in the voice recognition text whose matching degree with the instructions in the corpus is greater than a preset matching threshold.
[0076] The second determining module 230 is used to determine the number of successfully executed instructions based on the instruction execution result status; the number of successfully executed instructions is the number of instructions in the simulated instruction execution where the instruction execution result status is successful.
[0077] The third determining module 240 is used to determine the number of redundant instructions based on the simulated execution instructions.
[0078] The fourth determining module 250 is used to determine the number of invalid commands based on the voice recognition text and the command execution result status. The number of invalid commands is the number of data entries in the voice recognition text that match the commands in the corpus and whose corresponding command execution result status is failure.
[0079] The fifth determination module 260 is used to determine the recognition accuracy score of the simulated execution instruction based on the number of matched instructions and the preset accuracy weight; to determine the recognition validity score of the simulated execution instruction based on the number of successfully executed instructions and the preset success weight; to determine the process standardization score based on the number of missing instructions, the number of instructions with incorrect execution order, and the number of redundant instructions; and to determine the scenario awareness score based on the number of invalid state instructions and the total number of executed instructions.
[0080] The sixth determination module 270 is used to sum the recognition accuracy score, recognition effectiveness score, process standardization score, and scenario awareness score to obtain the controller's corresponding simulation training score; and to determine the corresponding simulation training evaluation level based on the controller's corresponding simulation training score.
[0081] In this embodiment, optionally, the third determining module 240 is specifically used for: Obtain duplicate instructions from the simulated execution instructions, and record the execution order and corresponding instruction execution result status of the duplicate instructions; if the duplicate instructions are determined to be in an adjacent order based on their execution order, and the execution result status of the duplicate instructions is the same, then the duplicate instructions are determined to be redundant instructions; determine the number of redundant instructions based on the total number of redundant instructions.
[0082] In this embodiment, optionally, the fifth determining module 260 is specifically used for: Calculate the ratio of the number of matched instructions to the total number of executed instructions; then multiply the ratio of the number of matched instructions to the total number of executed instructions by a preset accuracy weight and normalize it to obtain the recognition accuracy score for the simulated executed instructions.
[0083] In this embodiment, optionally, the fifth determining module 260 is specifically used for: Calculate the ratio of the number of successfully executed instructions to the total number of executed instructions; multiply the ratio of the number of successfully executed instructions to the total number of executed instructions by a preset success rate weight and normalize it to obtain the recognition effectiveness score of the simulated executed instructions.
[0084] In this embodiment, optionally, the fifth determining module 260 is specifically used for: Instruction integrity is assessed based on the number of missing instructions and the number of standard process instructions, resulting in an integrity score. Sequence correctness is assessed based on the number of instructions with execution order errors and the number of standard process instructions, resulting in a correctness score. Instruction redundancy is assessed based on the number of redundant instructions and the number of standard process instructions, resulting in a redundancy score. The process standardization score is determined based on the integrity score, correctness score, redundancy score, and preset standardization weights.
[0085] In this embodiment, optionally, the fifth determining module 260 is specifically used for: The number of invalid state instructions and the total number of executed instructions are used to calculate the proportion of valid state instructions to the total number of executed instructions. The proportion of valid state instructions to the total number of executed instructions is multiplied by the preset scene awareness weight and normalized to obtain the scene awareness score.
[0086] In this embodiment, optionally, a building module is also included.
[0087] The module is used to build a rating-level mapping table; the rating-level mapping table stores the correspondence between the simulation training rating range and the model training evaluation level.
[0088] The sixth determining module 270 is specifically used for: The simulation training score corresponding to the controller is matched with the simulation training score range corresponding to each model training evaluation level in the score-level mapping table to obtain the simulation training evaluation level corresponding to the controller.
[0089] The voice call scoring device for controller simulation training provided in this disclosure can execute the above-described method embodiments. Its specific implementation principle and technical effects can be found in the above-described method embodiments, and will not be repeated here.
[0090] This application also provides a computer device. Please refer to the following for details. Figure 3 , Figure 3 This is a basic structural block diagram of the computer device in this embodiment.
[0091] The computer device includes a memory 310 and a processor 320 that are interconnected via a system bus. It should be noted that only a computer device with memory 310 and processor 320 is shown in the figure; however, it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented alternatively. Those skilled in the art will understand that the computer device described herein is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.
[0092] Computer devices can include desktop computers, laptops, handheld computers, and cloud servers. These devices allow for human-computer interaction with users through keyboards, mice, remote controls, touchpads, or voice-activated devices.
[0093] The memory 310 includes at least one type of readable storage medium, including non-volatile memory or volatile memory, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. RAM may include static RAM or dynamic RAM. In some embodiments, the memory 310 may be an internal storage unit of a computer device, such as the hard disk or memory of the computer device. In other embodiments, the memory 310 may also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, or flash card equipped on the computer device. Of course, the memory 310 may include both internal storage units and external storage devices of the computer device. In this embodiment, the memory 310 is typically used to store the operating system and various application software installed on the computer device, such as the program code of the method described above. In addition, the memory 310 can also be used to temporarily store various types of data that have been output or will be output.
[0094] Processor 320 is typically used to perform overall operations of a computer device. In this embodiment, memory 310 is used to store program code or instructions, including computer operation instructions, and processor 320 is used to execute the program code or instructions stored in memory 310 or process data, such as program code that runs the methods described above.
[0095] In this article, the bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus system can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.
[0096] Another embodiment of this application also provides a computer-readable medium, which may be a computer-readable signal medium or a computer-readable medium. A processor in a computer reads computer-readable program code stored in the computer-readable medium, enabling the processor to execute the functional actions specified in each step or combination of steps in the above method; and to generate means for implementing the functional actions specified in each block or combination of blocks in the block diagram.
[0097] Computer-readable media include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared memory or semiconductor systems, devices or apparatuses, or any suitable combination thereof, wherein the memory is used to store program code or instructions, the program code including computer operation instructions, and the processor is used to execute the program code or instructions of the above-described methods stored in the memory.
[0098] The definitions of memory and processor can be found in the description of the foregoing computer device embodiments, and will not be repeated here.
[0099] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0100] In the various embodiments of this application, the functional units or modules can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0101] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0102] In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" as described in this application does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims listing several means, several units of these means may be embodied by the same item of hardware. The use of "first," "second," and "third," etc., does not indicate any order and these words should be interpreted as names. Unless otherwise specified, the steps in the above embodiments should not be construed as limiting the order of execution.
[0103] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A voice call scoring method for controller simulation training, characterized in that, include: The controllers were trained to conduct voice communication simulations using a simulation training system, and the voice recognition results data output by the voice server in the simulation training system were obtained. The speech recognition result data includes: the speech recognition text of the call, the simulated execution command, and the command execution result status; The total number of execution instructions and the number of matching instructions are determined based on the voice recognition text of the call; the number of matching instructions is the number of data entries in the voice recognition text whose matching degree with the instructions in the corpus is greater than a preset matching threshold; The number of successfully executed instructions is determined based on the execution result status of the instructions; the number of successfully executed instructions is the number of instructions in the simulated execution instructions whose execution result status is successful. The number of redundant instructions is determined based on the simulated execution instructions; Based on the voice recognition text of the call and the execution result status of the instruction, the number of invalid instructions is determined; the number of invalid instructions is the number of data entries in the voice recognition text that have a matching relationship with the instruction in the corpus and whose corresponding execution result status is failure. The accuracy score for recognizing the simulated execution instructions is determined based on the number of matched instructions and a preset accuracy weight; the effectiveness score for recognizing the simulated execution instructions is determined based on the number of successfully executed instructions and a preset success weight; the process standardization score is determined based on the number of missing instructions, the number of instructions with incorrect execution order, and the number of redundant instructions; and the scenario awareness score is determined based on the number of invalid state instructions and the total number of executed instructions. The accuracy score, effectiveness score, process standardization score, and scenario awareness score are summed to obtain the simulation training score for the controller; and the corresponding simulation training evaluation level is determined based on the controller's simulation training score.
2. The method according to claim 1, characterized in that, Determining the number of redundant instructions based on the simulated execution instructions includes: Obtain the repeated instructions in the simulated execution instructions, and record the execution order of the repeated instructions and the corresponding instruction execution result status; If the execution order of the repeated instructions determines that the repeated instructions belong to an adjacent order, and the execution result states of the instructions corresponding to the repeated instructions are all the same, then the repeated instructions are determined to be redundant instructions. The number of redundant instructions is determined based on the total value of the redundant instructions.
3. The method according to claim 1, characterized in that, The accuracy score for recognizing the simulated execution instructions is determined based on the number of matched instructions and a preset accuracy weight, including: Calculate the ratio of the number of matched instructions to the total number of executed instructions; The ratio of the number of matched instructions to the total number of executed instructions is multiplied by a preset accuracy weight and normalized to obtain the recognition accuracy score for the simulated executed instructions.
4. The method according to claim 1, characterized in that, The identification effectiveness score of the simulated execution instructions is determined based on the number of successfully executed instructions and a preset success rate weight, including: Calculate the ratio of the number of successfully executed instructions to the total number of executed instructions; The ratio of the number of successfully executed instructions to the total number of executed instructions is multiplied by a preset success rate weight and normalized to obtain a score for the effectiveness of identifying the simulated executed instructions.
5. The method according to claim 1, characterized in that, The process standardization score is determined based on the number of missing instructions, the number of instructions with incorrect execution order, and the number of redundant instructions, including: Instruction integrity is assessed based on the number of missing instructions and the number of standard process instruction sets, and an integrity assessment score is obtained. The correctness of the execution order is evaluated based on the number of instructions with execution order errors and the number of standard process instructions, and a correctness evaluation score is obtained. The redundancy of instructions is evaluated based on the number of redundant instructions and the number of standard process instructions, and a redundancy evaluation score is obtained. The process standardization score is determined based on the completeness assessment score, the correctness assessment score, the redundancy assessment score, and the preset standardization weight.
6. The method according to claim 1, characterized in that, The scene awareness score is determined based on the number of invalid state instructions and the total number of executed instructions, including: The number of invalid instructions and the total number of executed instructions are used to calculate the proportion of the number of valid instructions in the total number of executed instructions. The percentage of valid state instructions in the total number of executed instructions is multiplied by a preset scene awareness weight and normalized to obtain the scene awareness score.
7. The method according to claim 1, characterized in that, Before determining the corresponding simulation training assessment level based on the controller's simulation training score, the following steps are also included: Construct a rating-level mapping table; the rating-level mapping table stores the correspondence between the simulation training rating range and the model training evaluation level; The corresponding simulation training evaluation level is determined based on the controller's simulation training score, including: The simulation training score corresponding to the controller is matched with the simulation training score range corresponding to each model training evaluation level in the score-level mapping table to obtain the simulation training evaluation level corresponding to the controller.
8. A voice communication scoring device for controller simulation training, characterized in that, include: The acquisition module is used to simulate voice communication for controllers through a simulation training system and acquire the voice recognition result data output by the voice server in the simulation training system. The speech recognition result data includes: the speech recognition text of the call, the simulated execution command, and the command execution result status; The first determining module is used to determine the total number of execution instructions and the number of matching instructions based on the voice recognition text of the call; the number of matching instructions is the number of data entries in the voice recognition text whose matching degree with the instructions in the corpus is greater than a preset matching threshold; The second determining module is used to determine the number of successfully executed instructions based on the instruction execution result status; the number of successfully executed instructions is the number of instructions in the simulated execution instructions whose instruction execution result status is successful. The third determining module is used to determine the number of redundant instructions based on the simulated execution instructions; The fourth determining module is used to determine the number of invalid instructions based on the voice recognition text of the call and the execution result status of the instruction; the number of invalid instructions is the number of data entries in the voice recognition text of the call that have a matching relationship with the instruction in the corpus and whose corresponding execution result status is failure. The fifth determining module is used to determine the recognition accuracy score of the simulated execution instruction based on the number of matched instructions and a preset accuracy weight; determine the recognition validity score of the simulated execution instruction based on the number of successfully executed instructions and a preset success weight; determine the process standardization score based on the number of missing instructions, the number of instructions with incorrect execution order, and the number of redundant instructions; and determine the scene awareness score based on the number of invalid state instructions and the total number of executed instructions. The sixth determining module is used to sum the recognition accuracy score, the recognition effectiveness score, the process standardization score, and the scenario awareness score to obtain the simulation training score corresponding to the controller; and to determine the corresponding simulation training evaluation level based on the simulation training score corresponding to the controller.
9. A computer device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the voice call scoring method for controller simulation training as described in any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the voice call scoring method for controller simulation training as described in any one of claims 1 to 7.