End-to-end voice customer service work order intelligent classification method and device

Abstract

The present invention discloses an end-to-end intelligent voice work order classification method, which receives voice information input by customers, extracts voice features in the voice information, constructs a voice classification model, trains the model, inputs the voice features into the classification model, and outputs the classification result. Compared with voice recognition first and text classification later, the present invention optimizes the problem as a whole through end-to-end modeling technology, and has reached the global optimum; compared with manual preliminary analysis of voice information into text before work order classification modeling, the present invention directly analyzes and processes voice information, reduces the workload of human intervention, and further reduces the workload of grassroots service personnel; compared with other voice classification technologies, the multi-task needle model constructed by the present invention, one of the tasks is for the text content in the voice information, which is more in line with the task of work order classification. Furthermore, in the later model application deployment stage, the model deployment of the present invention occupies less space.

Description

Technical Field

[0001] The present invention belongs to the field of electric power customer service, and in particular relates to an end-to-end intelligent voice work order classification method and device. Background Art

[0002] Traditional customer service calls are generally classified and identified manually, but with the increase in users and service types, higher requirements are also put forward for service efficiency and quality. With the deepening of machine learning applications, more and more intelligent methods are being used in work order classification, but the current methods mainly convert the user's voice information into text first, and then classify the text, such as patent CN202011194254.3 and patent CN201810755334.8. Early text classification mainly represents the text by establishing a word vector dictionary, and then uses traditional machine learning methods such as decision trees to classify the text. This type of method has poor results because the word vector is fixed and does not associate with the context. Later, with the development of NLP technology, texts are classified through more accurate and in-depth deep learning models. However, this two-stage classification method has the following problems. First, this method requires two models to cooperate to complete the classification of voice work orders, a speech recognition model and a text classification model. Considering the later application deployment, it will occupy more space and memory; the two-stage analysis has the problem of error accumulation and cannot achieve the optimization of the overall task. There are also deep learning models that directly classify voice information, such as CN201911244364.3, which uses a model to perform sentiment analysis on the voice sequences of customer service and customers for voice service quality scoring. This model cannot achieve the purpose of classifying the content of voice work orders. Patent CN201811346110.8 proposes a multi-task capsule network architecture for the problem of voice classification, combining high- and low-level features for loss definition, but does not start from the perspective of voice content and cannot fully reason about the features in voice information related to work order classification. Summary of the invention

[0003] In order to overcome the shortcomings of the above-mentioned existing work order classification technology, the present invention provides an end-to-end intelligent voice work order classification method. The method comprises the following steps:

[0004] S1: Receive voice information input by the customer;

[0005] S2: Extract speech features from speech information;

[0006] S3: Build a speech classification model and train the model;

[0007] S4: Input the speech features into a classification model and output the classification result.

[0008] Accordingly, the present invention also proposes an end-to-end intelligent voice work order classification device, which includes:

[0009] Information receiving module: receiving voice information input by customers;

[0010] Feature extraction module: extracting speech features from speech information;

[0011] Model building module: build a speech classification model and train the model;

[0012] Classification module: inputs the speech features into the classification model and outputs the classification results.

[0013] Furthermore, the present invention also proposes a terminal, comprising a processor and a storage medium; the storage medium is used to store instructions; the processor is used to operate according to the instructions to execute the steps of the method of the present invention.

[0014] Furthermore, the present invention also proposes a computer-readable storage medium having a computer program stored thereon, characterized in that the program implements the steps of the method of the present invention when executed by a processor.

[0015] Compared with speech recognition first and text classification later, the present invention optimizes the problem as a whole through end-to-end modeling technology and has achieved global optimization;

[0016] Compared with manually analyzing the voice information and converting it into text before classifying and modeling the work order, the present invention directly analyzes and processes the voice information, thereby reducing the workload of human intervention and further reducing the workload of grassroots service personnel.

[0017] Compared with other speech classification technologies, the multi-task model constructed by the present invention, in which one task is aimed at the text content in the speech information, is more in line with the task of work order classification.

[0018] Furthermore, in the later stage of model application deployment, the model deployment of the present invention occupies a smaller space. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is an overall flow chart of the method of the present invention.

[0020] Figure 2 It is a model architecture of an embodiment provided by the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0022] The present invention provides an end-to-end voice customer service work order intelligent classification method, such as Figure 1 Trivia, including:

[0023] Information in the work order: address, name, contact information, professional involvement, customer request, account number, call time, completion time

[0024] Step 1: Receive voice information input by the customer

[0025] This step mainly uses microphone equipment to collect the customer's voice information for subsequent analysis.

[0026] Step 2: Extract voice features from voice information

[0027] Specifically, in order to facilitate the processing of subsequent models, each frame of speech signal needs to be preprocessed. The main purpose of this step is to convert the continuous time domain signal into a discrete frequency domain signal. The feature extraction method here is not limited. There are currently a variety of feature extraction methods in the field of speech recognition technology. The present invention adopts the MFCC (Mel Cepstral Coefficient) feature extraction method to extract the received speech signal at a certain sampling frequency through the MFCC feature to obtain a two-dimensional feature, which mainly includes frame stacking and downsampling;

[0028] 1) Frame stacking: N consecutive frames are spliced ​​together as a new frame;

[0029] 2) Downsampling: every M frames, only one frame of information is retained;

[0030] 3) After frame stacking and downsampling, the original speech signal (T, P) is transformed into (T / M, P*N).

[0031] By stacking frames and downsampling, the feature dimension can be reduced while retaining the information in the initial speech, making subsequent model training and prediction feasible.

[0032] Step 3: Input the processed speech features into the model and output the classification results.

[0033] Specifically, the steps include:

[0034] 1) Input an audio feature sequence to be recognized and a start label to the speech classification model, and the speech recognition model outputs the prediction result at the initial time;

[0035] 2) Concatenate the start tag with the prediction result at the initial moment to decode the historical prediction result at the next moment;

[0036] 3) Input the audio feature sequence to be identified at the next moment and the historical prediction results into the speech classification model, and use the decoder to calculate and decode the prediction result sequence at each moment;

[0037] 4) Join the prediction result sequence at each moment with the prediction result at that moment to update the historical prediction result;

[0038] 5) Repeat the above steps until the sound recognition model predicts the termination label. At this time, the decoding is completed. The first position in the final historical prediction result is the classification label. The classification label is input into the subsequent fully connected layer to predict the classification result and output the classification result. The other positions in the final historical prediction result correspond to the phoneme recognition results at each moment. After decoding, the speech prediction result can be obtained.

[0039] The key to the above-mentioned speech classification process lies in the speech classification model. The process of training the model is described in detail below. Figure 2 This is an example of a model architecture that uses a Transformer-based model architecture.

[0040] Taking the above Transformer network architecture as an example, the specific training process includes the following steps:

[0041] A. Data Preprocessing

[0042] The method requires data including voice, text and the category to which the voice belongs. Currently, the non-emergency repair work orders of State Grid 95598 include eight categories: consultation, complaint, report, opinion, suggestion, service application and praise. Each category is further divided into more detailed subcategories. A large amount of data content has been accumulated in the early stage, which needs to be manually sorted and labeled.

[0043] The sorted data set is divided into training set, test set and validation set in a ratio of 8:1:1.

[0044] B. Determination of network architecture

[0045] The network model of the method is equivalent to an acoustic model. Generally, the process of speech recognition includes an acoustic model and a language model (LM). The acoustic model converts speech signals into phonemes (pronunciation features), while the language model forms corresponding texts based on pronunciation features. A good speech recognition model requires that both models have high accuracy.

[0046] The purpose of the present invention is not speech recognition, but speech classification, so the network architecture only needs to recognize phonemes, and by simplifying the process of speech recognition, the network's fitting ability and generalization performance are increased. The network architecture used in the present invention can be multiple, not limited to Transformer, including CNN (convolutional neural network), RNN (recurrent neural network) and other commonly used deep learning network architectures.

[0047] In this example, a Transformer-based network structure is used, which mainly includes a self-attention layer, an attention mechanism, and a feed forward layer. The self-attention mechanism is followed by a normalization layer (add&norm). Specifically, the speech features pass through a four-layer network, each of which is composed of a self-attention layer and a feed forward layer; the text features pass through a six-layer network, each of which includes a self-attention layer, an attention layer, and a feed forward layer. After passing through the deep neural network architecture, the abstract high-order features of the speech features are extracted, and then through the linear layer, the phoneme probability features corresponding to each frame of speech features are obtained. In order to achieve the purpose of speech classification, the Bert classification algorithm is used to add a one-dimensional feature to the dimension of the output feature as a classification vector. After the classification vector is changed through the linear layer, the probability of each category is obtained. In summary, for each frame of speech feature input, the probability and classification probability of the corresponding phoneme are output, which is a two-dimensional matrix, where the length is the input frame + 1, where 1 is the added classification vector, and the width is the larger value of the number of phonemes and the number of categories of the recognized speech. By setting a specific loss function, the model is trained to output the correct classification results. After the model training is completed, the speech sample to be classified is input into the model, and the final category can be obtained by checking the newly added classification vector.

[0048] C. Definition of loss function

[0049] The training objectives of the present invention are two: one is that the model accurately predicts the corresponding phonemes, and the other is that the model accurately judges the category. For these two objectives, corresponding loss functions are constructed. The accurate prediction of objective one is of great help to objective two. The judgment of the type is mainly based on the content information in the speech. The accurate phoneme sequence represents more accurate content, so the two optimization objectives are consistent.

[0050] In summary, the model is a multi-task training process. The loss function for phonemes can be CTC (Continuous Time Classification). The input speech sequence is of length T, and the output of the speech encoder is {q 1 ,q 2 ...q T}, where q TIt is a vector that represents the probability of 1-N phonemes appearing at time T. N phonemes include recognized phonemes and empty markers Φ. The CTC method makes subsequent decoding more accurate by using an additional empty output Φ.

[0051] CTC_loss = -log P(y|x)

[0052]

[0053]

[0054] Where a represents a phoneme sequence of length T output by the model, p(a|x) represents the probability of forming a phoneme sequence a of length T, represents the probability that the predicted phoneme of the t-th frame speech is n, P(y|x) represents the conditional probability that the output is the label sequence y given the input speech x, and defines a many-to-one mapping B(a)=l, where l is the sequence after removing continuous and empty inputs in sequence a.

[0055] The second task is a classification task, using the cross entropy loss function

[0056]

[0057] Where p = [p 0 ,p 1 ,p 2 ...p c ] is a probability distribution, each element pi represents the probability y = [y 1 ,y 2 ,y 3 ...y c-1 ] is the one-hot representation of the sample label. When the sample belongs to the i-th category, y i =1, otherwise y i = 0. The task of the present invention is to combine the two, so the total loss function is:

[0058] Loss = Loss_k + CTC_loss

[0059] D. Model Training

[0060] After several rounds of iterative training, when the model accuracy meets the requirements and the accuracy remains unchanged for 10 consecutive iterations, the model can be considered to have been trained and the trained model can be output.

[0061] Application scenarios, in addition to solving the classification of traditional telephone customer service work orders, the present invention can also be applied to online work order processing systems, supporting customers to directly leave voice messages to solve problems, and then directly classify and distribute customer voice messages, which is convenient and fast.

[0062] Accordingly, the present invention also proposes an end-to-end intelligent voice work order classification device, which includes:

[0063] Information receiving module: receiving voice information input by customers;

[0064] Feature extraction module: extracting speech features from speech information;

[0065] Model building module: build a speech classification model and train the model;

[0066] Classification module: inputs the speech features into the classification model and outputs the classification results.

[0067] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0068] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0069] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0070] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0071] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0072] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0073] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0074] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0075] The applicant has made a detailed explanation and description of the embodiments of the present invention in conjunction with the drawings in the specification. However, those skilled in the art should understand that the above embodiments are only preferred implementation schemes of the present invention, and the detailed description is only to help readers better understand the spirit of the present invention, but not to limit the scope of protection of the present invention. On the contrary, any improvement or modification based on the spirit of the present invention should fall within the scope of protection of the present invention.

Claims

1. An end-to-end intelligent voice work order classification method, characterized in that: The method comprises the following steps: S1: Receive voice information input by the customer; S2: Extract speech features from speech information; use the Mel-frequency cepstral coefficient feature extraction method to extract the received speech signal at a certain sampling frequency through MFCC feature extraction to obtain two-dimensional features, including frame stacking and downsampling; frame stacking splices N consecutive frames together as a new frame; Downsampling means that only one frame of information is retained every M frames. After the frame stacking and downsampling process, the original speech signal (T, P) is transformed into (T / M, P*N). S3: Construct a speech classification model and train the model; the classification model adopts a Transformer-based network structure, in which speech features pass through a four-layer network, each layer of which is composed of a self-attention layer and a feedforward layer; text features pass through a six-layer network, each layer of which includes a self-attention layer, an attention layer, and a feedforward layer; after passing through the deep neural network architecture, the abstract high-order features of the speech features are extracted, and then passed through a linear layer to obtain the phoneme probability features corresponding to each frame of speech features; The speech classification model adds a one-dimensional feature to the dimension of the output feature as a classification vector. After the classification vector is transformed through a linear layer, the probability of each category is obtained. The speech classification model outputs the probability and classification probability of the corresponding phoneme for each frame of speech feature input, which is a two-dimensional matrix, wherein the length is the input frame + 1, wherein 1 is the added classification vector, and the width is the larger value of the number of phonemes and the number of categories of the recognized speech; S4: Input the speech features into a classification model and output the classification result.

2. The method according to claim 1, characterized in that By setting the loss function, the classification model is trained so that the classification model can output the correct classification results.

3. The method according to claim 2, characterized in that The loss function is defined as Loss = Loss_k + CTC_loss; Where Loss_k is the cross entropy loss function Where p=[p0,p1,p2...p c ] is a probability distribution, each element pi represents the probability that the sample belongs to the i-th category y = [y1, y2, y3...y c-1 ] is the one-hot representation of the sample label. When the sample belongs to the i-th category, y i =1, otherwise y i =0; CTC_loss = -log P(y|x) Where a represents a phoneme sequence of length T output by the model, p(a|x) represents the probability of forming a phoneme sequence a of length T, represents the probability that the predicted phoneme of the t-th frame speech is n, P(y|x) represents the conditional probability that the output is the label sequence y given the input speech x, and defines a many-to-one mapping B(a)=l, where l is the sequence after removing continuous and empty inputs in sequence a.

4. The method according to claim 3, characterized in that The step S4 comprises: S1: Input a sequence of audio features to be identified and a start label to the speech classification model, and the model outputs the prediction result at the initial time; S2: concatenate the start label with the prediction result at the initial moment as the historical prediction result for decoding the next moment; S3: Input the audio feature sequence to be recognized at the next moment and the historical prediction results into the speech classification model, and calculate and decode the prediction result sequence at each moment through the decoder; S4: concatenate the prediction result sequence of each moment with the prediction result of the moment, and update the historical prediction result; S5: Repeat the above steps until the speech recognition model predicts the end tag, at which point the decoding is complete.

5. The method according to claim 4, characterized in that The first position in the final historical prediction result is the classification label, which is input into the subsequent fully connected layer to predict the classification result and output the classification result; the other positions in the final historical prediction result correspond to the phoneme recognition results at each moment, and the speech prediction result can be obtained after decoding.

6. An end-to-end intelligent voice work order classification device, used to implement the method of claims 1-5, characterized in that: The device includes: Information receiving module: receiving voice information input by customers; Feature extraction module: extracting speech features from speech information; Model building module: build a speech classification model and train the model; Classification module: inputs the speech features into the classification model and outputs the classification results.

7. A terminal, characterized in that: including processor and storage medium; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the method according to any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images