Customer service dialogue evaluation method and device, equipment, storage medium and product

Abstract

The invention discloses a customer service dialogue evaluation method and device, equipment, a storage medium and a product, and is applied to the technical field of data processing, and the method comprises the steps: obtaining multi-modal dialogue data generated in a customer service process, carrying out the preprocessing of the multi-modal dialogue data, and generating structured dialogue data; based on the structured dialogue data, basic dialogue evaluation indexes are calculated, and the basic dialogue evaluation indexes comprise a service efficiency index, a service quality index, a business capability index, a compliance index and a customer stickiness index; based on the structured dialogue data and the basic dialogue evaluation index, respectively generating a customer satisfaction index, a customer loss early warning index and a potential service demand identification index through a machine learning model; and inputting the basic dialogue evaluation index, the customer satisfaction index, the customer loss early warning index and the potential service demand identification index into the service dialogue evaluation model to obtain a service dialogue evaluation result, so that the actual service quality of the dialogue can be comprehensively and accurately reflected.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and in particular relates to a customer service dialogue evaluation method, apparatus, equipment, storage medium and product. Background Technology

[0002] In industries highly reliant on customer service, such as finance and e-commerce, effectively evaluating the quality of customer service conversations is crucial for improving service standards and optimizing customer experience. Currently, common methods for assessing conversation quality primarily rely on manual sampling and automated detection based on fixed rules. For example, quality management personnel randomly select a portion of conversation records and subjectively rate service attitude and problem-solving based on their personal experience. Alternatively, a series of keywords (such as sensitive words or greetings) or simple indicators (such as average response time) are pre-defined, and pattern matching or threshold judgments are used to identify irregular or abnormal conversations.

[0003] However, existing technologies can only make subjective scores based on personal experience, or identify illegal or abnormal conversations based on preset keywords or simple indicators, which cannot fully and accurately reflect the actual service quality of the conversation. Summary of the Invention

[0004] This application provides a customer service dialogue evaluation method, apparatus, equipment, storage medium, and product that can comprehensively and accurately reflect the actual service quality of the dialogue.

[0005] In a first aspect, embodiments of this application provide a customer service dialogue evaluation method, the method comprising: The system acquires multimodal dialogue data generated during customer service, preprocesses the multimodal dialogue data, and generates structured dialogue data. The multimodal dialogue data includes at least one of text data, voice data, and video data, and the structured dialogue data includes at least two of dialogue metadata, text content features, and sentiment features. Based on structured dialogue data, basic dialogue evaluation indicators are calculated. These indicators include at least one of the following: service efficiency indicators, service quality indicators, business capability indicators, compliance indicators, and customer stickiness indicators. Based on structured dialogue data and basic dialogue evaluation metrics, customer satisfaction metrics, customer churn warning metrics, and potential service demand identification metrics are generated through machine learning models. By inputting basic dialogue evaluation metrics, customer satisfaction metrics, customer churn warning metrics, and potential service demand identification metrics into the service dialogue evaluation model, the service dialogue evaluation results are obtained.

[0006] In some possible implementations, basic dialogue evaluation metrics, customer satisfaction metrics, customer churn warning metrics, and potential service demand identification metrics are input into the service dialogue evaluation model to obtain service dialogue evaluation results, including: Based on the business objectives corresponding to the multimodal dialogue data, determine the first weight for each indicator; Based on the time corresponding to the multimodal dialogue data, determine the second weight corresponding to each indicator; Based on the customer identifiers corresponding to the multimodal dialogue data, the third weights corresponding to each indicator are determined. Based on at least one of the first weight, second weight, and third weight, the basic dialogue evaluation index, customer satisfaction index, customer churn warning index, and potential service demand identification index are weighted and summed to obtain the service dialogue evaluation result.

[0007] In some possible implementations, the machine learning model is a customer satisfaction model; Before generating customer satisfaction metrics, customer churn warning metrics, and potential service demand identification metrics through machine learning models based on structured dialogue data and basic dialogue evaluation metrics, the process also includes: Obtain the customer satisfaction training dataset, which includes multiple customer satisfaction training data samples. Each customer satisfaction training data sample includes: sentiment feature samples, service efficiency index samples, service quality index samples, and customer satisfaction labels. Based on the customer satisfaction training dataset, the customer satisfaction model to be trained is trained to obtain a well-trained customer satisfaction model.

[0008] In some possible implementations, the machine learning model is a customer churn prediction model; Before generating customer satisfaction metrics, customer churn warning metrics, and potential service demand identification metrics through machine learning models based on structured dialogue data and basic dialogue evaluation metrics, the process also includes: Obtain a customer churn warning training dataset. The customer churn warning training dataset includes multiple customer churn warning training data samples. Each customer churn warning training sample includes: customer question frequency change characteristics, customer conversation interval duration change characteristics, negative sentiment change characteristics, and customer churn warning labels. Based on the customer churn warning training dataset, the customer churn warning model to be trained is trained to obtain a well-trained customer churn warning model.

[0009] In some possible implementations, machine learning models identify potential service demands. Before generating customer satisfaction metrics, customer churn warning metrics, and potential service demand identification metrics through machine learning models based on structured dialogue data and basic dialogue evaluation metrics, the process also includes: Obtain the potential service demand identification training dataset, which includes multiple potential service demand identification training samples. Each potential service demand identification training sample includes: customer demand feature samples and business transaction record labels. Based on the potential service demand identification training dataset, the potential service demand identification model to be trained is trained to obtain a well-trained potential service demand identification model.

[0010] In some possible implementations, target multimodal dialogue data with service dialogue evaluation results higher than a preset evaluation result threshold are selected from multimodal dialogue data to construct a reinforcement learning training sample set. For each target multimodal dialogue data, perform the following steps: Input customer inquiry data from the target multimodal dialogue data into the intelligent customer service model to be trained, and obtain customer service response data generated by the intelligent customer service model to be trained; New dialogue data to be evaluated is generated based on customer inquiry data and customer service response data. The process of preprocessing multimodal dialogue data to generate structured dialogue data is then returned, and new service dialogue evaluation results are regenerated. Determine whether the service dialogue evaluation results meet the preset training stop conditions; If the conditions are not met, adjust the parameters of the intelligent customer service model to be trained, and continue training the intelligent customer service model using the reinforcement learning training sample set until the preset training stopping condition is met, thus obtaining the trained intelligent customer service model.

[0011] Secondly, embodiments of this application provide a customer service dialogue evaluation device, the device comprising: The acquisition module is used to acquire multimodal dialogue data generated during customer service, preprocess the multimodal dialogue data, and generate structured dialogue data. The multimodal dialogue data includes at least one of text data, voice data, and video data, and the structured dialogue data includes at least two of dialogue metadata, text content features, and sentiment features. The calculation module is used to calculate basic dialogue evaluation indicators based on structured dialogue data. The basic dialogue evaluation indicators include at least one of the following: service efficiency indicators, service quality indicators, business capability indicators, compliance indicators, and customer stickiness indicators. The first processing module is used to generate customer satisfaction indicators, customer churn warning indicators, and potential service demand identification indicators based on structured dialogue data and basic dialogue evaluation indicators through machine learning models. The second processing module is used to input basic dialogue evaluation indicators, customer satisfaction indicators, customer churn warning indicators, and potential service demand identification indicators into the service dialogue evaluation model to obtain service dialogue evaluation results.

[0012] Thirdly, embodiments of this application provide an electronic device, the device comprising: A processor and a memory storing computer program instructions; a customer service dialogue evaluation method that implements any of the above when the processor executes the computer program instructions.

[0013] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement a customer service dialogue evaluation method as described above.

[0014] Fifthly, embodiments of this application provide a computer program product in which instructions, when executed by the processor of an electronic device, enable the electronic device to perform any of the above-mentioned customer service dialogue evaluation methods.

[0015] The customer service dialogue evaluation method, apparatus, device, storage medium, and product provided in this application acquire multimodal dialogue data generated during customer service, preprocess the multimodal dialogue data to generate structured dialogue data, and then calculate various basic dialogue evaluation indicators based on the structured dialogue data, including service efficiency indicators, service quality indicators, business capability indicators, compliance indicators, and customer stickiness indicators. Furthermore, based on the structured dialogue data and the basic dialogue evaluation indicators, higher-level indicators such as customer satisfaction indicators, customer churn warning indicators, and potential service demand identification indicators are generated through machine learning models. The evaluation dimensions are more comprehensive and more closely aligned with the actual needs of service dialogue business. Therefore, the service dialogue evaluation results obtained based on the service dialogue evaluation model can more comprehensively and accurately reflect the actual service quality of the dialogue. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating a customer service dialogue evaluation method provided in one embodiment of this application is shown; Figure 2 A flowchart illustrating a customer service dialogue evaluation method provided in another embodiment of this application is shown; Figure 3A flowchart illustrating a customer service dialogue evaluation method provided in another embodiment of this application is shown; Figure 4 A flowchart illustrating a customer service dialogue evaluation method provided in another embodiment of this application is shown; Figure 5 A schematic diagram of the customer service dialogue evaluation device provided in an embodiment of this application is shown; Figure 6 A schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0018] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0019] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0020] It should be noted that the acquisition, storage, use, and processing of data in this application embodiment all comply with the relevant provisions of national laws and regulations.

[0021] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, they do not mean that the applicant has used or necessarily used the solution.

[0022] Currently, service dialogue evaluation mainly relies on quality management personnel randomly selecting a portion of dialogue records and subjectively scoring service attitude and problem-solving based on personal experience. Alternatively, it uses preset simple matching rules, such as pre-setting a series of keywords (e.g., sensitive words, greetings) or simple indicators (e.g., average response time), to identify violations or abnormal dialogues through pattern matching or threshold judgment. However, subjective scoring based on personal experience or identifying violations or abnormal dialogues based on preset keywords or simple indicators cannot comprehensively and accurately reflect the actual service quality of the dialogue.

[0023] To address the problems of existing technologies, embodiments of this application provide a customer service dialogue evaluation method, apparatus, device, storage medium, and product. This method can acquire multimodal dialogue data generated during customer service, preprocess the multimodal dialogue data to generate structured dialogue data, and then calculate various basic dialogue evaluation indicators based on the structured dialogue data, including service efficiency indicators, service quality indicators, business capability indicators, compliance indicators, and customer stickiness indicators. Furthermore, based on the structured dialogue data and the basic dialogue evaluation indicators, a machine learning model is used to generate higher-level indicators such as customer satisfaction indicators, customer churn warning indicators, and potential service demand identification indicators. This results in a more comprehensive evaluation dimension and better aligns with the actual needs of service dialogue business. Therefore, the final service dialogue evaluation results obtained based on the service dialogue evaluation model can more comprehensively and accurately reflect the actual service quality of the dialogue.

[0024] The following section first introduces a customer service dialogue evaluation method provided by an embodiment of the present invention.

[0025] Figure 1 A flowchart illustrating a customer service dialogue evaluation method provided in one embodiment of this application is shown. Figure 1 As shown, the method may include the following steps: S101: Acquire multimodal dialogue data generated during customer service, preprocess the multimodal dialogue data, and generate structured dialogue data; wherein, the multimodal dialogue data includes at least one of text data, voice data, and video data, and the structured dialogue data includes at least two of dialogue metadata, text content features, and sentiment features.

[0026] In this embodiment, multimodal dialogue data refers to data generated during conversations between customers and customer service representatives, such as text data (chat messages), voice data (voice calls, voice messages), and video data (video calls). In one example, multimodal dialogue data may also include attachments generated during the chat, such as files, compressed files, links, and images. It should be noted that multimodal dialogue data can be obtained from various customer service platforms, such as WeChat Work, DingTalk, Lark, or a self-built online customer service platform.

[0027] After receiving multimodal dialogue data, the data is preprocessed, including cleaning, format conversion, and feature extraction, to generate structured dialogue data. Structured dialogue data refers to standardized data that has been organized according to fixed logic and can be directly used for subsequent metric calculations and model training. This includes dialogue metadata, text content features, and sentiment features. In one example, the cleaning, format conversion, and feature extraction operations on the multimodal data involve model processing. High-quality labeled dialogue data is often scarce in the early stages of business or in specific scenarios. This application improves the quality of model training and further enhances the accuracy of structured dialogue data extraction by performing data augmentation on text, speech, and images. Among these features, text data can be processed with synonym replacement, back-translation (Chinese to English and then back to Chinese), or random insertion or deletion; background noise can be added to speech data to simulate the environmental noise that may exist in a customer service center, enhancing the robustness of the speech recognition model in real noisy environments and ensuring the accuracy of speech-to-text conversion; speed and pitch can be varied to change the speech rate and pitch, making the model independent of specific sound patterns and enhancing its generalization ability; image data can be rotated, cropped, or have its brightness adjusted to simulate situations that may occur when users upload images, such as incorrect angles, incomplete screenshots, and poor lighting, ensuring that the image-to-text (OCR) model can accurately extract text under various visual conditions.

[0028] Dialogue metadata consists of basic information describing the dialogue itself, such as: a unique session identifier for this dialogue, the precise timestamp of each message, the sender and receiver identifiers of the messages, and the session duration.

[0029] Text content features refer to features that reflect the substantive content of the dialogue, such as: clean text after cleaning, word sequences after word segmentation, and key entities extracted by the Named Entity Recognition (NER) model, such as names of people, organizations, products, amounts, and times.

[0030] Emotional characteristics refer to the features that reflect the customer's emotions in this conversation. For example, through sentiment analysis models, each customer or customer service statement can be judged and labeled with an emotional polarity tag, such as positive, negative or neutral, or an emotional intensity score, such as 0-1.

[0031] The following describes a specific implementation method for preprocessing text data in multimodal dialogue data: Data cleaning: Irrelevant information is removed from text data using rule-based matching and regular expressions. For example: automatically generated system messages are matched and deleted using a pre-defined list of system prompt keywords (e.g., "You have added this person as a friend," "Welcome to add me," etc.); emojis are removed by identifying and filtering characters or specific text symbols within the Unicode encoding range; web links are removed using regular expression pattern recognition; red envelope, transaction notification messages, and redundant content are identified and removed using keyword matching (e.g., "red envelope," "transfer," "um," "oh"). Duplicate messages are identified and merged by checking for complete consistency between two consecutive messages; and whitespace characters at the beginning and end of messages are removed using string trimming functions.

[0032] Special formats are handled using string parsing and regular expression techniques. For example, the time embedded in the text is converted into a unified timestamp format (such as 2023-10-01 10:00:00). For user nickname tags, the user nickname tags before the colon are identified by regular expressions and named as different identifiers (such as "User A").

[0033] Chinese word segmentation and part-of-speech tagging: After cleaning, the text is segmented using a Chinese word segmentation tool, dividing the sentence into independent word units. The part-of-speech tagging function of the Chinese analysis tool is then called to tag the part of speech (such as noun, verb, adjective) of each word, enabling more refined semantic analysis in the future, such as identifying keywords or descriptive content.

[0034] Stop word filtering: After Chinese word segmentation and part-of-speech tagging, independent word units are identified based on the Chinese stop word list, and words belonging to the Chinese stop word list are filtered out to reduce data noise and highlight key content. In one example, customized stop words (such as certain high-frequency but meaningless business jargon) can be added based on the characteristics of business areas such as finance and customer service.

[0035] Text standardization: Traditional Chinese characters in the text are uniformly converted to simplified Chinese characters to ensure consistency between text encoding and content. For pinyin abbreviations in specific scenarios (such as "wx" referring to "WeChat") or industry-specific abbreviations, a pre-defined mapping dictionary can be used for restoration. Furthermore, text correction tools based on pre-trained language models (such as BERT) can be used to detect and correct spelling errors and homophones in the text, improving text quality.

[0036] Structured annotation: Dialogue role separation has been completed by recognizing the user nickname tag before the colon using regular expressions. Now, the speaker identifier can be encapsulated with the corresponding text. By calling a pre-trained or domain-fine-tuned named entity recognition model, key entities such as person name, company name, product name, amount, and date are extracted from the text and their types are labeled. At the same time, a sentiment analysis model is used to analyze each sentence of text and output its sentiment polarity classification. Finally, all messages are organized according to a predetermined structure (such as JSON format) to form a structured record (e.g., "User A: Hello" → {"speaker": "A", "text": "Hello"}).

[0037] The following describes a specific implementation method for preprocessing speech data in multimodal dialogue data: First, the acquired raw audio data is converted into a common audio format to ensure compatibility with subsequent processing tools. After conversion, digital signal processing techniques or deep learning-based speech denoising models are used to denoise the audio, effectively filtering out background noise, current noise, and other interference, thus improving speech clarity. For longer voice messages, Voice Activity Detection (VAD) technology is used for intelligent segmentation, automatically cutting a continuous long speech into multiple independent speech segments separated by silence, decomposing them into more easily analyzed and transcribed segments. The denoised and segmented speech segments are then transcribed using Automatic Speech Recognition (ASR). Finally, the transcribed text data is processed according to the preprocessing steps outlined above.

[0038] In one example, a voiceprint recognition model can be used to extract acoustic feature vectors from speech segments. By comparing the voiceprint features of different speech segments, different speakers in a dialogue can be distinguished for role differentiation. A speech emotion recognition model can be used to directly extract emotion-related features from the speech signal, including but not limited to: pitch, speech rate, volume, and spectral features, to complement the emotion analysis results of the transcribed text.

[0039] The following describes a specific implementation method for preprocessing video data in multimodal dialogue data: First, the video file undergoes basic parsing to extract key metadata, including total video duration, encoding format, resolution, frame rate, and file size, to assess data quality and allocate computational resources for subsequent processing steps. Since video data contains both visual and audio information, an audio-video separation tool is used to extract the audio track from the video file and save it as a separate audio file. This audio file can then be processed according to the specific implementation method for preprocessing audio data described earlier.

[0040] The keyframe detection algorithm is used to extract representative image frames from the video stream, avoiding the massive computation required for frame-by-frame processing, while effectively capturing the core changes in the video content. Optical character recognition (OCR) is then performed on the image frames to extract the text information contained therein, and the text data is processed in the same way as the preprocessing process described above.

[0041] For attachments (such as images, documents, compressed files, etc.) included in multimodal dialogue data, content extraction and unified management are required. For document content, text content is extracted; for images, text information can be extracted using optical character recognition (OCR) technology. To avoid data redundancy and establish efficient retrieval associations, after content parsing, the hash value of each attachment file is calculated as its unique identifier. A structured metadata database is created to record key information for each attachment. Each metadata record includes at least: the attachment's hash identifier, original filename, file type, size, parsed text content, and its associated dialogue context information. The context information is linked using a session identifier and the timestamp of the attachment's appearance in the dialogue, ensuring that any attachment can be accurately located in its respective dialogue record.

[0042] Finally, the structured dialogue data extracted from text data, voice data, video data, and attachment information can be associated and stored in the dataset based on timestamps or session identifiers.

[0043] S102: Based on structured dialogue data, calculate basic dialogue evaluation indicators, which include at least one of the following: service efficiency indicators, service quality indicators, business capability indicators, compliance indicators, and customer stickiness indicators.

[0044] In this embodiment of the application, basic dialogue evaluation indicators can be directly calculated based on structured dialogue data. These basic dialogue evaluation indicators are a series of directly quantifiable metrics that reflect the core performance dimensions in the customer service process.

[0045] The calculation methods for service efficiency indicators, service quality indicators, business capability indicators, compliance indicators, and customer stickiness indicators will be explained one by one below.

[0046] Service efficiency metrics are used to quantify the timeliness of customer service personnel's response to customer needs. These include response time metrics and non-working-hour response rate metrics. Response time metrics refer to the time interval required for a customer service representative to respond to a single customer inquiry. This can be achieved by extracting adjacent "customer message timestamps" and "customer service message timestamps" from the timestamp field of structured dialogue data, calculating the time difference between these adjacent message pairs, and taking the median of these time differences as the response time metric. Non-working-hour response rate metrics refer to the proportion of messages replied to by customer service personnel outside of legal or agreed-upon working hours (such as nighttime or weekends) out of the total number of replied messages. This can be achieved by parsing the specific occurrence time of each customer service message from the timestamp field of structured dialogue data, determining whether it falls within a predefined "non-working-hour" interval, counting the number of customer service messages during non-working hours, and then dividing by the total number of customer service messages to obtain the non-working-hour response rate metric.

[0047] Service quality indicators (MMIs) assess customer service representatives' ability to resolve customer issues and include: emotional polarity score and problem resolution rate. Emotional polarity score refers to the overall emotional tone of a customer's statements in a conversation. It can be obtained by summarizing the emotional characteristics attributed to customer statements in structured dialogue data, counting the number of statements marked as "positive," and then dividing by the total number of customer statements. Problem resolution rate refers to the percentage of customer service conversations where problems are explicitly indicated as resolved (e.g., using keywords). It can be obtained by searching the text content characteristics attributed to customer service statements for predefined keywords confirming resolution (e.g., "resolved," "completed," "processed for you," etc.), counting the number of conversation rounds containing at least one such keyword, and then dividing by the total number of conversation rounds in which customer service representatives participated in responding.

[0048] Business capability metrics are used to measure customer service representatives' ability to identify business opportunities and accurately push products. These include: product recommendation accuracy and business keyword coverage. Product recommendation accuracy refers to the percentage of products or services recommended by customer service representatives that are ultimately adopted and converted by customers. This can be achieved by identifying the specific product recommended by customer service representatives (e.g., "financial product A") from the text content features of structured dialogue data, then matching this recommendation record with transaction records, counting the number of recommendations that resulted in associated transactions, and dividing that number by the total number of product recommendations made by customer service representatives. Business keyword coverage refers to the professionalism of core business terminology used in the dialogue. This can be achieved by counting the number of different types of business keywords appearing from the text content features of structured dialogue data. Business keywords can be identified by pre-defined keyword libraries related to core business (e.g., "interest rate" and "financial management").

[0049] Compliance metrics are used to monitor whether customer service conversations comply with laws, regulations, and internal risk control requirements. These include: the number of times sensitive words are triggered. The number of times sensitive words are triggered refers to the number of times customer service representatives use prohibited or cautiously used words in conversations. This can be achieved by pre-defining a compliance sensitive word library (such as "guaranteed principal," "high returns," "promise," etc.), then matching the text content of customer service statements against this library, and counting the number of times all successfully matched sensitive words appear.

[0050] Customer engagement metrics are used to evaluate customer service representatives' ability to proactively maintain customer relationships and promote ongoing interaction. These include: the percentage of conversations initiated proactively by customer service representatives and the customer response rate. The percentage of conversations initiated proactively by customer service representatives is the proportion of conversations initiated by them out of the total number of conversations. This can be achieved by extracting the initiator identifier for each independent conversation from the conversation metadata, counting the number of conversations initiated by the customer service representative's identifier, and then dividing by the total number of conversations. The customer response rate is the percentage of conversations in which customers respond to messages initiated by customer service representatives. This can be achieved by checking if the speaker of the next message in the conversation rounds where customer service representatives send messages is a customer, counting the number of rounds in which customers respond, and then dividing by the total number of rounds in which customer service representatives send messages.

[0051] S103: Based on structured dialogue data and basic dialogue evaluation metrics, customer satisfaction metrics, customer churn warning metrics, and potential service demand identification metrics are generated through machine learning models.

[0052] In this embodiment, basic indicators can only reflect the surface quantitative characteristics of services and cannot uncover the deeper information hidden in the dialogue. For example, customer satisfaction indicators not only depend on response speed, but are also related to emotional experience and problem-solving effectiveness. Customer churn risk and potential service needs cannot be directly judged by a single basic indicator. Therefore, by using machine learning models to integrate multi-dimensional data to generate higher-level indicators such as customer satisfaction indicators, customer churn warning indicators, and potential service need identification indicators, it is possible to more comprehensively and accurately reflect the real customer experience.

[0053] In one example, the machine learning model can be a multi-task learning model. This model has a shared low-level feature encoder (for processing the input structured dialogue data and basic metrics) and three independent output layers at the top, corresponding to the three tasks of predicting satisfaction metrics, predicting customer churn prediction metrics, and identifying potential service demand metrics. During training, the model simultaneously learns to perform these three related but distinct prediction tasks through a joint loss function. The shared low-level encoder learns general feature representations useful for multiple tasks from the structured dialogue data and basic metrics, thereby reducing the overall number of parameters and computational overhead.

[0054] In another example, the machine learning model consists of three independent, dedicated machine learning models: a customer satisfaction model, a customer churn warning model, and a potential service demand identification model, which are built and trained respectively.

[0055] S104: Input the basic dialogue evaluation indicators, customer satisfaction indicators, customer churn warning indicators, and potential service demand identification indicators into the service dialogue evaluation model to obtain the service dialogue evaluation results.

[0056] In this embodiment, basic indicators and advanced indicators reflect the performance in customer service dialogues from different dimensions. Through the service dialogue evaluation model, a comprehensive service dialogue evaluation result is generated, which can comprehensively and accurately reflect the actual service quality of the dialogue.

[0057] In one example, the service dialogue evaluation method of this application can be applied to business scenarios such as financial product recommendation, loan business consultation, complaint handling efficiency, and customer renewal rate. It should be noted that the service dialogue evaluation results obtained by this application also represent an assessment of the service capabilities of different customer service personnel; therefore, the service dialogue evaluation results of this application can be used to evaluate the performance and effectiveness of different customer service personnel.

[0058] In another example, after obtaining the service dialogue evaluation results, a dynamic dashboard can be generated to display the rating, ranking, and improvement suggestions in real time. It can also display and analyze the abnormal service dialogue evaluation results and their corresponding multimodal dialogue data and structured dialogue data to determine the cause of the service dialogue anomaly.

[0059] In one example, the service dialogue evaluation model can be a machine learning model, such as the Extreme Gradient Boosting Tree (XGBOOST) model. The XGBOOST model is a high-performance ensemble learning algorithm that can effectively capture the complex nonlinear relationships between input features (i.e., various indicators) to output a comprehensive service dialogue evaluation result, such as A, B, C, or D grades.

[0060] In one example, another specific implementation of step S104 is as follows: Based on the business objectives corresponding to the multimodal dialogue data, determine the first weight for each indicator; In the embodiments of this application, the core objectives are different for different business scenarios. For example, when the bank's quarterly focus is on "compliance inspection", the importance of compliance indicators should be higher than other indicators; when the focus is on "promotion of wealth management products", business capability indicators should be higher than other indicators.

[0061] In one example, a set of corresponding first weight templates for each specific business objective can be pre-configured. By identifying the currently effective business objective, the corresponding weight templates can be automatically matched and loaded, thereby quickly determining the first weight of each indicator in the current evaluation period.

[0062] Based on the time corresponding to the multimodal dialogue data, the second weight corresponding to each indicator is determined.

[0063] In this embodiment of the application, the quality of customer service dialogue is dynamic. Recent service performance is a better indicator of current customer service capabilities and service levels than historical service performance. Therefore, by setting a time decay factor, different importance weights can be assigned to the indicator data generated in different time windows.

[0064] In one example, a decay function (such as an exponential decay function) can be defined. For instance, the weight of conversation data from the past 30 days could be specified as 0.8, data from 30-60 days as 0.5, and data older than 60 days as 0.2. For each metric, if its calculation depends on historical data (such as conversation frequency in customer stickiness metrics), the data for each time segment will be multiplied by the corresponding time decay factor before being aggregated.

[0065] Based on the customer identifiers corresponding to the multimodal dialogue data, the third weights corresponding to each indicator are determined.

[0066] In this application embodiment, different customers have different value levels and service needs. For example, high-net-worth customers pay more attention to service quality and personalized experience, ordinary customers focus more on response speed and business processing efficiency, and potential customers pay more attention to the accuracy of demand identification. Therefore, based on different customer identifiers in the dialogue metadata, the third weight corresponding to each indicator can be dynamically determined.

[0067] In one example, differentiated indicator weight configurations can be pre-set for different customer segments (such as "high-net-worth clients," "ordinary clients," and "potential clients"). The system can query or determine the customer's segment based on the customer identifier in the conversation, and then automatically load the pre-set third weight configuration for that segment.

[0068] Based on at least one of the first weight, second weight, and third weight, the basic dialogue evaluation index, customer satisfaction index, customer churn warning index, and potential service demand identification index are weighted and summed to obtain the service dialogue evaluation result.

[0069] In one example, based on the actual application scenario, at least one of the first weight, second weight, and third weight is selected, and multiplied by the scores of the corresponding basic dialogue evaluation indicators, customer satisfaction indicators, customer churn warning indicators, and potential service demand identification indicators, and then summed to obtain the service dialogue evaluation result.

[0070] In one example, the final service dialogue evaluation result can also be determined by combining reward and penalty items, where reward and penalty items refer to adding or deducting points based on predefined key event rules. For example, bonus items may include "successfully identifying and converting high-value customers" (such as recommending private equity products and closing a deal); deduction items may include "compliance violations occurring and being complained about by customers" (such as using prohibited terms such as "guaranteed principal" or "promised returns").

[0071] The following is a formula for calculating service dialogue evaluation results: (1) in, Indicates the service dialogue evaluation results; Indicates the indicator sequence number; Indicates the total number of indicators; Indicates the indicator weight value; Indicates the index value; This indicates rewards and punishments.

[0072] In another example, if the first weight, the second weight, and the third weight are selected for calculation at the same time, the weights can be merged by superimposing the weight factors and normalizing them.

[0073] In another example, after obtaining the service dialogue evaluation results, the service dialogue evaluation results, along with the original basic dialogue evaluation indicators, customer satisfaction indicators, customer churn warning indicators, and potential service demand identification indicators, can be input into the XGBOOST model to correct the service dialogue evaluation results.

[0074] In this embodiment, the first weight is dynamically determined by business objectives, aligning the service dialogue evaluation results with current business priorities and preventing the evaluation results from becoming disconnected from business objectives, thereby improving the adaptability of the service dialogue evaluation results to the business. The second weight is dynamically determined by time, allowing the service dialogue evaluation results to reflect the dynamic changes in customer service capabilities in a timely manner, avoiding interference from outdated data, and enabling the service dialogue evaluation results to accurately capture the latest service level. The third weight is determined by customer identification, meeting the differentiated needs of customers with different values, and enabling the service dialogue evaluation results to match the actual experience of different customers.

[0075] In another example, to continuously optimize the response quality of the intelligent customer service model being trained, and to ensure that its outputs meet high-quality service standards and are adapted to actual business scenarios, the process after obtaining the service dialogue evaluation results also includes: From the multimodal dialogue data, target multimodal dialogue data with service dialogue evaluation results higher than the preset evaluation result threshold are selected to construct a reinforcement learning training sample set.

[0076] In this embodiment, the training effect of the intelligent customer service model depends on the quality of the training samples. If massive amounts of unfiltered dialogue data are used directly, they may contain poor, illegal, or low-quality dialogues, causing the intelligent customer service model to learn incorrect service logic or scripts. By selecting target multimodal dialogue data whose service dialogue evaluation results are higher than a preset evaluation result threshold (e.g., service dialogue evaluation result > 90 points or service dialogue evaluation result grade A), it can be ensured that the data in the reinforcement learning training sample set are examples that have been objectively evaluated and certified and meet high-quality service standards. This allows the intelligent customer service model to learn the conversational patterns of high-quality dialogue data.

[0077] In one example, the multimodal dialogue data used to build the sample set could be historical dialogues between human customer service representatives and customers, or dialogue records from previous versions of the intelligent customer service model.

[0078] For each target multimodal dialogue data, perform the following steps: Input customer inquiry data from the target multimodal dialogue data into the intelligent customer service model to be trained, and obtain customer service response data generated by the intelligent customer service model to be trained.

[0079] In this embodiment, the core of reinforcement learning is the interaction between the agent and the environment. Feedback is obtained through actual interaction, thereby learning and optimizing strategies to train the intelligent customer service model. In this application, the intelligent customer service model to be trained simulates real customer service, responding to specific customer questions and generating evaluable behavior. Customer inquiry data is separated from the target multimodal dialogue data and input into the intelligent customer service model to be trained. The model can automatically generate a text as customer service response data.

[0080] In one example, the intelligent customer service model to be trained could be a dialogue generation model based on deep learning.

[0081] New dialogue data to be evaluated is generated based on customer inquiry data and customer service response data. The process of preprocessing multimodal dialogue data to generate structured dialogue data is then returned, and new service dialogue evaluation results are regenerated.

[0082] In this embodiment, customer inquiry data and customer service response data are combined to form new dialogue data to be evaluated. Then, the process of processing multimodal dialogue data as described above to generate service dialogue evaluation results is reused to evaluate the dialogue data to be evaluated, thereby obtaining new service dialogue evaluation results. It should be noted that the new service dialogue evaluation results are the reward function values ​​in the traditional reinforcement learning model, used to update the parameter values ​​of the intelligent customer service model.

[0083] Determine whether the service dialogue evaluation results meet the preset training stop conditions.

[0084] In this embodiment, the service dialogue evaluation result can be judged to determine whether the service dialogue evaluation result meets the preset training stop condition by judging whether the service dialogue evaluation result reaches the preset evaluation result threshold. When the service dialogue evaluation result reaches the preset evaluation result threshold, it can be determined that the current intelligent customer service model can stably generate responses that meet the high-quality service standards and has ideal dialogue capabilities. Therefore, training can be stopped to prevent overfitting.

[0085] If the conditions are not met, adjust the parameters of the intelligent customer service model to be trained, and continue training the intelligent customer service model using the reinforcement learning training sample set until the preset training stopping condition is met, thus obtaining the trained intelligent customer service model.

[0086] In this embodiment of the application, if the service dialogue evaluation result does not reach the preset evaluation result threshold, it can be determined that the current intelligent customer service model cannot output a response that meets the high-quality service standard. Therefore, by adjusting the parameters of the intelligent customer service model to be trained and using the reinforcement learning training sample set to continue training the intelligent customer service model until the preset evaluation result threshold is reached, the trained intelligent customer service model is obtained.

[0087] In one example, the parameters of the intelligent customer service model being trained can be adjusted using deep reinforcement learning algorithms, such as proximal policy optimization algorithms.

[0088] In this embodiment, the core service dialogue evaluation results of this application are reused as the screening criteria to select high-quality target multimodal dialogue data and construct a reinforcement learning training sample set. This enables the intelligent customer service model to accurately learn the correct dialogue patterns, professional language, and effective solutions that conform to customer service dialogue standards. Furthermore, when training the intelligent customer service model, the service dialogue evaluation results are used as the reward function value to update the parameters of the intelligent customer service model. Since the service dialogue evaluation results can comprehensively and accurately reflect the actual service quality of the dialogue, the trained intelligent customer service model can comprehensively weigh multiple requirements such as service efficiency, professional capabilities, compliance, and customer experience to generate high-quality dialogue responses.

[0089] In this embodiment, compared to the prior art which relies on subjective scoring based on personal experience or identifies violations or abnormal dialogues based on preset keywords or simple indicators, resulting in an inability to comprehensively and accurately reflect the actual service quality of the dialogue, this application obtains multimodal dialogue data generated during customer service, preprocesses the multimodal dialogue data to generate structured dialogue data, and then calculates various basic dialogue evaluation indicators based on the structured dialogue data, including service efficiency indicators, service quality indicators, business capability indicators, compliance indicators, and customer stickiness indicators. Furthermore, based on the structured dialogue data and basic dialogue evaluation indicators, a machine learning model is used to generate higher-order indicators such as customer satisfaction indicators, customer churn warning indicators, and potential service demand identification indicators. The evaluation dimensions are more comprehensive and more in line with the actual needs of service dialogue business. Therefore, the service dialogue evaluation results obtained based on the service dialogue evaluation model can more comprehensively and accurately reflect the actual service quality of the dialogue.

[0090] Figure 2 A flowchart illustrating a customer service dialogue evaluation method provided in another embodiment of this application is shown. Figure 2 As shown above, in the above Figure 1 Based on the illustrated embodiment, the method may include the following steps prior to step S103: S201: Obtain the customer satisfaction training dataset. The customer satisfaction training dataset includes multiple customer satisfaction training data samples. Each customer satisfaction training data sample includes: sentiment feature samples, service efficiency index samples, service quality index samples, and customer satisfaction labels.

[0091] In this embodiment of the application, customer satisfaction labels can be marked by expert scoring or customer evaluation questionnaires. In one example, the customer satisfaction label can be a binary label (satisfied or dissatisfied), a multi-level label (1-5), or a continuous score (e.g., a continuous score between 0 and 1).

[0092] In one example, the service quality metric sample can be further taken from the problem resolution rate metric sample, and the service efficiency metric sample can be further taken from the response time metric sample.

[0093] S202: Based on the customer satisfaction training dataset, train the customer satisfaction model to be trained to obtain a well-trained customer satisfaction model.

[0094] In this embodiment, a logistic regression model can be used as the customer satisfaction model. The logistic regression model has non-linear fitting ability and can integrate emotional feature samples, service efficiency index samples, and service quality index samples to obtain the predicted value of customer satisfaction. Therefore, based on the customer satisfaction training dataset, the parameters of the customer satisfaction model to be trained are adjusted by minimizing the gap between the predicted value of customer satisfaction and the true label through optimization algorithms, such as gradient descent algorithm, so as to obtain a trained customer satisfaction model.

[0095] In one example, after obtaining a trained customer satisfaction model, emotional characteristic indicators, service efficiency indicators, and service quality indicators can be input into the trained customer satisfaction model to obtain customer satisfaction indicators.

[0096] In another example, customer satisfaction metrics can be calculated directly based on weights assigned to emotional characteristic metrics, service efficiency metrics, and service quality metrics.

[0097] In another example, a customer satisfaction model can consist of three parallel feature processing sub-networks, an attention fusion layer, and a prediction layer. The three parallel feature processing sub-networks are used for deep processing of sentiment features, service efficiency indicators, and service quality indicators, respectively. Each sub-network consists of a fully connected layer that maps the raw features to a high-dimensional semantic space. The outputs of the three sub-networks are then fed into the attention fusion layer, which automatically calculates and outputs a set of weights representing the relative importance of the three dimensions—"sentiment," "efficiency," and "quality"—to the current conversation's satisfaction judgment. For example, in a complaint conversation, the weight of the sentiment feature is automatically increased; in a complex business consultation, the weight of the service quality indicator is automatically increased. These dynamic weights are then used to perform a weighted sum of the three feature representations, resulting in a comprehensive feature vector that integrates multi-dimensional information. This comprehensive feature vector is then input into the subsequent fully connected layer for non-linear transformation, and finally, the predicted customer satisfaction value is obtained through the output layer.

[0098] In this embodiment of the application, by obtaining a customer satisfaction training dataset, the customer satisfaction model to be trained is trained using the customer satisfaction training dataset to obtain a trained customer satisfaction model. This allows the trained customer satisfaction model to deeply explore the non-linear relationship between emotional features, service efficiency, service quality and customer satisfaction, capture customer satisfaction in different business scenarios, and thus improve the comprehensiveness and accuracy of service dialogue evaluation results.

[0099] Figure 3 A flowchart illustrating a customer service dialogue evaluation method provided in another embodiment of this application is shown. Figure 3 As shown above, in the above Figure 1Based on the illustrated embodiment, the method may include the following steps prior to step S103: S301: Obtain the customer churn warning training dataset. The customer churn warning training dataset includes multiple customer churn warning training data samples. Each customer churn warning training sample includes: customer question frequency change characteristics, customer conversation interval duration change characteristics, negative sentiment change characteristics, and customer churn warning labels.

[0100] In this embodiment of the application, the customer question frequency, customer conversation interval duration, and negative sentiment quantity corresponding to multiple consecutive time windows can be calculated. Then, based on the customer question frequency, customer conversation interval duration, and negative sentiment quantity corresponding to multiple time windows, the characteristics of customer question frequency change, customer conversation interval duration change, and negative sentiment change can be obtained.

[0101] In another example, to maximize the model's autonomous learning ability, instead of pre-calculating complex changing features, the frequency of customer inquiries, the duration of customer conversations, and the amount of negative sentiment can be directly calculated for multiple consecutive time windows.

[0102] In one example, a customer churn warning label can be defined by setting an end point for the observation period for each customer sample. If, within a predefined future time period after this end point (e.g., 30 days), the customer does not generate any further proactive service conversations or has officially closed their account, the churn label is marked as "1" (positive example); otherwise, it is marked as "0" (negative example).

[0103] S302: Based on the customer churn warning training dataset, train the customer churn warning model to be trained to obtain a trained customer churn warning model.

[0104] In this embodiment, if the input is a pre-calculated sequence of changing features, i.e., the calculated features are the frequency of customer inquiries, the duration of customer conversations, and the negative sentiment, it can be directly input into a fully connected neural network or logistic regression model for training, thereby obtaining a trained customer churn warning model.

[0105] In one example, if the input is the original time series of "statistics," that is, the frequency of customer inquiries, the duration of customer conversations, and the number of negative sentiments corresponding to multiple consecutive time windows, then a Long Short-Term Memory (LSTM) network can be used as a customer churn prediction model. The LSTM will sequentially read the vector of [frequency of customer inquiries, duration of customer conversations, and number of negative sentiments] for each time window. Through its internal gating mechanism, the LSTM can automatically capture and remember long-term dependencies, and then output a customer churn probability between 0 and 1.

[0106] In another example, to further improve the accuracy of customer churn prediction, a graph convolutional neural network (GNN) can be superimposed before a long short-term memory (LSTM) network. Specifically, a heterogeneous graph of customer-customer interaction can be constructed, containing two types of nodes: "customer" nodes and "customer service" nodes. If a customer has a conversation with a customer service representative within a time window, an edge is established between them. Each edge can carry features of that conversation, such as the frequency of customer questions, the duration of the conversation interval, and the amount of negative sentiment. Through the message propagation mechanism of the GNN, feature information from all associated edges of the target customer node (i.e., historical interactions with different customer service representatives) can be aggregated. This allows the model to learn high-risk features hidden in the interaction network, such as "the customer expressed dissatisfaction in interactions with multiple customer service representatives." Subsequently, the customer node embedding features obtained from the GNN are fused with the behavioral temporal features obtained from an LSTM and input into the final classifier for prediction.

[0107] In this embodiment, by acquiring a customer churn warning training dataset, the customer churn warning model to be trained is trained using the customer churn warning training dataset to obtain a trained customer churn warning model. The trained customer churn warning model can deeply capture the temporal trend of behavioral changes in customer dialogue, thereby identifying the customer churn risk score, and thus improving the comprehensiveness and accuracy of service dialogue evaluation results.

[0108] Figure 4 A flowchart illustrating a customer service dialogue evaluation method provided in another embodiment of this application is shown. Figure 4 As shown above, in the above Figure 1 Based on the illustrated embodiment, the method may include the following steps prior to step S103: S401: Obtain the potential service demand identification training dataset. The potential service demand identification training dataset includes multiple potential service demand identification training samples. Each potential service demand identification training sample includes: customer demand feature samples and business transaction record labels.

[0109] In this embodiment, customer demand characteristics are all business-related keywords appearing in the conversation (such as interest rates, studying abroad, and mortgages). The business transaction record label is determined by whether a new business transaction strongly related to the topic of the conversation occurred within a preset specific time period (e.g., 7 days) after the conversation (e.g., mentioning "cash flow" followed by a loan; mentioning "children studying abroad" followed by foreign exchange purchases or student loans). If yes, it is marked as 1 (positive sample, representing a genuine business opportunity); otherwise, it is marked as 0 (negative sample).

[0110] In another example, customer demand feature samples may also include: intent vectors and contextual dialogue vectors. The intent vector can be generated by using a pre-trained intent recognition model to encode conversational sentences containing business keywords into a fixed-dimensional semantic vector; the contextual dialogue vector is generated by inputting the entire dialogue (multi-turn interactions between the customer and customer service) into a sentence encoding model (BERT), and taking the output vector at the [CLS] position as the deep semantic representation of the entire dialogue context.

[0111] S402: Based on the potential service demand identification training dataset, train the potential service demand identification model to be trained to obtain a trained potential service demand identification model.

[0112] In this embodiment of the application, a gradient boosting decision tree model can be used as a potential service demand identification model. Based on the potential service demand identification training dataset, the gradient boosting decision tree model can be trained to obtain a trained potential service demand identification model.

[0113] In one example, after obtaining a trained potential service demand identification model, keywords, intent vectors, and contextual dialogue vectors from the dialogue content can be input into the trained potential service demand identification model to obtain potential service demand identification metrics.

[0114] In this embodiment of the application, by obtaining a potential service demand identification training dataset, the potential service demand identification model to be trained is trained using the potential service demand identification training dataset to obtain a trained potential service demand identification model. This allows the trained potential service demand identification model to deeply integrate the demand features in multimodal dialogue, accurately capture customer needs, and thus identify potential service demand identification capabilities, thereby improving the comprehensiveness and accuracy of service dialogue evaluation results.

[0115] Based on the customer service dialogue evaluation method provided in the above embodiments, this application also provides a specific implementation of a customer service dialogue evaluation device. Please refer to the following embodiments.

[0116] First see Figure 5 , Figure 5 The diagram shows a customer service dialogue evaluation device 500 provided in an embodiment of this application. The customer service dialogue evaluation device 500 provided in this embodiment includes: an acquisition module 501, a calculation module 502, a first processing module 503, and a second processing module 504.

[0117] The acquisition module 501 is used to acquire multimodal dialogue data generated during customer service, preprocess the multimodal dialogue data, and generate structured dialogue data; wherein, the multimodal dialogue data includes at least one of text data, voice data, and video data, and the structured dialogue data includes at least two of dialogue metadata, text content features, and sentiment features; The calculation module 502 is used to calculate basic dialogue evaluation indicators based on structured dialogue data. The basic dialogue evaluation indicators include at least one of the following: service efficiency indicators, service quality indicators, business capability indicators, compliance indicators, and customer stickiness indicators. The first processing module 503 is used to generate customer satisfaction indicators, customer churn warning indicators, and potential service demand identification indicators based on structured dialogue data and basic dialogue evaluation indicators through machine learning models. The second processing module 504 is used to input basic dialogue evaluation indicators, customer satisfaction indicators, customer churn warning indicators and potential service demand identification indicators into the service dialogue evaluation model to obtain service dialogue evaluation results.

[0118] In one example, the second processing module 504 includes: The determination submodule is used to determine the first weight of each indicator based on the business objectives corresponding to the multimodal dialogue data; The determination submodule is used to determine the second weight of each indicator based on the time corresponding to the multimodal dialogue data; The determination submodule is used to determine the third weight corresponding to each indicator based on the customer identifier corresponding to the multimodal dialogue data; The weighted summation submodule is used to perform weighted summation on the basic dialogue evaluation index, customer satisfaction index, customer churn warning index and the potential service demand identification index based on at least one of the first weight, second weight and third weight, so as to obtain the service dialogue evaluation result.

[0119] In one example, the customer service dialogue assessment device 500 also includes: The first acquisition module is used to acquire the customer satisfaction training dataset. The customer satisfaction training dataset includes multiple customer satisfaction training data samples. Each customer satisfaction training data sample includes: sentiment feature samples, service efficiency index samples, service quality index samples, and customer satisfaction labels. The first training module is used to train the customer satisfaction model to be trained based on the customer satisfaction training dataset, so as to obtain a trained customer satisfaction model.

[0120] In one example, the customer service dialogue assessment device 500 also includes: The second acquisition module is used to acquire the customer churn warning training dataset. The customer churn warning training dataset includes multiple customer churn warning training data samples. Each customer churn warning training sample includes: customer question frequency change characteristics, customer conversation interval duration change characteristics, negative sentiment change characteristics, and customer churn warning labels. The second training module is used to train the customer churn warning model based on the customer churn warning training dataset, and obtain the trained customer churn warning model.

[0121] In one example, the customer service dialogue assessment device 500 also includes: The third acquisition module is used to acquire the potential service demand identification training dataset. The potential service demand identification training dataset includes multiple potential service demand identification training samples. Each potential service demand identification training sample includes: customer demand feature samples and business transaction record labels. The third training module is used to train the potential service demand identification model based on the potential service demand identification training dataset, and obtain the trained potential service demand identification model.

[0122] In one example, the customer service dialogue assessment device 500 also includes: The filtering module is used to filter out target multimodal dialogue data from multimodal dialogue data whose service dialogue evaluation results are higher than the preset evaluation result threshold, and to construct a reinforcement learning training sample set. For each target multimodal dialogue data, perform the following steps: The input module is used to input customer inquiry data from the target multimodal dialogue data into the intelligent customer service model to be trained, and obtain customer service response data generated by the intelligent customer service model to be trained. The generation module is used to generate new dialogue data to be evaluated based on customer inquiry data and customer service response data, and return the steps of preprocessing multimodal dialogue data to generate structured dialogue data, and regenerate new service dialogue evaluation results. The judgment module is used to determine whether the service dialogue evaluation results meet the preset training stop conditions; The fourth training module is used to adjust the parameters of the intelligent customer service model to be trained when the conditions are not met, and to continue training the intelligent customer service model using the reinforcement learning training sample set until the preset training stopping condition is met, thus obtaining the trained intelligent customer service model.

[0123] The various modules of the customer service dialogue evaluation device provided in this application embodiment can achieve Figures 1 to 4 It provides functionality for each step of the customer service dialogue evaluation method and achieves the corresponding technical effects. For the sake of brevity, it will not be elaborated here.

[0124] Figure 6 A schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application is shown.

[0125] An electronic device may include a processor 601 and a memory 602 storing computer program instructions.

[0126] Specifically, the processor 601 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0127] Memory 602 may include mass storage for data or instructions. For example, and not limitingly, memory 602 may include a hard disk drive (HDD), a floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. In one instance, memory 602 may include removable or non-removable (or fixed) media, or memory 602 may be a non-volatile solid-state memory.

[0128] In one instance, memory 602 may be read-only memory (ROM). In one instance, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0129] Memory 602 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, generally, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this disclosure.

[0130] The processor 601 implements a customer service dialogue evaluation method in the above-described embodiment by reading and executing computer program instructions stored in the memory 602.

[0131] In one example, the electronic device may also include a communication interface 606 and a bus 604. Wherein, as... Figure 6 As shown, the processor 601, memory 602, and communication interface 606 are connected through bus 604 and complete communication with each other.

[0132] The communication interface 606 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0133] Bus 604 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 604 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.

[0134] The customer service dialogue evaluation method described in the above embodiments can be implemented using a computer-readable storage medium. This computer storage medium stores computer program instructions; when executed by a processor, these computer program instructions implement any of the customer service dialogue evaluation methods described in the above embodiments.

[0135] This application also provides a computer program product, including a computer program that, when executed by a processor, implements any of the customer service dialogue evaluation methods described in the above embodiments.

[0136] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0137] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0138] The above are merely specific embodiments of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A customer service dialogue evaluation method, characterized in that, include: The system acquires multimodal dialogue data generated during customer service, preprocesses the multimodal dialogue data, and generates structured dialogue data. The multimodal dialogue data includes at least one of text data, voice data, and video data, and the structured dialogue data includes at least two of dialogue metadata, text content features, and sentiment features. Based on the structured dialogue data, basic dialogue evaluation indicators are calculated, including at least one of the following: service efficiency indicators, service quality indicators, business capability indicators, compliance indicators, and customer stickiness indicators. Based on the structured dialogue data and the basic dialogue evaluation metrics, a customer satisfaction metric, a customer churn warning metric, and a potential service demand identification metric are generated through a machine learning model. The service dialogue evaluation indicators, customer satisfaction indicators, customer churn warning indicators, and potential service demand identification indicators are input into the service dialogue evaluation model to obtain the service dialogue evaluation results.

2. The method according to claim 1, characterized in that, The step of inputting the basic dialogue evaluation indicators, the customer satisfaction indicators, the customer churn warning indicators, and the potential service demand identification indicators into the service dialogue evaluation model to obtain service dialogue evaluation results includes: Based on the business objectives corresponding to the multimodal dialogue data, determine the first weight corresponding to each indicator; Based on the time corresponding to the multimodal dialogue data, determine the second weight corresponding to each indicator; Based on the customer identifiers corresponding to the multimodal dialogue data, the third weights corresponding to each indicator are determined. Based on at least one of the first weight, the second weight, and the third weight, the basic dialogue evaluation index, the customer satisfaction index, the customer churn warning index, and the potential service demand identification index are weighted and summed to obtain the service dialogue evaluation result.

3. The method according to claim 1, characterized in that, The machine learning model is a customer satisfaction model; Before generating customer satisfaction metrics, customer churn warning metrics, and potential service demand identification metrics respectively through machine learning models based on the structured dialogue data and the basic dialogue evaluation metrics, the process also includes: Obtain a customer satisfaction training dataset, which includes multiple customer satisfaction training data samples. Each customer satisfaction training data sample includes: sentiment feature samples, service efficiency index samples, service quality index samples, and customer satisfaction labels. Based on the customer satisfaction training dataset, the customer satisfaction model to be trained is trained to obtain a well-trained customer satisfaction model.

4. The method according to claim 1, characterized in that, The machine learning model is a customer churn early warning model; Before generating customer satisfaction metrics, customer churn warning metrics, and potential service demand identification metrics respectively through machine learning models based on the structured dialogue data and the basic dialogue evaluation metrics, the process also includes: Obtain a customer churn warning training dataset, which includes multiple customer churn warning training data samples. Each customer churn warning training sample includes: customer question frequency change characteristics, customer conversation interval duration change characteristics, negative sentiment change characteristics, and customer churn warning labels. Based on the customer churn warning training dataset, the customer churn warning model to be trained is trained to obtain a trained customer churn warning model.

5. The method according to claim 1, characterized in that, The machine learning model is a potential service demand identification model; Before generating customer satisfaction metrics, customer churn warning metrics, and potential service demand identification metrics respectively through machine learning models based on the structured dialogue data and the basic dialogue evaluation metrics, the process also includes: Obtain a potential service demand identification training dataset, which includes multiple potential service demand identification training samples. Each potential service demand identification training sample includes: customer demand feature samples and business transaction record labels. Based on the potential service demand identification training dataset, the potential service demand identification model to be trained is trained to obtain a trained potential service demand identification model.

6. The method according to claim 1, characterized in that, Also includes: From the multimodal dialogue data, target multimodal dialogue data whose service dialogue evaluation results are higher than a preset evaluation result threshold are selected to construct a reinforcement learning training sample set; For each target multimodal dialogue data, perform the following steps: The customer inquiry data in the target multimodal dialogue data is input into the intelligent customer service model to be trained, and the customer service response data generated by the intelligent customer service model to be trained is obtained. Based on the customer inquiry data and the customer service response data, new dialogue data to be evaluated is generated, and the process of preprocessing the multimodal dialogue data to generate structured dialogue data is returned to regenerate new service dialogue evaluation results. Determine whether the service dialogue evaluation result meets the preset training stop condition; If the conditions are not met, the parameters of the intelligent customer service model to be trained are adjusted, and the intelligent customer service model is continued to be trained using the reinforcement learning training sample set until the preset training stopping condition is met, thus obtaining the trained intelligent customer service model.

7. A customer service dialogue evaluation device, characterized in that, The device includes: The acquisition module is used to acquire multimodal dialogue data generated during customer service, preprocess the multimodal dialogue data, and generate structured dialogue data; wherein the multimodal dialogue data includes at least one of text data, voice data, and video data, and the structured dialogue data includes at least two of dialogue metadata, text content features, and sentiment features; The calculation module is used to calculate basic dialogue evaluation indicators based on the structured dialogue data. The basic dialogue evaluation indicators include at least one of the following: service efficiency indicators, service quality indicators, business capability indicators, compliance indicators, and customer stickiness indicators. The first processing module is used to generate customer satisfaction indicators, customer churn warning indicators, and potential service demand identification indicators respectively based on the structured dialogue data and the basic dialogue evaluation indicators through a machine learning model. The second processing module is used to input the basic dialogue evaluation indicators, the customer satisfaction indicators, the customer churn warning indicators, and the potential service demand identification indicators into the service dialogue evaluation model to obtain the service dialogue evaluation results.

8. An electronic device, characterized in that, The device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements a customer service dialogue evaluation method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement a customer service dialogue evaluation method as described in any one of claims 1-6.

10. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device is able to perform a customer service dialogue evaluation method as described in any one of claims 1-6.

Images