Voice processing method and device, electronic equipment and storage medium

Abstract

The invention discloses a voice processing method and device, electronic equipment and a storage medium, and relates to the field of financial science and technology and the field of artificial intelligence. The method comprises the following steps: acquiring a voice signal of a first object and a voice signal of a second object; performing feature processing operation on the voice signal of the first object and the voice signal of the second object to obtain a voice feature sequence of the first object and a voice feature sequence of the second object; based on the same time axis, combining the voice feature sequence of the first object and the voice feature sequence of the second object into a target feature sequence; determining a time sequence association relationship between the voice features of the first object and the voice features of the second object according to the target feature sequence; and determining a service quality score of the first object to the second object according to the time sequence association relationship. The technical problem that the detection accuracy of the customer voice service quality is low in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the fields of financial technology and artificial intelligence, and specifically, to a speech processing method, device, electronic device, and storage medium. Background Art

[0002] High-quality customer service can significantly improve customer satisfaction and establish a positive corporate image. Customer satisfaction ratings are a key indicator of a company's service quality. However, how to objectively and accurately score telephone service has always been a core issue in satisfaction evaluation.

[0003] Currently, traditional telephone service evaluation methods rely primarily on subjective customer ratings, which are susceptible to emotional fluctuations, lack objectivity, and have limited support for business decision-making. While some existing methods analyze the voice content of bank customer service calls for quality control, these methods often focus solely on the voice content itself, ignoring other important factors. This results in low accuracy in assessing the quality of customer service calls.

[0004] To address the above problems, no effective and comprehensive solution has been proposed yet. Summary of the Invention

[0005] The embodiments of the present application provide a voice processing method, apparatus, electronic device, and storage medium to at least solve the technical problem of low accuracy in detecting customer voice service quality in the prior art.

[0006] According to one aspect of an embodiment of the present application, a speech processing method is provided, including: with authorization from a first object and a second object, collecting a speech signal of the first object and a speech signal of the second object, wherein the first object provides financial consulting services to the second object; performing feature processing operations on the speech signal of the first object and the speech signal of the second object, respectively, to obtain a speech feature sequence of the first object and a speech feature sequence of the second object; based on the same time axis, combining the speech feature sequence of the first object and the speech feature sequence of the second object into a target feature sequence; determining a temporal correlation relationship between the speech features of the first object and the speech features of the second object according to the target feature sequence; and determining a service quality score of the first object to the second object according to the temporal correlation relationship.

[0007] Optionally, the feature processing operation includes: deleting the noise signal in the speech signal to obtain an intermediate speech signal; detecting the signal strength of each signal segment in the intermediate speech signal; deleting the signal segments in the intermediate speech signal whose signal strength is lower than a preset threshold to obtain a target speech signal; extracting the sound features and semantic features in the target speech signal; and combining the extracted sound features and semantic features into a speech feature sequence.

[0008] Optionally, deleting the noise signal in the speech signal to obtain an intermediate speech signal includes: determining the speech spectrum information of the speech signal; determining the noise spectrum information in the speech spectrum information by using Fourier transform; deleting the noise spectrum information from the speech spectrum information, and using the signal corresponding to the remaining spectrum information as the intermediate speech signal.

[0009] Optionally, based on the same time axis, the speech feature sequence of the first object and the speech feature sequence of the second object are combined into a target feature sequence, including: based on the same time axis, performing an alignment operation on the speech feature sequence of the first object and the speech feature sequence of the second object, wherein the alignment operation is used to ensure that the features at each time point come from the speech signal at the same moment; after completing the alignment operation, at each time point, the speech features of the first object and the second object are spliced to obtain the target feature sequence.

[0010] Optionally, the service quality score of the first object for the second object is determined based on the temporal association relationship, including: based on the temporal association relationship, taking the voice features of the second object at time T+1 and the voice features of the first object at time T as a feature combination, wherein time T is any time when the first object speaks, and time T+1 is the time when the second object speaks for the first time after time T; and determining the service quality score of the first object for the second object based on the feature combination.

[0011] Optionally, determining a service quality score of a first object for a second object based on a feature combination includes: inputting the feature combination into a target model, wherein the target model includes an input layer, a convolutional layer, a densely connected layer, and an output layer, wherein adjacent network layers of the target model are spliced in a fully connected manner; the input layer of the target model is used to receive the feature combination and transmit the feature combination to multiple convolutional layers, each convolutional layer is used to determine local correlation features between feature combinations; the densely connected layer is used to integrate the local correlation features output by multiple convolutional layers into global correlation features; utilizing the prior knowledge learned by the target model in the model training phase to determine the service quality score of the first object for the second object based on the global correlation features; and outputting the service quality score through the output layer of the target model.

[0012] Optionally, the speech processing method also includes: when there are N feature combinations, determining a service quality score based on each feature combination to obtain N service quality scores, where N is an integer greater than 1; performing weighted average calculation on the N service quality scores to obtain a target service quality score of the first object to the second object, where the later the speech time corresponding to the speech feature in the feature combination, the higher the weight value of the feature combination.

[0013] Optionally, the training process of the target model includes the following steps: obtaining a training data set, wherein the training data set includes the customer service's voice signal and the customer's voice signal; annotating the voice signals in the training data set, wherein the annotated content of the voice signal includes the customer's rating of the service quality; based on the training data set, extracting the customer service's voice feature sequence and the customer's voice feature sequence; combining the customer service's voice feature sequence and the customer's voice feature sequence into a training feature sequence; and performing multiple iterative training on the neural network based on the training feature sequence and the annotated content of the voice signal until the target model is obtained through training.

[0014] According to another aspect of an embodiment of the present application, a voice processing device is also provided, including: a collection unit, for collecting the voice signal of the first object and the voice signal of the second object with the authorization of the first object and the second object, wherein the first object provides financial consulting services to the second object; a first processing unit, for performing feature processing operations on the voice signal of the first object and the voice signal of the second object, respectively, to obtain a voice feature sequence of the first object and a voice feature sequence of the second object; a second processing unit, for combining the voice feature sequence of the first object and the voice feature sequence of the second object into a target feature sequence based on the same time axis; a first determination unit, for determining the temporal correlation relationship between the voice features of the first object and the voice features of the second object according to the target feature sequence; and a second determination unit, for determining the service quality score of the first object to the second object based on the temporal correlation relationship.

[0015] According to another aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program runs, the device where the computer-readable storage medium is located executes the above-mentioned speech processing method.

[0016] According to another aspect of an embodiment of the present application, an electronic device is also provided, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors execute the above-mentioned speech processing method.

[0017] As can be seen from the above content, the present application collects the voice signal of the first object and the voice signal of the second object, wherein the first object provides financial consulting services to the second object; performs feature processing operations on the voice signal of the first object and the voice signal of the second object respectively to obtain the voice feature sequence of the first object and the voice feature sequence of the second object; based on the same time axis, combines the voice feature sequence of the first object and the voice feature sequence of the second object into a target feature sequence; determines the temporal correlation relationship between the voice features of the first object and the voice features of the second object according to the target feature sequence; and determines the service quality score of the first object to the second object according to the temporal correlation relationship.

[0018] In this application, a dual-channel emotion analysis method is adopted to achieve the purpose of finely identifying the emotional changes and temporal correlation in the interaction between the two parties by fusing the voice feature sequences of the first object and the second object, thereby realizing the technical effect of multi-dimensional service quality scoring, and further solving the technical problem of low accuracy in detecting customer voice service quality in the existing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 is a flowchart of an optional voice processing method according to an embodiment of the present application;

[0021] Figure 2 is a flowchart of another optional voice processing method according to an embodiment of the present application;

[0022] Figure 3 is a schematic diagram of an optional speech processing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] It should also be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) collected by this application are information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse. For example, an interface is set up between this system and relevant users or institutions. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or institution through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or institution.

[0026] According to an embodiment of the present application, an embodiment of a speech processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0027] Optionally, according to an embodiment of the present application, a speech processing system (hereinafter referred to as the system) is provided as the execution subject of the speech processing method of the embodiment of the present application, wherein the system can be a software system or an embedded system combining software and hardware. Of course, the method execution subject in the embodiment of the present application can also be other forms of execution subjects, such as devices, equipment, etc. Those skilled in the art should know that this application does not specifically limit the specific form of expression of the method execution subject.

[0028] Figure 1 is a flow chart of an optional voice processing method according to an embodiment of the present application, such as Figure 1 As shown, the method includes the following steps:

[0029] Step S101: collecting a speech signal of a first subject and a speech signal of a second subject.

[0030] In step S101 , a first object provides financial consulting services to a second object.

[0031] Optionally, the first object may refer to a financial consulting service provider (eg, a bank customer service representative, a sales consultant, or other service personnel); the second object may refer to a financial consulting service recipient (ie, a client or user).

[0032] Optionally, the voice signal collection should cover as many financial consulting service scenarios as possible, including but not limited to account inquiries, loan inquiries, credit card services, investment advice, etc., to ensure the broad applicability and robustness of the model. Specifically, during the voice signal collection process, the system can focus on those voice signals that directly reflect the second subject's emotional experience.

[0033] Optionally, given the potential for background noise and signal intermittency during financial advisory services, the system should rigorously monitor signal clarity and integrity. Only when the voice signal quality meets a pre-set standard will it be included in the dataset, facilitating subsequent emotional feature extraction.

[0034] Optionally, before collecting the second object's data (such as voice signals, etc.), it is first necessary to obtain the second object's explicit authorization, including: informing the second object of the purpose of data collection, how to use it, and how to protect its privacy, and clearly publicizing the privacy policy to the second object, including data collection, use, storage and deletion policies, and how the second object can exercise its rights, such as accessing, correcting or deleting its own data. The second object has the right to choose whether to agree to the collection and use of data. In addition, when collecting the second object's data, the system will use encryption technology to ensure the security of the data during transmission and storage. The use of the second object's data will strictly comply with relevant laws and regulations and the scope of the second object's authorization. The second object's data will be stored in a secure database, and regular security audits will be conducted to ensure the security of the data and the effectiveness of privacy protection measures. When the second object's data is no longer needed, the system will securely delete the second object's data according to the second object's requirements and the provisions of laws and regulations to ensure that the data cannot be restored.

[0035] Optionally, unlike traditional quality inspection that only focuses on the voice expression of the first object, collecting two-way emotional signals, that is, capturing the voice emotions of the first and second objects at the same time, can fully reflect the real situation of service interaction, significantly improve the comprehensiveness and objectivity of service quality evaluation, and lay the foundation for accurately identifying two-way emotional dynamics.

[0036] Step S102 : performing feature processing operations on the speech signal of the first object and the speech signal of the second object respectively to obtain a speech feature sequence of the first object and a speech feature sequence of the second object.

[0037] Optionally, during feature processing and subsequent operations on the voice signal data, the system may desensitize the data to remove or replace information that could identify the second subject. Furthermore, when processing and analyzing the second subject's data, the system ensures that all operations are performed in a secure environment to prevent data leakage or unauthorized access.

[0038] Optionally, in order to improve the clarity of the speech signal, it is necessary to reduce or eliminate the characteristics of the speech signal. The system first analyzes the signal segment by segment to obtain a set of local "spectra" of the signal, that is, the spectrum of the signal at each moment in the time domain, and then performs Fourier transform on the instantaneous spectrum to obtain the noise spectrum and speech spectrum. Finally, the noise spectrum is subtracted from the speech spectrum to obtain the denoised speech signal. The above feature processing can improve the clarity of the speech signal by removing background noise, ensuring that the target model (dense convolutional neural network model) can receive pure speech information and avoid recognition errors caused by external interference.

[0039] Optionally, the system can further focus on the active part of the audio signal (high-energy audio clips), that is, the period of time with greater speech intensity. By performing energy detection on the audio and screening out high-energy clips, the system can effectively filter out irrelevant or low-information parts, thereby improving the efficiency and accuracy of emotion recognition. Specifically, first, the system obtains the root mean square value of the audio signal, and then sets an energy threshold. If the audio clip is higher than the threshold, it is judged to be a high-energy part, that is, the target audio. Among them, the time length of the target audio is set to be consistent.

[0040] Optionally, after denoising and selecting high-energy segments, the system can apply advanced audio analysis techniques, such as MFCC (Mel Frequency Cepstral Coefficients), to extract features that contain emotional information. These features are organized into speech feature sequences, representing the emotional expression patterns of the first and second subjects, respectively.

[0041] Optionally, when extracting speech features, it's important to note that some features are particularly useful for capturing variations in the timbre of speech, while others (such as fundamental frequency) can reveal a speaker's state of tension or relaxation. By properly combining these features, a feature set can be constructed that comprehensively reflects emotional states.

[0042] Optionally, the feature processing pipeline can not only clean up the raw speech but also convert it into a structured sequence of speech features.

[0043] Step S103 : Based on the same time axis, the speech feature sequence of the first object and the speech feature sequence of the second object are combined into a target feature sequence.

[0044] Alternatively, in financial advisory services, the speech of the first and second speakers alternate, forming a natural time sequence. Therefore, the system must first ensure that the sampling and processing of the speech data of both parties are performed on the same timeline. Specifically, the system must precisely align the speech feature points in both directions so that at any given time point, the system can simultaneously analyze the speech features of the first and second speakers.

[0045] Optionally, the system extracts features from the speech signals of the first subject and the second subject respectively. Common features include but are not limited to MFCC, speaking speed, pitch change, number of pauses, etc. These features can reflect the speaker's emotional state and psychological tendency.

[0046] Optionally, after feature extraction, the system combines the first participant's speech feature sequence with the second participant's speech feature sequence to generate a target feature sequence that comprehensively reflects the emotional dynamics of both parties during the financial advisory service. This ensures that the target model simultaneously considers the emotional information of both parties, improving the accuracy and comprehensiveness of emotion recognition.

[0047] Optionally, the construction of the target feature sequence fully considers the diversity and complexity of the second-party's emotional expressions in different service scenarios. For example, in a complaint handling scenario, even a calm tone of the second-party should be considered a potential signal of dissatisfaction; whereas in a consulting service, a similar tone might be interpreted as neutral or satisfied. Therefore, the system needs to flexibly adjust the emotion mapping strategy based on the service type to adapt to the emotion recognition needs of specific situations.

[0048] Optionally, through the above steps, the system can create a comprehensive feature library containing the emotional information of both parties, which not only enhances the accuracy of emotion recognition, but also promotes a deeper understanding of the service interaction context, and helps to improve the comprehensiveness and objectivity of service quality evaluation.

[0049] Step S104: determining a temporal correlation relationship between the speech features of the first object and the speech features of the second object according to the target feature sequence.

[0050] Optionally, the system can analyze target feature sequences to identify the mutual influence of the emotional expressions of the first and second subjects. For example, it can determine whether the first subject promptly soothed the second subject's anxiety when asking a question, or whether the first subject's patient tone when explaining a policy received positive feedback from the second subject. This analysis focuses not only on individual emotions but also on the transmission and response of emotions, helping to understand the deeper meaning of service interactions.

[0051] Alternatively, during financial consulting services, specific phrases or events may trigger emotional changes, such as the second party's dissatisfaction upon hearing a fee explanation, or their satisfaction after the issue is resolved. The system can identify these triggers through time-series correlation analysis, providing data support for process optimization.

[0052] Step S105 : determining a service quality score of the first object to the second object according to the temporal association relationship.

[0053] Optionally, through emotion recognition technology, the system can build a more three-dimensional service quality evaluation system, which not only takes into account the final satisfaction score of the second object, but also deeply analyzes the dynamic trajectory of the customer service's emotional expression and the second object's emotional response during the service process.

[0054] Optionally, after obtaining the service quality score of the first object for the second object, calibration may be performed in combination with the manual review result to obtain a final quality score.

[0055] From the above content, it can be seen that this application adopts a dual-channel emotion analysis method, and by fusing the voice feature sequences of the first object and the second object, it achieves the purpose of finely identifying the emotional changes and temporal correlation in the interaction between the two parties, thereby realizing the technical effect of multi-dimensional service quality scoring, and thus solving the technical problem of low accuracy in detecting customer voice service quality in the existing technology.

[0056] Optionally, Figure 2 is a flow chart of another optional voice processing method according to an embodiment of the present application, such as Figure 2 As shown, first, the system collects the voice signals of the first object and the second object. These signals can come from different service scenarios, such as customer service, online consultation, etc. Then, the system can establish a module-voice-evaluation voice data set. Combined with the second object's rating of the service quality in the service scenario, the system can classify the second object's voice signal data into different emotional categories, such as satisfaction, dissatisfaction, neutrality, etc. At the same time, the second object data without evaluation is eliminated to ensure the relevance and accuracy of the data set. In addition, considering that there may be significant differences in the emotional expression patterns between different service modules, the system's classification criteria can be flexibly adjusted to be closer to the emotional dynamics of the actual service scenario. For example, in the scenario of complaint handling, the system can set a lower threshold to determine the dissatisfied voice features, so as to improve the sensitivity of identifying dissatisfaction.

[0057] The system then manually evaluates and filters the collected speech signal data to ensure that the dataset contains a variety of emotional categories, such as anger, frustration, and happiness. This helps identify speech data with rich emotions and clear tonal expressions, while eliminating invalid or inaccurate data to enhance the reliability of the target model. Furthermore, the system balances the emotional categories in the dataset, ensuring that each category has sufficient and equally representative samples.

[0058] Next, the system preprocesses the speech signal data. First, the system denoises the speech signal to improve the clarity of the speech signal. The system can first analyze the signal segment by segment to obtain a set of local "spectra" of the signal, that is, the spectrum of the signal at each moment in the time domain, and then perform Fourier transform on the instantaneous spectrum to obtain the noise spectrum and speech spectrum. Finally, the speech spectrum is subtracted from the noise spectrum to obtain the denoised speech signal. The system can then trim invalid content in the audio, such as blank parts or parts with a flat tone, and retain the high-energy parts. By calculating the root mean square value of the audio signal and setting the energy threshold, the target audio (audio clips above the set energy threshold) is screened out to enhance information readability and improve model recognition efficiency.

[0059] Next, the system constructs a dense convolutional neural network model (i.e., the target model) consisting of an input layer, a convolutional layer, a densely connected layer, and an output layer. The input layer receives preprocessed speech features, the convolutional layer automatically extracts high-level emotion-related features, the densely connected layer integrates the features, and the output layer outputs the probability distribution of emotion categories (i.e., service quality evaluation).

[0060] Finally, the system uses the evaluation-speech dataset to train the target model. During training, the target model learns how to identify emotions from speech features. After training, the target model is fused with different weights to create a new model. After the target model is trained on test data from different scenarios and services, it automatically adjusts its weight parameters to accommodate different emotion representation requirements.

[0061] In an optional embodiment, the feature processing operation includes: deleting the noise signal in the speech signal to obtain an intermediate speech signal; detecting the signal strength of each signal segment in the intermediate speech signal; deleting the signal segments in the intermediate speech signal whose signal strength is lower than a preset threshold to obtain a target speech signal; extracting the sound features and semantic features in the target speech signal; and combining the extracted sound features and semantic features into a speech feature sequence.

[0062] Alternatively, during financial advisory services, background noise (such as ambient noise and radio interference) may interfere with the accurate extraction of emotional features. Using signal processing techniques such as spectrum analysis and adaptive filtering, the system can identify and filter out these non-speech components, resulting in a purer intermediate speech signal.

[0063] The system can then further analyze the intensity distribution of the intermediate speech signal to ensure that subsequent analysis focuses on valid speech segments. The system can set a signal strength threshold; any segment below this threshold will be deemed invalid or non-critical and excluded from the feature sequence to avoid interference with emotion recognition.

[0064] The system then conducts an in-depth analysis of the target speech signal, extracting emotional features (such as intonation, speech rate, and pitch fluctuations) and semantic features (such as vocabulary and syntactic structure). These features comprehensively reflect the emotion and intent in the speech signal and are key information sources for identifying the satisfaction and emotional state of the second subject.

[0065] The system can then integrate the extracted acoustic and semantic features into a speech feature sequence in chronological order, forming a data set that comprehensively describes the changes in speech features throughout the entire financial consulting service process. This speech feature sequence can then be used as input for the target model.

[0066] In an optional embodiment, a noise signal in a speech signal is deleted to obtain an intermediate speech signal, including: determining speech spectrum information of the speech signal; determining noise spectrum information in the speech spectrum information by using Fourier transform; deleting the noise spectrum information from the speech spectrum information, and using a signal corresponding to the remaining spectrum information as the intermediate speech signal.

[0067] Optionally, first, the system needs to convert the speech signal in the time domain into the frequency domain through Fourier transform, so that the component frequencies of the signal and the intensity of each frequency component can be visualized, and the characteristics of the speech signal and the noise signal can be identified.

[0068] After obtaining the speech signal's spectral information, the system can further analyze the spectrum to distinguish between speech and noise components. Noise typically appears as dispersed, low-energy components in the spectrum, while speech signals are concentrated within a specific frequency range and have a more concentrated energy distribution. By setting the noise's spectral characteristics, such as the frequency range and energy threshold, the system can accurately identify noise spectral information.

[0069] Once the noise spectrum is identified, the system removes it from the overall spectrum, retaining only the spectrum corresponding to the speech signal. Next, the system performs an inverse Fourier transform on the noise-removed spectrum, restoring it to the time domain signal—the intermediate speech signal. This step ensures that subsequent processing is free of noise interference, improving the accuracy of emotion recognition.

[0070] In an optional embodiment, based on the same time axis, the speech feature sequence of the first object and the speech feature sequence of the second object are combined into a target feature sequence, including: based on the same time axis, performing an alignment operation on the speech feature sequence of the first object and the speech feature sequence of the second object, wherein the alignment operation is used to ensure that the features at each time point come from the speech signal at the same moment; after completing the alignment operation, at each time point, the speech features of the first object and the second object are spliced to obtain the target feature sequence.

[0071] Optionally, because the conversation between the first and second parties is a dynamic and unevenly distributed process, the speech features of the two parties may vary slightly over time, especially during extended financial consulting sessions. Timeline alignment ensures that the speech features of the first and second parties at each time point originate from the same speech signal, which is crucial for subsequent speech feature transformation analysis. This process involves calibrating the speech signal's timestamps, fine-tuning the speech segments, and accurately matching feature points.

[0072] Optionally, after completing the timeline alignment, the system can concatenate the speech features of the first and second subjects at each time point to form a comprehensive feature sequence containing the speech feature information of both parties—the target feature sequence. This sequence can reflect the parallel changes and interactions of the speech features of both parties within a specific time window, providing a basis for in-depth analysis of the dynamic changes in the speech features of both parties. The concatenation operation can include simple vector concatenation or the fusion of different types of features into a unified feature representation through a transformation matrix or neural network.

[0073] In an optional embodiment, the service quality score of the first object for the second object is determined based on the temporal association relationship, including: based on the temporal association relationship, taking the voice features of the second object at time T+1 and the voice features of the first object at time T as a feature combination, wherein time T is any time when the first object speaks, and time T+1 is the time when the second object speaks for the first time after time T; and determining the service quality score of the first object for the second object based on the feature combination.

[0074] Optionally, the system uses the concept of dynamic time windows to capture the immediate impact of the voice interaction between the first and second parties. Specifically, a correlation is established between the voice features of the first party at time T and the voice features of the second party at the next moment, time T+1. Here, time T can refer to any time the first party speaks, and time T+1 corresponds to the time when the second party first responds after that time. This combination of features can effectively reflect the immediate impact of the customer service voice expression on the customer's status.

[0075] Optionally, based on the feature combination, the system can analyze the effect of the voice interaction between the first and second subjects through a target model to determine a service quality score. The target model can be designed to consider multiple factors, such as intonation, emotional intensity, language choice, and speaking speed, to comprehensively assess the first subject's ability to influence the second subject's emotions in a specific situation. The scoring results can not only reflect the first subject's professionalism and service attitude, but also reflect how the first subject, through language and voice, influences the changes in the second subject's voice characteristics, thereby affecting the overall service satisfaction.

[0076] In an optional embodiment, determining a service quality score of a first object for a second object based on a feature combination includes: inputting the feature combination into a target model, wherein the target model includes an input layer, a convolutional layer, a densely connected layer, and an output layer, wherein adjacent network layers of the model are spliced in a fully connected manner; the input layer of the target model is used to receive the feature combination and transmit the feature combination to multiple convolutional layers, each convolutional layer is used to determine local correlation features between feature combinations; the densely connected layer is used to integrate the local correlation features output by multiple convolutional layers into global correlation features; utilizing the prior knowledge learned by the target model in the model training phase to determine the service quality score of the first object for the second object based on the global correlation features; and outputting the service quality score through the output layer of the target model.

[0077] Optionally, in the architecture of the target model, the input layer is the starting point of the dense convolutional neural network, responsible for receiving the preprocessed speech features, i.e., feature combinations. The input layer passes the feature combinations to the subsequent convolutional layers.

[0078] Optionally, the convolutional layer is the core of the target model, used to automatically extract high-level emotion-related features (i.e., local correlation features) from speech features. In the target model, the convolutional layer sets up multiple convolution operations, each of which slides over the input features through the convolution kernel to extract local correlation features. These convolution operations can capture the time series and frequency features in the speech signal, thereby identifying emotion-related patterns. For example, features such as the tone, rhythm, and intensity of speech play an important role in predicting customer satisfaction, and the convolutional layer can effectively extract these features.

[0079] Optionally, the densely connected layer can integrate and map the features extracted by the convolutional layer. Through dense connections, each layer can directly use the feature maps of all previous layers. This design enables dense convolutional neural networks to obtain rich features at different levels and enhance feature extraction capabilities. In audio feature processing, this multi-level feature extraction can better capture complex semantic features (i.e., global correlation features), such as subtle changes in emotions. In addition, due to the dense connections between the layers, the gradient has more paths to pass during the backpropagation process, which enables the gradient to update the network parameters more effectively, avoiding the problem of rapid attenuation of the gradient as the number of network layers increases in traditional deep networks.

[0080] In an optional embodiment, the speech processing method further includes: when there are N feature combinations, determining a service quality score based on each feature combination to obtain N service quality scores, where N is an integer greater than 1; performing weighted average calculation on the N service quality scores to obtain a target service quality score of the first object to the second object, where the later the speech time corresponding to the speech feature in the feature combination, the higher the weight value of the feature combination.

[0081] Optionally, each feature combination is a set of service quality-related features extracted from speech signal data. By analyzing each feature combination, the target model can determine a service quality score. For example, if a feature combination contains a high number of negative words or a high speaking rate, it may indicate that the user is currently impatient, and the service quality score may be low; conversely, if the speech features exhibit a calm tone and clear expression, the service quality score may be high.

[0082] Optionally, since there are N feature combinations, the target model will calculate the service quality score for each feature combination, ultimately obtaining N service quality scores. These scores can reflect the evaluation results of different feature combinations on service quality and provide a basis for subsequent comprehensive scoring.

[0083] Next, the target model calculates the service quality score of the first subject to the second subject by performing a weighted average calculation on the N service quality scores. The weighted average calculation is based on the following principle: the later the speech time corresponding to the speech feature in the feature combination, the higher the weight of the feature combination.

[0084] This principle is based on the following considerations: During a voice interaction, the second party's speech at a later point in the conversation is likely to be closer to the current state of the interaction and better reflect their true intentions and immediate perception of service quality. For example, in a financial consulting conversation, the second party may initially simply express a question, but later in the conversation they may comment on the effectiveness of the service resolution. These later comments often better reflect the ultimate performance of service quality.

[0085] Therefore, assigning higher weights to feature combinations from later speech times allows the target model to focus more on recent interactions, allowing it to better adapt to dynamically changing service scenarios. This weight distribution method can improve the target model's sensitivity and accuracy in assessing service quality.

[0086] In addition, the weight distribution can be adjusted according to the actual application scenario. For example, in some scenarios, it may be necessary to pay more attention to the speech features within a specific time period. The target model can implement a flexible evaluation strategy by adjusting the weights.

[0087] In an optional embodiment, the training process of the target model includes the following steps: obtaining a training data set, wherein the training data set includes the customer service's voice signal and the customer's voice signal; annotating the voice signals in the training data set, wherein the annotated content of the voice signal includes the customer's rating of the service quality; based on the training data set, extracting the customer service's voice feature sequence and the customer's voice feature sequence; combining the customer service's voice feature sequence and the customer's voice feature sequence into a training feature sequence; and performing multiple iterative training on the neural network based on the training feature sequence and the annotated content of the voice signal until the target model is obtained through training.

[0088] Optionally, when training the target model, a training dataset must first be obtained. The training dataset includes the speech signal of the first subject and the speech signal of the second subject. To ensure the generalization capability of the model, the training dataset should cover a variety of service scenarios, customer types, and customer service performance to reflect the diversity of real-world applications.

[0089] To train the target model, the speech signals in the training dataset need to be annotated. These annotations include the second subject's rating of the service quality. These annotations guide the target model in identifying service quality based on speech features.

[0090] Optionally, the annotation methods include but are not limited to manual annotation, customer service feedback and multi-dimensional annotation. Among them, manual annotation refers to the service quality scoring of the voice signal by a professional annotation team according to predefined standards. The annotation criteria may include multiple dimensions such as voice clarity, speaking speed, tone, customer satisfaction, etc. Customer feedback means that if the customer (ie, the second object) provides a direct satisfaction rating during the interaction process (for example, through a questionnaire or automatic feedback system), these ratings can be directly used as annotation content. In addition, the system can perform multi-dimensional annotation of the voice signal, such as emotional tendency (positive, negative, neutral), problem solving status (solved, unsolved), etc., to enrich the learning content of the model.

[0091] Next, the target model can extract a speech feature sequence of the first object and a speech feature sequence of the second object based on the training dataset. The speech feature sequence is a set of features related to quality of service extracted from the speech signal, which can reflect the physical and semantic characteristics of the speech signal.

[0092] Next, the target model can combine the speech feature sequence of the first subject and the speech feature sequence of the second subject into a training feature sequence. This combination can be done by directly concatenating the speech feature sequences of the first and second subjects to form a longer feature sequence, or by fusing the speech features of the first and second subjects through methods such as weighted averaging and feature crossover to generate a more representative feature sequence. Furthermore, the speech feature sequences of the first and second subjects need to be time-aligned to ensure consistency of the feature sequences in the temporal dimension.

[0093] Finally, based on the training feature sequence and the annotated content of the speech signal, the system iteratively trains the neural network multiple times until the target model is obtained. During the iterative training process, the target model can better fit the training data by continuously adjusting the parameters of the neural network. Among them, the system can update the parameters of the neural network to minimize the loss function by using optimization algorithms such as SGD (Stochastic Gradient Descent) and Adam (Adaptive Moment Estimation). The system can also use a validation set to verify the target model to prevent overfitting and select the best model parameters.

[0094] It should be noted that during the process of multiple iterative training of neural networks, the training data set needs to be anonymized and regularly subject to third-party compliance audits to ensure that the data is used in accordance with ethical standards.

[0095] According to another aspect of the embodiment of the present application, a speech processing device is further provided, wherein: Figure 3is a schematic diagram of an optional speech processing device according to an embodiment of the present application, such as Figure 3 As shown, the speech processing device includes: a collection unit 301 , a first processing unit 302 , a second processing unit 303 , a first determination unit 304 , and a second determination unit 305 .

[0096] Optionally, the collection unit 301 is used to collect the voice signal of the first object and the voice signal of the second object, wherein the first object provides financial consulting services to the second object; the first processing unit 302 is used to perform feature processing operations on the voice signal of the first object and the voice signal of the second object, respectively, to obtain the voice feature sequence of the first object and the voice feature sequence of the second object; the second processing unit 303 is used to combine the voice feature sequence of the first object and the voice feature sequence of the second object into a target feature sequence based on the same time axis; the first determination unit 304 is used to determine the temporal correlation relationship between the voice features of the first object and the voice features of the second object according to the target feature sequence; the second determination unit 305 is used to determine the service quality score of the first object to the second object based on the temporal correlation relationship.

[0097] Optionally, the first processing unit 302 includes: a first deletion subunit, a detection subunit, a second deletion subunit, an extraction subunit, and a first processing subunit. The first deletion subunit is configured to delete noise signals from the speech signal to obtain an intermediate speech signal; the detection subunit is configured to detect the signal strength of each signal segment in the intermediate speech signal; the second deletion subunit is configured to delete signal segments in the intermediate speech signal whose signal strength is lower than a preset threshold to obtain a target speech signal; the extraction subunit is configured to extract acoustic features and semantic features from the target speech signal; and the first processing subunit is configured to combine the extracted acoustic features and semantic features into a speech feature sequence.

[0098] Optionally, the first deletion subunit includes: a first determination module, a second determination module, and a processing module. The first determination module is configured to determine speech spectrum information of the speech signal; the second determination module is configured to determine noise spectrum information in the speech spectrum information using a Fourier transform; and the processing module is configured to delete the noise spectrum information from the speech spectrum information and use the signal corresponding to the remaining spectrum information as the intermediate speech signal.

[0099] Optionally, the second processing unit 303 includes: a second processing subunit and a third processing subunit. The second processing subunit is configured to align the speech feature sequence of the first object and the speech feature sequence of the second object based on the same time axis, wherein the alignment operation is used to ensure that the features at each time point are derived from the speech signal at the same moment; and the third processing subunit is configured to, after completing the alignment operation, concatenate the speech features of the first object and the second object at each time point to obtain a target feature sequence.

[0100] Optionally, the second determination unit 305 includes: a fourth processing subunit and a first determination subunit. The fourth processing subunit is configured to combine, based on a temporal association relationship, the speech features of the second subject at time T+1 and the speech features of the first subject at time T as a feature combination, where time T is any time when the first subject speaks and time T+1 is the time when the second subject speaks for the first time after time T; and the first determination subunit is configured to determine, based on the feature combination, a service quality score of the first subject for the second subject.

[0101] Optionally, the first determination subunit includes: an input module, a third determination module, and an output module. The input module is used to input the feature combination into the target model, wherein the target model includes an input layer, a convolutional layer, a densely connected layer, and an output layer, wherein adjacent network layers of the model are spliced in a fully connected manner; the input layer of the target model is used to receive the feature combination and transmit the feature combination to multiple convolutional layers, each convolutional layer is used to determine the local correlation features between the feature combinations; the densely connected layer is used to integrate the local correlation features output by multiple convolutional layers into global correlation features; the third determination module is used to utilize the prior knowledge learned by the target model during the model training phase to determine the service quality score of the first object for the second object based on the global correlation features; the output module is used to output the service quality score through the output layer of the target model.

[0102] Optionally, the second determination unit 305 further includes: a second determination subunit and a fifth processing subunit. The second determination subunit is configured to, when there are N feature combinations, determine a service quality score based on each feature combination to obtain N service quality scores, where N is an integer greater than 1; and the fifth processing subunit is configured to perform a weighted average calculation on the N service quality scores to obtain a target service quality score of the first object for the second object, where the later the speech time corresponding to the speech feature in the feature combination, the higher the weight value of the feature combination.

[0103] Optionally, the input module includes: an acquisition submodule, a first processing submodule, an extraction submodule, a second processing submodule, and a third processing submodule. The acquisition submodule is used to acquire a training data set, wherein the training data set includes the customer service's voice signal and the customer's voice signal; the first processing submodule is used to annotate the voice signals in the training data set, wherein the annotated content of the voice signals includes the customer's rating of the service quality; the extraction submodule is used to extract the customer service's voice feature sequence and the customer's voice feature sequence based on the training data set; the second processing submodule is used to combine the customer service's voice feature sequence and the customer's voice feature sequence into a training feature sequence; and the third processing submodule is used to perform multiple iterative training on the neural network based on the training feature sequence and the annotated content of the voice signal until the target model is obtained through training.

[0104] According to another aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program runs, the device where the computer-readable storage medium is located executes the above-mentioned speech processing method.

[0105] According to another aspect of an embodiment of the present application, an electronic device is also provided, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors execute the above-mentioned speech processing method.

[0106] The above-mentioned embodiments or examples disclosed in this application are not exhaustive, but are only illustrations of some embodiments or examples, and are not intended to be specific limitations on the scope of protection disclosed in this application. In the absence of contradiction, each step in a certain embodiment or example in this application can be implemented as an independent example, and the steps can be arbitrarily combined. For example, the solution after removing some steps in a certain embodiment or example can also be implemented as an independent example, and the order of the steps in a certain embodiment or example can be arbitrarily exchanged. In addition, the optional methods or optional examples in a certain embodiment or example can be arbitrarily combined; in addition, the various embodiments or examples can be arbitrarily combined. For example, some or all of the steps in different embodiments or examples can be arbitrarily combined, and a certain embodiment or example can be arbitrarily combined with the optional methods or optional examples of other embodiments or examples.

[0107] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0108] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0109] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0110] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected to achieve the purpose of the present embodiment according to actual needs.

[0111] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0112] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program code.

[0113] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A speech processing method, characterized in that: include: With authorization from a first subject and a second subject, collecting a voice signal of the first subject and a voice signal of the second subject, wherein the first subject provides financial consulting services to the second subject; Performing feature processing operations on the speech signal of the first object and the speech signal of the second object respectively to obtain a speech feature sequence of the first object and a speech feature sequence of the second object; Based on the same time axis, combining the speech feature sequence of the first object and the speech feature sequence of the second object into a target feature sequence; determining a temporal correlation relationship between the speech features of the first object and the speech features of the second object according to the target feature sequence; A service quality score of the first object to the second object is determined according to the temporal association relationship.

2. The method according to claim 1, characterized in that The feature processing operation includes the following steps: Delete the noise signal in the speech signal to obtain the intermediate speech signal; detecting the signal strength of each signal segment in the intermediate speech signal; Deleting a signal segment whose signal strength is lower than a preset threshold in the intermediate speech signal to obtain a target speech signal; Extracting sound features and semantic features from the target speech signal; The extracted sound features and semantic features are combined into a speech feature sequence.

3. The method according to claim 2, characterized in that Delete the noise signal from the speech signal to obtain the intermediate speech signal, including: determining speech spectrum information of a speech signal; Determining noise spectrum information in the speech spectrum information by using Fourier transform; The noise spectrum information is deleted from the speech spectrum information, and a signal corresponding to the remaining spectrum information is used as the intermediate speech signal.

4. The method according to claim 1, wherein Combining the speech feature sequence of the first object and the speech feature sequence of the second object into a target feature sequence based on the same time axis includes: Based on the same time axis, aligning the speech feature sequence of the first object and the speech feature sequence of the second object, wherein the alignment operation is used to ensure that the features at each time point come from the speech signal at the same moment; After completing the alignment operation, at each time point, the speech features of the first object and the second object are concatenated to obtain the target feature sequence.

5. The method according to claim 1, wherein Determining a service quality score of the first object to the second object according to the temporal association relationship includes: Based on the temporal association, the speech features of the second subject at time T+1 and the speech features of the first subject at time T are used as a feature combination, wherein time T is any time when the first subject speaks, and time T+1 is the time when the second subject speaks for the first time after time T; A service quality score of the first object to the second object is determined according to the feature combination.

6. The method according to claim 5, characterized in that Determining a service quality score of the first object to the second object according to the feature combination includes: Inputting the feature combination into a target model, wherein the target model includes an input layer, a convolutional layer, a densely connected layer, and an output layer, wherein adjacent network layers of the target model are spliced in a fully connected manner; the input layer of the target model is used to receive the feature combination and transmit the feature combination to multiple convolutional layers, each of which is used to determine local correlation features between the feature combinations; the densely connected layer is used to integrate the local correlation features output by multiple convolutional layers into global correlation features; Determining a service quality score of the first object to the second object based on the global association feature using the prior knowledge learned by the target model during the model training phase; The service quality score is outputted through the output layer of the target model.

7. The method according to claim 5, characterized in that The method further comprises: In the case where there are N feature combinations, a service quality score is determined according to each feature combination to obtain N service quality scores, where N is an integer greater than 1; A weighted average calculation is performed on the N service quality scores to obtain a target service quality score of the first object for the second object, wherein the later the speaking time corresponding to the voice feature in the feature combination is, the higher the weight value of the feature combination is.

8. The method according to claim 6, characterized in that The training process of the target model includes the following steps: Acquire a training data set, wherein the training data set includes a customer service voice signal and a customer voice signal; Annotating the speech signals in the training data set, wherein the annotated content of the speech signals includes the customer's rating of the service quality; Extracting the customer service representative's speech feature sequence and the customer's speech feature sequence based on the training data set; Combining the customer service's voice feature sequence and the client's voice feature sequence into a training feature sequence; The neural network is iteratively trained multiple times according to the training feature sequence and the annotated content of the speech signal until the target model is obtained through training.

9. A speech processing device, characterized in that: include: a collection unit, configured to collect a voice signal of the first subject and a voice signal of the second subject upon authorization by the first subject and the second subject, wherein the first subject provides financial consulting services to the second subject; a first processing unit, configured to perform feature processing operations on the speech signal of the first object and the speech signal of the second object, respectively, to obtain a speech feature sequence of the first object and a speech feature sequence of the second object; a second processing unit, configured to combine the speech feature sequence of the first object and the speech feature sequence of the second object into a target feature sequence based on the same time axis; a first determining unit, configured to determine a temporal correlation relationship between the speech features of the first object and the speech features of the second object according to the target feature sequence; A second determining unit is configured to determine a service quality score of the first object to the second object according to the temporal association relationship.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located executes the speech processing method according to any one of claims 1 to 8.

11. An electronic device, characterized in that: The invention comprises one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the speech processing method according to any one of claims 1 to 8.

Images