A method and system for real-time recognition of user conversation emotions based on ASR

Abstract

The present invention proposes a method and system for real-time recognition of user conversation emotions based on ASR. The method and system for real-time recognition of user conversation emotions based on ASR include: real-time collection of voice data during user conversations, and segmentation processing of the voice data using a comprehensive voice segmentation strategy combined with a first segmentation coefficient and a second segmentation coefficient to obtain a plurality of voice data segments to be recognized; using an ASR automatic voice recognition system to perform text conversion on the voice data segments to be recognized, and obtain text data corresponding to the voice data segments to be recognized; preprocessing the text data in combination with the voice data segments to be recognized, and obtaining preprocessed text data; inputting the preprocessed text data into a trained deep learning model for emotion analysis, and obtaining emotion recognition results corresponding to the preprocessed text data. The system includes modules corresponding to the steps of the method.

Description

Technical Field

[0001] The invention proposes a method and system for real-time recognition of user conversation emotions based on ASR, belonging to the technical field of emotion recognition. Background Art

[0002] In the current communication environment, emotion recognition technology for user calls is gradually becoming one of the key technologies for improving customer service quality and enhancing human-computer interaction experience. Traditional emotion recognition methods often rely on manual annotation or post-analysis, which is not only inefficient but also unable to capture and respond to user emotional changes in real time. With the rapid development of speech recognition (ASR, Automatic Speech Recognition) technology, it has become possible to automatically recognize user call content and analyze user emotions through ASR technology.

[0003] However, directly applying ASR technology to emotion recognition faces many challenges. First, the voice data during user calls often contains various background noises, changes in speech speed, and dialect accents, which seriously affect the recognition accuracy of the ASR system. Second, even if the ASR system can accurately recognize the voice content and convert it into text, how to effectively extract emotional features from these text data and perform accurate analysis is also a complex problem. Most existing text emotion analysis technologies are based on static text data, and their applicability is limited to dynamic and continuous voice data such as real-time calls. Summary of the invention

[0004] The present invention provides a method and system for real-time recognition of user call emotions based on ASR, which is used to solve the technical problems of the above-mentioned prior art. The technical solutions adopted are as follows:

[0005] A method for real-time recognition of user conversation emotions based on ASR, the method comprising:

[0006] Collecting voice data of a user in a call in real time, and performing segmentation processing on the voice data by using a comprehensive voice segmentation strategy in combination with a first segmentation coefficient and a second segmentation coefficient, to obtain a plurality of voice data segments to be recognized;

[0007] Using an ASR automatic speech recognition system to convert the speech data segment to be recognized into text, and obtaining text data corresponding to the speech data segment to be recognized;

[0008] Preprocessing the text data in combination with the speech data segment to be recognized to obtain preprocessed text data;

[0009] The preprocessed text data is input into a trained deep learning model for sentiment analysis to obtain sentiment recognition results corresponding to the preprocessed text data.

[0010] Furthermore, voice data of the user during the call is collected in real time, and the voice data is segmented using a comprehensive voice segmentation strategy combined with a first segmentation coefficient and a second segmentation coefficient to obtain a plurality of voice data segments to be recognized, including:

[0011] Collect voice data of users during calls in real time;

[0012] Performing initial segmentation on the voice data using a preset initial speech rate threshold to obtain first voice data and second voice data;

[0013] Wherein, the first voice data is voice data whose speed exceeds a preset initial speech speed threshold; the second voice data is voice data whose speed does not exceed a preset initial speech speed threshold;

[0014] The first voice data and the second voice data are segmented respectively using the comprehensive voice segmentation strategy in combination with the first segmentation coefficient and the second segmentation coefficient to obtain the voice data segments to be recognized corresponding to the first voice data and the second voice data.

[0015] Furthermore, the comprehensive speech segmentation strategy includes:

[0016] Retrieving the speech rate parameter, volume intensity and pitch change rate corresponding to the first voice data;

[0017] Obtaining a first segmentation coefficient using the speech rate parameter, volume intensity, and pitch change rate corresponding to the first voice data, and adjusting a preset first segmentation speech rate threshold using the first segmentation coefficient to obtain an adjusted first segmentation speech rate threshold;

[0018] Compare the speech rate in the first speech data with the adjusted first segmented speech rate threshold to obtain first target speech data and second target speech data;

[0019] The first target voice data is the first voice data whose speech speed exceeds the adjusted first segmented speech speed threshold; the second target voice data is the first voice data whose speech speed does not exceed the adjusted first segmented speech speed threshold;

[0020] Furthermore, the first target voice data and the second target voice data are the voice data segments to be recognized.

[0021] Furthermore, the process of obtaining the first segment coefficient includes:

[0022] Retrieving the speech rate parameter, volume intensity and pitch change rate corresponding to the first voice data;

[0023] Acquire a first segmentation coefficient corresponding to the first speech data using a speech rate parameter, a volume intensity, and a pitch change rate corresponding to the first speech data;

[0024] The first segment coefficient is obtained by the following formula:

[0025]

[0026] Among them, S 01 represents the first segment coefficient; n represents the number of unit time contained in the first speech data, and the value of the unit time is 5s-3min; K i represents the number of syllables contained in the first speech data of the i-th unit time; K c Indicates the preset reference value of the number of syllables; P si represents the pitch change rate at the start time of the i-th unit time; P zi represents the pitch change rate at the end time of the i-th unit time; T di represents the time length corresponding to the i-th unit time; P maxi represents the maximum value of the pitch change rate per unit time; P mini represents the minimum value of the pitch change rate per unit time; T mi represents the time interval between the maximum value of the pitch change rate and the minimum value of the pitch change rate corresponding to the i-th unit time; B maxi Indicates the maximum volume intensity that occurs in the i-th unit time; B mini Indicates the minimum volume intensity that occurs in the i-th unit time; B zpi represents the average volume intensity of all syllables that appear in the i-th unit time; K di represents the number of syllables that appear in the time period between the maximum volume intensity moment and the minimum volume intensity moment of the i-th unit time;

[0027] Using the first segmentation coefficient to adjust the first segmentation speech rate threshold, and obtain the adjusted first segmentation speech rate threshold;

[0028] The adjusted first segment speech rate threshold is obtained by the following formula:

[0029]

[0030] Among them, K t01 represents the adjusted speech rate threshold of the first segment; K 01 Indicates the speech rate threshold of the first segment; S 01 represents the first segment coefficient; X p01Represents the average slope of the pitch change rate corresponding to n unit times of the first speech data.

[0031] Furthermore, the comprehensive speech segmentation strategy further includes:

[0032] Retrieving the speech speed parameter, volume intensity and pitch change rate corresponding to the second voice data;

[0033] Obtaining a second segmentation coefficient using the speech rate parameter, volume intensity, and pitch change rate corresponding to the second voice data, and adjusting a preset second segmentation speech rate threshold using the second segmentation coefficient to obtain an adjusted second segmentation speech rate threshold;

[0034] Compare the speech rate in the second speech data with the adjusted second segmented speech rate threshold to obtain third target speech data and fourth target speech data;

[0035] The third target voice data is the second voice data whose speech speed exceeds the adjusted second segment speech speed threshold; the fourth target voice data is the second voice data whose speech speed does not exceed the adjusted second segment speech speed threshold;

[0036] Furthermore, the third target voice data and the fourth target voice data are the voice data segments to be recognized.

[0037] Furthermore, the process of obtaining the second segment coefficient includes:

[0038] Retrieving the speech speed parameter, volume intensity and pitch change rate corresponding to the second voice data;

[0039] Acquire a second segmentation coefficient corresponding to the second voice data using a speech rate parameter, a volume intensity, and a pitch change rate corresponding to the second voice data;

[0040] The second segment coefficient is obtained by the following formula:

[0041]

[0042] Among them, S 02 represents the second segment coefficient; m represents the number of unit time contained in the second speech data, and the value of the unit time is 20s-3mjn; K j K represents the number of syllables contained in the second speech data of the jth unit time; c Indicates the preset reference value of the number of syllables; T dj represents the time length corresponding to the jth unit time; P maxj represents the maximum value of the pitch change rate per j-th unit time; P mjnj represents the minimum value of the pitch change rate per j-th unit time; T mjrepresents the time interval between the maximum pitch change rate moment and the minimum pitch change rate moment corresponding to the jth unit time; B maxj Indicates the maximum volume intensity that occurs in the jth unit time; B mjnj Indicates the minimum volume intensity that occurs in the jth unit time; B zpj represents the average volume intensity of all syllables that appear in the jth unit time; K dj represents the number of syllables that appear in the time period between the maximum volume intensity moment and the minimum volume intensity moment of the j-th unit time;

[0043] Using the second segmentation coefficient to adjust the second segmentation speech rate threshold, and obtain the adjusted second segmentation speech rate threshold;

[0044] The adjusted second segment speech rate threshold is obtained by the following formula:

[0045]

[0046] Among them, K t02 represents the adjusted speech rate threshold of the second segment; K 02 Indicates the speech rate threshold of the second segment; S 02 Represents the second segment coefficient; X p02 Represents the average slope of the pitch change rate corresponding to m unit times of the second speech data.

[0047] Further, the ASR automatic speech recognition system is used to convert the speech data segment to be recognized into text, and the text data corresponding to the speech data segment to be recognized is obtained, including:

[0048] Extracting multiple speech data segments to be recognized;

[0049] Adjust the unit time, obtain the adjusted unit time length, and use the adjusted unit time as the unit time period; wherein the value range of the unit time period is 10s-1min;

[0050] Performing speech preprocessing on each speech data segment to be recognized, and obtaining a plurality of speech data segments to be recognized after preprocessing; wherein the preprocessing includes but is not limited to noise suppression processing and echo cancellation processing;

[0051] Retrieving the speech rate parameter corresponding to each speech data segment to be recognized after preprocessing;

[0052] Extracting the time frame length corresponding to the preset frame processing;

[0053] Obtaining the overlap ratio using the speech rate parameter corresponding to each speech data segment to be recognized;

[0054] The overlap ratio is obtained by the following formula:

[0055]

[0056] Among them, P bI Indicates the overlap ratio; P bl0 represents the preset overlap ratio reference value; k represents the number of unit time periods contained in each speech data segment to be recognized; K xi represents the number of syllables corresponding to the i-th unit time period; K wp K represents the average number of syllables per unit time of the parent speech data segment corresponding to each speech data segment to be recognized; c Indicates the preset reference value of the number of syllables;

[0057] Performing frame processing on each speech data segment to be recognized after preprocessing according to the preset time frame length corresponding to the frame processing and the overlap ratio, and obtaining an audio data frame corresponding to each speech data segment to be recognized;

[0058] The audio data frame corresponding to each speech data segment to be recognized is input into the ASR automatic speech recognition system to generate text data corresponding to each speech data segment to be recognized.

[0059] Further, the text data is preprocessed in combination with the speech data segment to be recognized to obtain the preprocessed text data, including:

[0060] Retrieving text data corresponding to each voice data segment to be recognized;

[0061] Remove the blank characters at the beginning and several digits of the text data, redundant spaces, and special symbols that are not Chinese characters, numbers, or English, to obtain first sample data;

[0062] Using a word segmentation tool to perform word segmentation processing on the first sample data to obtain second sample data after the word segmentation processing;

[0063] Performing error correction and revision processing on the second sample data to obtain error-corrected and revised text data;

[0064] The error-corrected and amended text data is the pre-processed text data.

[0065] Furthermore, the deep learning model is a convolutional neural network, and the structure of the deep learning model is as follows:

[0066] Input layer, used to receive preprocessed text data;

[0067] An embedding layer for mapping the index of each word in the text data into a dense vector space of fixed dimension;

[0068] A convolutional layer, used to extract features from the text data and obtain a feature map corresponding to the text data;

[0069] A pooling layer, used for performing dimensionality reduction processing on the feature map;

[0070] The concatenation layer is used to concatenate the feature maps after dimensionality reduction to form feature vectors;

[0071] Fully connected layer, used to integrate the features of feature vectors;

[0072] The output layer is used to output emotion recognition results.

[0073] A user conversation emotion real-time recognition system based on ASR, the user conversation emotion real-time recognition system based on ASR comprising:

[0074] A voice segmentation module is used to collect voice data of a user in a call in real time, and to segment the voice data using a comprehensive voice segmentation strategy combined with a first segmentation coefficient and a second segmentation coefficient to obtain a plurality of voice data segments to be recognized;

[0075] A text data acquisition module, used to convert the speech data segment to be recognized into text using the ASR automatic speech recognition system, and obtain text data corresponding to the speech data segment to be recognized;

[0076] A text data preprocessing module, used to preprocess the text data in combination with the speech data segment to be recognized, and obtain the preprocessed text data;

[0077] The emotion recognition module is used to input the preprocessed text data into the trained deep learning model for emotion analysis, and obtain the emotion recognition results corresponding to the preprocessed text data.

[0078] Beneficial effects of the present invention:

[0079] The present invention proposes a method and system for real-time recognition of user call emotions based on ASR. The method can collect voice data during user calls in real time, and perform segmentation processing and text conversion, so as to realize real-time recognition of emotions. This helps to capture the user's emotional changes in a timely manner and provide a basis for subsequent decision-making and services. Through the application of comprehensive voice segmentation strategy and ASR automatic voice recognition system, the method can more accurately recognize the user's voice content and convert it into text information. At the same time, the application of deep learning models also improves the accuracy of emotion recognition. The method can be applied to a variety of scenarios, such as customer service, smart home, medical health and other fields. By adjusting the segmentation coefficient and the parameters of the deep learning model, it can adapt to the emotion recognition needs in different scenarios. In response to the needs of different user groups and application scenarios, the method can provide more personalized customized services. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 is a flow chart of the method of the present invention;

[0081] Figure 2 The system block diagram of the system of the present invention. DETAILED DESCRIPTION

[0082] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0083] The embodiment of the present invention proposes a method for real-time recognition of user conversation emotions based on ASR. Figure 1 As shown, the ASR-based user call emotion real-time recognition method includes:

[0084] S1. collecting voice data of a user during a call in real time, and performing segmentation processing on the voice data by using a comprehensive voice segmentation strategy in combination with a first segmentation coefficient and a second segmentation coefficient to obtain a plurality of voice data segments to be recognized;

[0085] S2, using the ASR automatic speech recognition system to convert the speech data segment to be recognized into text, and obtaining text data corresponding to the speech data segment to be recognized;

[0086] S3, preprocessing the text data in combination with the speech data segment to be recognized to obtain preprocessed text data;

[0087] S4: input the preprocessed text data into a trained deep learning model for sentiment analysis to obtain sentiment recognition results corresponding to the preprocessed text data.

[0088] The working principle of the above technical solution is as follows: the method first collects the voice data of the user during the call in real time to ensure the timeliness and accuracy of the data. Then, the voice data is segmented by combining the first segmentation coefficient and the second segmentation coefficient using a comprehensive voice segmentation strategy. These coefficients may be set based on the volume, speech speed, pause and other characteristics of the voice to ensure that the segmented voice data segments can more accurately reflect the user's emotional changes. Through segmentation processing, multiple voice data segments to be recognized are obtained to provide basic data for subsequent steps. The ASR automatic speech recognition system is used to convert the voice data segments to be recognized into text. The ASR system can convert voice signals into text information, which is a basic step to achieve emotion recognition. In this process, the ASR system needs to overcome various challenges, such as noise interference, accent differences, etc., to ensure the accuracy and reliability of text conversion. The text data is preprocessed in combination with the voice data segments to be recognized. The preprocessing steps may include removing irrelevant characters, standardizing text formats, correcting spelling errors, etc. The preprocessed text data is clearer and more accurate, which is conducive to the subsequent emotion analysis steps. The preprocessed text data is input into the trained deep learning model for emotion analysis. The deep learning model can learn the emotional features in the text, classify and identify them. Through the calculation and analysis of the deep learning model, the emotion recognition results corresponding to the preprocessed text data are obtained. These results may include multiple emotion categories such as anger, sadness, happiness, etc.

[0089] The effect of the above technical solution is that the method can collect the voice data of the user during the call in real time, and perform segmentation processing and text conversion, so as to realize the real-time recognition of emotions. This helps to capture the emotional changes of users in a timely manner and provide a basis for subsequent decision-making and services. Through the application of comprehensive voice segmentation strategy and ASR automatic voice recognition system, the method can more accurately recognize the user's voice content and convert it into text information. At the same time, the application of deep learning model also improves the accuracy of emotion recognition. The method can be applied to a variety of scenarios, such as customer service, smart home, medical health and other fields. By adjusting the segmentation coefficient and the parameters of the deep learning model, it can adapt to the emotion recognition needs in different scenarios. In response to the needs of different user groups and application scenarios, the method can provide more personalized customized services. For example, in the field of customer service, the service strategy can be adjusted and the service process can be optimized according to the user's emotional state; in the field of smart home, the operation mode and settings of home appliances can be adjusted according to the user's emotional changes.

[0090] In summary, the real-time recognition method of user call emotions based on ASR has technical effects such as real-time, accuracy, flexibility and personalization, and can provide users with a more intelligent, convenient and efficient service experience.

[0091] In one embodiment of the present invention, voice data of a user during a call is collected in real time, and the voice data is segmented using a comprehensive voice segmentation strategy in combination with a first segmentation coefficient and a second segmentation coefficient to obtain a plurality of voice data segments to be recognized, including:

[0092] S101, real-time collection of voice data during user calls;

[0093] S102, using a preset initial speech speed threshold to initially segment the speech data, and obtain first speech data and second speech data; wherein the first speech data is speech data whose speed exceeds the preset initial speech speed threshold; and the second speech data is speech data whose speed does not exceed the preset initial speech speed threshold;

[0094] S103: Use the comprehensive speech segmentation strategy in combination with the first segmentation coefficient and the second segmentation coefficient to segment the first speech data and the second speech data respectively, and obtain speech data segments to be recognized corresponding to the first speech data and the second speech data.

[0095] The working principle of the above technical solution is: during the user's call, the system captures and collects voice data in real time. This step is the basis for all subsequent processing to ensure the integrity and real-time nature of the voice data. The system presets an initial speech rate threshold, which is used to distinguish between faster and slower speech data. The first voice data (speech rate exceeds the threshold) and the second voice data (speech rate does not exceed the threshold). This initial segmentation helps to perform more refined processing of speech data with different speech rates in the future.

[0096] The system applies a comprehensive speech segmentation strategy to the first speech data and the second speech data. The strategy combines two segmentation coefficients (a first segmentation coefficient and a second segmentation coefficient), which may be adjusted according to specific characteristics of the speech data (such as volume, pitch change, pause, etc.). By applying these coefficients and strategies, the system further segments the first speech data and the second speech data into a plurality of speech data segments to be recognized.

[0097] The effect of the above technical solution is: by preliminarily segmenting the voice data according to the speaking speed, and combining it with the comprehensive voice segmentation strategy for fine segmentation, the system can recognize the voice content more accurately. This segmentation method helps to reduce recognition errors caused by changes in speaking speed. Real-time collection and processing of voice data ensures that the system can quickly respond to and process the voice information in the user's call. Segmentation processing reduces the amount of data for a single recognition, which helps to improve processing speed and efficiency. By setting and adjusting the speaking speed threshold and segmentation coefficient, the system can adapt to the voice characteristics and habits of different users. This adaptability enables the system to maintain high recognition performance and user experience in a variety of scenarios. The technical solution can handle various voice changes during user calls, such as speaking speed, volume, etc. This enables the system to maintain a stable recognition effect in complex scenarios such as noisy environments, different languages ​​or dialects.

[0098] In summary, this technical solution achieves accurate, efficient and adaptable processing of user call voice data by real-time voice data collection, preliminary segmentation using the initial speech rate threshold, and fine segmentation using a comprehensive voice segmentation strategy.

[0099] In one embodiment of the present invention, the comprehensive speech segmentation strategy includes:

[0100] Step 1a, retrieving speech rate parameters, volume intensity and pitch change rate corresponding to the first voice data;

[0101] Step 2a, obtaining a first segmentation coefficient using the speech rate parameter, volume intensity, and pitch change rate corresponding to the first voice data, and adjusting a preset first segmentation speech rate threshold using the first segmentation coefficient to obtain an adjusted first segmentation speech rate threshold;

[0102] Step 3a, comparing the speech rate in the first voice data with the adjusted first segmented speech rate threshold to obtain first target voice data and second target voice data;

[0103] The first target voice data is the first voice data whose speech speed exceeds the adjusted first segmented speech speed threshold; the second target voice data is the first voice data whose speech speed does not exceed the adjusted first segmented speech speed threshold;

[0104] Furthermore, the first target voice data and the second target voice data are the voice data segments to be recognized.

[0105] The working principle of the above technical solution is as follows: first, retrieve the speech rate parameter, volume intensity and pitch change rate from the first speech data. These parameters can reflect the dynamic characteristics and rhythmic changes of the speech data. Using the retrieved speech rate parameters, volume intensity and pitch change rate, the first segmentation coefficient is calculated through a certain algorithm or model. Then, the preset first segmentation speech rate threshold is adjusted using this first segmentation coefficient to obtain an adjusted first segmentation speech rate threshold that is more in line with the characteristics of the current speech data. The speech rate in the first speech data is compared with the adjusted first segmentation speech rate threshold. According to the comparison result, the speech data is divided into two parts: the part whose speech rate exceeds the adjusted threshold is used as the first target speech data, and the part whose speech rate does not exceed the adjusted threshold is used as the second target speech data.

[0106] The effect of the above technical solution is: by introducing multiple speech parameters (speech rate, volume intensity and pitch change rate) and calculating the segmentation coefficient, the strategy can more accurately reflect the characteristics of the speech data, thereby improving the accuracy of the segmentation. Since the strategy can adjust the segmentation speech rate threshold according to the actual situation of the speech data, it has strong adaptability to speech data with different speech rates, volumes and pitch changes. The segmented speech data (first target speech data and second target speech data) can be more conveniently processed in subsequent processes, such as speech recognition, speech synthesis, etc. In particular, in speech recognition, segmentation processing helps to improve the accuracy and efficiency of recognition. Through segmentation processing, computing resources and storage resources can be allocated more reasonably. For example, when processing large segments of continuous speech data, it can be divided into smaller segments for processing, thereby optimizing the utilization of computing resources.

[0107] In summary, the comprehensive speech segmentation strategy achieves accurate segmentation of speech data by introducing multiple speech parameters and calculating segmentation coefficients, and has the technical effects of strong adaptability, providing convenience for subsequent processing, and optimizing resource utilization.

[0108] In one embodiment of the present invention, the process of obtaining the first segment coefficient includes:

[0109] Step 201a, retrieve the speech rate parameters, volume intensity and pitch change rate corresponding to the first voice data;

[0110] Step 202a, obtaining a first segmentation coefficient corresponding to the first speech data using a speech rate parameter, a volume intensity, and a pitch change rate corresponding to the first speech data;

[0111] The first segment coefficient is obtained by the following formula:

[0112]

[0113] Among them, S 01represents the first segment coefficient; n represents the number of unit time contained in the first speech data, and the value of the unit time is 5s-3min; K i represents the number of syllables contained in the first speech data of the i-th unit time; K c Indicates the preset reference value of the number of syllables; P si represents the pitch change rate at the start time of the i-th unit time; P zi represents the pitch change rate at the end time of the i-th unit time; T di represents the time length corresponding to the i-th unit time; P maxi represents the maximum value of the pitch change rate per unit time; P mini represents the minimum value of the pitch change rate per unit time; T mi represents the time interval between the maximum value of the pitch change rate and the minimum value of the pitch change rate corresponding to the i-th unit time; B maxi Indicates the maximum volume intensity that occurs in the i-th unit time; B mini Indicates the minimum volume intensity that occurs in the i-th unit time; B zpi represents the average volume intensity of all syllables that appear in the i-th unit time; K di represents the number of syllables that appear in the time period between the maximum volume intensity moment and the minimum volume intensity moment of the i-th unit time;

[0114] Step 203a: adjusting the first segmented speech rate threshold using the first segmented coefficient to obtain the adjusted first segmented speech rate threshold;

[0115] The adjusted first segment speech rate threshold is obtained by the following formula:

[0116]

[0117] Among them, K t01 represents the adjusted speech rate threshold of the first segment; K 01 Indicates the speech rate threshold of the first segment; S 01 represents the first segment coefficient; X p01 Represents the average slope of the pitch change rate corresponding to n unit times of the first speech data.

[0118] The working principle of the above technical solution is: retrieve key information such as speech rate parameters, volume intensity and pitch change rate from the first voice data. For each unit time (5s-3min), calculate the relevant voice features, including the number of syllables, pitch change rate, volume intensity, etc. Using the above parameters, calculate the first segmentation coefficient through a specific formula. The formula comprehensively considers multiple aspects of the number of syllables, pitch change rate (starting time, ending time, maximum value, minimum value, time interval) and multiple characteristics of volume intensity (maximum value, minimum value, average value, number of syllables in a specific time period). This calculation method aims to fully reflect the rhythm and volume characteristics of the first voice data and provide an accurate basis for subsequent segmentation processing. Using the calculated first segmentation coefficient and the average slope of the pitch change rate of the first voice data, the first segmentation speech rate threshold is adjusted through a formula. The adjusted speech rate threshold is more in line with the actual situation of the first voice data, which helps to make the subsequent segmentation processing more accurate.

[0119] The effect of the above technical solution is: by comprehensively considering multiple parameters such as speech rate, volume intensity and pitch change rate, and calculating the first segmentation coefficient using a specific formula, the technical solution can more accurately reflect the characteristics of the first voice data. Furthermore, using the adjusted speech rate threshold for segmentation processing can improve the accuracy of segmentation and provide a more reliable data basis for subsequent processing (such as speech recognition, speech analysis, etc.). The technical solution can be flexibly adjusted according to the characteristics of different voice data and is suitable for voice data with different speech rates, volumes and pitch changes. This adaptability enables the technical solution to maintain a high processing effect in a variety of scenarios. Through accurate segmentation processing, the calculation amount and time cost of subsequent processing can be reduced. At the same time, the technical solution also uses concepts such as unit time, which helps to further improve processing efficiency while maintaining processing effects. In application scenarios such as speech recognition and speech synthesis, accurate segmentation processing helps to improve the recognition accuracy and synthesis quality of the system. This is of great significance for improving user experience and enhancing user satisfaction.

[0120] On the other hand, by retrieving the speech rate parameter, volume intensity and pitch change rate of the first speech data and calculating the first segment coefficient accordingly, the scheme can dynamically reflect the characteristic changes of the speech data. This dynamic adaptability enables the scheme to more accurately capture the subtle differences in the speech data, thereby improving the accuracy and reliability of subsequent processing. By adjusting the first segment speech rate threshold using the first segment coefficient, the speech rate threshold can be made more consistent with the characteristics of the actual speech data. This precise adjustment helps to more accurately judge the speech rate of the speech in the fields of speech recognition and speech analysis, and improve the performance of related applications. The scheme considers the characteristics of multiple dimensions such as speech rate parameters, volume intensity and pitch change rate, and calculates the first segment coefficient by combining these characteristics. This fusion of multi-dimensional features enables the scheme to more comprehensively reflect the characteristics of the speech data and improves the accuracy and robustness of the processing. The parameters in the formula (such as the value range of the unit time, the reference value of the number of syllables, etc.) can be adjusted according to the actual situation. This parameterized design makes the scheme more flexible. By adjusting these parameters, it is possible to adapt to the speech data processing requirements in different scenarios. Since the solution takes into account the characteristics of multiple dimensions and parameterized design, it can maintain relatively stable performance under different voice data. This stability is particularly important for long-term operation and large-scale application scenarios.

[0121] In summary, the technical effects of the above technical solutions in terms of performance indicators are mainly reflected in the enhancement of dynamic adaptability, more precise adjustment of speech rate threshold, multi-dimensional feature fusion, parametric design to improve flexibility, improved computing efficiency and performance stability. These technical effects make the solution have higher application value in the fields of speech recognition and speech analysis. At the same time, the technical solution comprehensively considers multiple speech parameters, calculates the first segmentation coefficient using a specific formula, and adjusts the speech rate threshold for segmentation processing accordingly, thereby achieving technical effects such as improving segmentation accuracy, enhancing adaptability, optimizing processing efficiency and improving user experience.

[0122] In one embodiment of the present invention, the comprehensive speech segmentation strategy further includes:

[0123] Step 1b, retrieving the speech rate parameters, volume intensity and pitch change rate corresponding to the second voice data;

[0124] Step 2b, using the speech rate parameter, volume intensity and pitch change rate corresponding to the second voice data to obtain a second segmentation coefficient, and using the second segmentation coefficient to adjust the preset second segmentation speech rate threshold to obtain the adjusted second segmentation speech rate threshold;

[0125] Step 3b: Compare the speech rate in the second voice data with the adjusted second segmented speech rate threshold to obtain third target voice data and fourth target voice data;

[0126] The third target voice data is the second voice data whose speech speed exceeds the adjusted second segment speech speed threshold; the fourth target voice data is the second voice data whose speech speed does not exceed the adjusted second segment speech speed threshold;

[0127] Furthermore, the third target voice data and the fourth target voice data are the voice data segments to be recognized.

[0128] The working principle of the above technical solution is: retrieve the speech rate parameters, volume intensity and pitch change rate from the second voice data. These parameters are the basic characteristics of the voice signal and can reflect the characteristics and changes of the voice. Using the above parameters, the second segment coefficient is calculated through a certain algorithm or model. This coefficient is a factor used to adjust the preset second segment speech rate threshold. The preset second segment speech rate threshold is adjusted using the calculated second segment coefficient to obtain a speech rate threshold that is more in line with the characteristics of the current voice data. The adjusted second segment speech rate threshold is compared with the actual speech rate in the second voice data. According to the comparison result, the second voice data is divided into two parts: the part whose speech rate exceeds the adjusted threshold (the third target voice data) and the part whose speech rate does not exceed the adjusted threshold (the fourth target voice data).

[0129] The effect of the above technical solution is: by considering multiple factors such as speech rate parameters, volume intensity and pitch change rate, the solution can more accurately reflect the characteristics of speech data, thereby improving the accuracy of speech segmentation. The solution can dynamically adjust the segmentation speech rate threshold according to different speech data characteristics, so it has strong adaptability. This helps to process speech data with different speech rates, volumes and pitch changes, and improves the flexibility of speech processing. After the speech data is divided into the third target speech data and the fourth target speech data, different processing strategies can be adopted for these two parts of data. For example, for the part with a faster speech rate, a more complex speech recognition algorithm or additional preprocessing may be required; while for the part with a slower speech rate, a relatively simple processing strategy can be adopted. This helps to optimize the speech processing process and improve the overall processing efficiency. Through accurate speech segmentation, more accurate and representative speech data can be provided for subsequent speech recognition tasks. This helps to reduce the misrecognition rate in speech recognition and improve the accuracy of recognition.

[0130] In summary, the comprehensive speech segmentation strategy achieves accurate segmentation of speech data by considering multiple speech feature parameters and dynamically adjusting the segmentation speech rate threshold. This not only improves the flexibility and efficiency of speech processing, but also provides more accurate data support for subsequent speech recognition tasks.

[0131] In one embodiment of the present invention, the process of obtaining the second segment coefficient includes:

[0132] Step 201b, retrieve the speech rate parameters, volume intensity and pitch change rate corresponding to the second voice data;

[0133] Step 202b: Acquire a second segmentation coefficient corresponding to the second voice data using a speech rate parameter, a volume intensity, and a pitch change rate corresponding to the second voice data;

[0134] The second segment coefficient is obtained by the following formula:

[0135]

[0136] Among them, S 02 represents the second segment coefficient; m represents the number of unit time contained in the second speech data, and the value of the unit time is 20s-3mjn; K j K represents the number of syllables contained in the second speech data of the jth unit time; c Indicates the preset reference value of the number of syllables; T dj represents the time length corresponding to the jth unit time; P maxj represents the maximum value of the pitch change rate per j-th unit time; P mjnj represents the minimum value of the pitch change rate per j-th unit time; T mj represents the time interval between the maximum pitch change rate moment and the minimum pitch change rate moment corresponding to the jth unit time; B maxj Indicates the maximum volume intensity that occurs in the jth unit time; B mjnj Indicates the minimum volume intensity that occurs in the jth unit time; B zpj represents the average volume intensity of all syllables that appear in the jth unit time; K dj represents the number of syllables that appear in the time period between the maximum volume intensity moment and the minimum volume intensity moment of the j-th unit time;

[0137] Step 203b: adjusting the second segmented speech rate threshold using the second segmented coefficient to obtain the adjusted second segmented speech rate threshold;

[0138] The adjusted second segment speech rate threshold is obtained by the following formula:

[0139]

[0140] Among them, K t02 represents the adjusted speech rate threshold of the second segment; K 02 Indicates the speech rate threshold of the second segment; S 02 Represents the second segment coefficient; X p02Represents the average slope of the pitch change rate corresponding to m unit times of the second speech data.

[0141] The working principle of the above technical solution is: extract key parameters from the second voice data, including speech rate parameters, volume intensity and pitch change rate. These parameters can reflect the rhythmic characteristics of the voice, volume changes and dynamic characteristics of the fundamental frequency. Using the extracted parameters, the second segmentation coefficient is calculated by a specific formula. The formula comprehensively considers multiple factors, including the number of syllables per unit time, the maximum and minimum values ​​of the pitch change rate, the maximum and minimum values ​​of the volume intensity, and the distribution of these features in time (such as the time interval between the maximum and minimum values ​​of the pitch change rate, the number of syllables appearing in the time period between the maximum and minimum values ​​of the volume intensity, etc.). The combination of these factors is intended to comprehensively evaluate the rhythm and volume characteristics of the second voice data, thereby providing an accurate basis for the calculation of the segmentation coefficient. Using the calculated second segmentation coefficient and the average value of the slope of the pitch change rate of the second voice data, the second segmentation speech rate threshold is adjusted by a formula. The adjusted speech rate threshold is more in line with the actual characteristics of the second voice data, which helps the accuracy and effectiveness of subsequent voice segmentation processing.

[0142] The effect of the above technical solution is: by comprehensively considering multiple speech feature parameters and calculating the second segmentation coefficient using a specific formula, the technical solution can more accurately reflect the rhythm and volume characteristics of the second speech data. Furthermore, using the adjusted speech rate threshold for segmentation processing can significantly improve the accuracy of segmentation and provide a more reliable data basis for subsequent processing (such as speech recognition, speech analysis, etc.). The technical solution can be flexibly adjusted according to the characteristics of different speech data and is suitable for speech data with different speech rates, volumes and pitch changes. This adaptability enables the technical solution to maintain a high processing effect in a variety of scenarios, improving the versatility and practicality of speech processing. Accurate segmentation processing helps to reduce the calculation amount and time cost of subsequent processing and improve the overall processing efficiency. At the same time, the technical solution also uses concepts such as unit time to make the processing process clearer and more orderly, which helps to optimize the processing flow. In application scenarios such as speech recognition and speech synthesis, accurate segmentation processing helps to improve the recognition accuracy and synthesis quality of the system. This is of great significance to improving user experience and enhancing user satisfaction, and helps to promote the development and application of speech technology.

[0143] On the other hand, by retrieving the speech rate parameters, volume intensity and pitch change rate of the second speech data in detail, the solution can process and analyze the speech data more finely. This fine processing helps to capture subtle changes in speech and improve the accuracy and reliability of subsequent processing. The second segmentation coefficient is used to dynamically adjust the second segmentation speech rate threshold so that the speech rate threshold is more consistent with the characteristics of the actual speech data. This dynamic adjustment mechanism helps to more accurately judge the speech rate of speech in the fields of speech recognition and speech analysis, and improve the performance of related applications. The solution integrates multiple dimensions of features such as speech rate parameters, volume intensity and pitch change rate, and calculates the second segmentation coefficient by combining these features. This fusion of multi-dimensional features enables the solution to more comprehensively reflect the characteristics of speech data and improve the accuracy and robustness of processing. The parameters in the formula (such as the value range of unit time, the reference value of the number of syllables, etc.) can be adjusted according to actual conditions. This parameterized design makes the solution more flexible. By adjusting these parameters, it can adapt to the speech data processing needs in different scenarios and improve the applicability of the solution. At the same time, because the solution takes into account the characteristics of multiple dimensions and parameterized design, it can maintain relatively stable performance under different voice data, improving the reliability of long-term operation and large-scale application. The technical solution can process voice data of different lengths, different speech speeds, different volume intensities and pitch change rates, showing strong adaptability. This adaptability enables the solution to be widely used in various fields such as speech recognition and speech analysis.

[0144] In summary, the technical effects of the above-mentioned technical solutions in terms of performance indicators are mainly reflected in the refined processing of voice data, dynamic adjustment of speech rate threshold, multi-dimensional feature fusion to improve accuracy, parametric design to enhance flexibility, computational efficiency and performance stability, and strong adaptability. These technical effects make the solution have higher application value in the fields of speech recognition, speech analysis, etc., and can meet the needs of speech data processing in different scenarios. At the same time, the technical solution comprehensively considers multiple speech feature parameters, calculates the second segmentation coefficient using a specific formula, and adjusts the speech rate threshold for segmentation processing accordingly, thereby achieving technical effects such as improving segmentation accuracy, enhancing adaptability, optimizing processing procedures, and improving user experience. These effects have jointly promoted the advancement of speech processing technology and the expansion of its application scope.

[0145] In one embodiment of the present invention, an ASR automatic speech recognition system is used to convert a speech data segment to be recognized into text, and text data corresponding to the speech data segment to be recognized is obtained, including:

[0146] S201, extracting multiple voice data segments to be recognized;

[0147] S202, adjusting the unit time, obtaining the adjusted unit time length, and using the adjusted unit time as the unit time period; wherein the value range of the unit time period is 10s-1min;

[0148] S203, performing speech preprocessing on each speech data segment to be recognized, and obtaining a plurality of speech data segments to be recognized after preprocessing; wherein the preprocessing includes but is not limited to noise suppression processing and echo cancellation processing;

[0149] S204, retrieving the speech rate parameter corresponding to each speech data segment to be recognized after preprocessing;

[0150] S205, extracting the time frame length corresponding to the preset frame processing;

[0151] S206, obtaining an overlap ratio using a speech rate parameter corresponding to each of the speech data segments to be recognized;

[0152] The overlap ratio is obtained by the following formula:

[0153]

[0154] Among them, P bI Indicates the overlap ratio; P bl0 represents the preset overlap ratio reference value; k represents the number of unit time periods contained in each speech data segment to be recognized; K xi represents the number of syllables corresponding to the i-th unit time period; K wp K represents the average number of syllables per unit time of the parent speech data segment corresponding to each speech data segment to be recognized; c Indicates the preset reference value of the number of syllables;

[0155] S207, performing frame processing on each speech data segment to be recognized after preprocessing according to the preset time frame length corresponding to the frame processing and the overlap ratio, and obtaining an audio data frame corresponding to each speech data segment to be recognized;

[0156] S208: Input the audio data frame corresponding to each speech data segment to be recognized into the ASR automatic speech recognition system to generate text data corresponding to each speech data segment to be recognized.

[0157] The working principle of the above technical solution is as follows: extract multiple speech data segments to be recognized from the original speech data, and these data segments are the basic units for subsequent processing. Adjust the unit time to determine a suitable unit time period length for subsequent framing and processing. The value range of this unit time period is between 10 seconds and 1 minute, and can be flexibly set according to the specific situation. Preprocess each speech data segment to be recognized, including noise suppression processing and echo cancellation processing, etc., to improve speech quality and reduce recognition errors. Obtain the speech rate parameters corresponding to each speech data segment to be recognized after preprocessing. These parameters reflect the speed of speech and have an important impact on subsequent framing and processing. Preset a time frame length corresponding to the framing process, which determines the duration of each audio data frame. Calculate an overlap ratio using the speech rate parameters corresponding to each speech data segment to be recognized, combined with the preset overlap ratio reference value, the number of syllables in the unit time period, the average number of syllables corresponding to the unit time of the upper-level speech data segment, and the preset syllable number reference value. This ratio is used to guide the subsequent frame processing to ensure that there is a certain overlap between adjacent frames, thereby improving the accuracy of recognition. According to the preset time frame length and the calculated overlap ratio, each voice data segment to be recognized is framed to generate multiple audio data frames. These frames are the basic units for the ASR automatic speech recognition system to convert text. The audio data frame corresponding to each voice data segment to be recognized is input into the ASR automatic speech recognition system to generate the corresponding text data. This system uses advanced speech recognition algorithms to convert audio data into text data.

[0158] The effect of the above technical solution is: through fine frame processing and calculation of overlap ratio, the technical solution can ensure that there is a certain overlap between adjacent frames, thereby improving the accuracy of speech recognition. Especially in the case of fast speech speed or poor speech quality, this overlap processing can significantly reduce the recognition error rate. The technical solution can be flexibly adjusted according to the characteristics of different speech data, such as the length of the unit time period, the length of the time frame, and the overlap ratio. This adaptability enables the technical solution to be applicable to a variety of speech data, improving its versatility and practicality. Through reasonable preprocessing and frame processing steps, the technical solution can significantly reduce the calculation amount and time cost of subsequent processing. At the same time, the ASR automatic speech recognition system is used for text conversion, realizing an automated and efficient processing flow. In application scenarios such as speech recognition, accurate text conversion results are crucial to user experience. By improving recognition accuracy and optimizing processing procedures, the technical solution can provide users with more accurate and efficient speech recognition services, thereby enhancing user satisfaction and experience.

[0159] On the other hand, by performing detailed preprocessing on the speech data segments to be recognized, such as noise suppression processing and echo cancellation processing, the impact of background noise and echo on speech recognition can be effectively reduced, thereby improving the accuracy of recognition. The scheme takes into account the speech rate parameters, calculates the overlap ratio based on the speech rate parameters, and then performs frame processing on the speech data. This adaptive frame processing based on speech rate can better capture the key information in the speech and help improve the accuracy of recognition. The technical scheme can process speech data under different speech rates and different noise environments, showing strong adaptability. This adaptability enables the ASR automatic speech recognition system to maintain stable performance in the face of complex and changeable speech environments. By adjusting parameters such as unit time length and overlap ratio, the scheme can flexibly respond to speech data segments of different lengths, further enhancing the robustness of the system. The scheme adopts a frame processing method to cut the speech data into smaller data frames for processing. This method can process multiple data frames in parallel, thereby improving processing efficiency. In addition, through reasonable parameter settings and algorithm optimization, the scheme can improve the calculation speed while ensuring accuracy, meeting application scenarios with high real-time requirements. This technical solution reduces unnecessary calculations through refined preprocessing and frame processing, thereby optimizing resource utilization. In practical applications, this helps reduce system energy consumption and costs and improve overall economic benefits. Accurate speech recognition and efficient processing speed can significantly improve user experience. Whether in smart homes, car navigation or other voice interaction scenarios, users can get a smoother and more natural voice interaction experience.

[0160] In summary, the technical effects of the above technical solution in terms of performance indicators are mainly reflected in improving the accuracy of speech recognition, enhancing the robustness of the system, improving processing efficiency, optimizing resource utilization, and improving user experience. These technical effects make the solution have broad application prospects and important practical value in the field of speech recognition. At the same time, the technical solution achieves technical effects such as improving recognition accuracy, enhancing adaptability, optimizing processing procedures, and improving user experience through fine frame processing, calculation of overlapping ratios, and the application of the ASR automatic speech recognition system. These effects have jointly promoted the development of speech recognition technology and the expansion of its application scope.

[0161] In one embodiment of the present invention, the text data is preprocessed in combination with the speech data segment to be recognized to obtain the preprocessed text data, including:

[0162] S301, retrieve text data corresponding to each voice data segment to be recognized;

[0163] S302, removing blank characters at the beginning and several digits of the text data, redundant spaces, and special symbols that are not Chinese characters, numbers, or English, to obtain first sample data;

[0164] S303, using a word segmentation tool to perform word segmentation processing on the first sample data to obtain second sample data after the word segmentation processing;

[0165] S304, performing error correction and correction processing on the second sample data to obtain error-corrected and corrected text data;

[0166] The error-corrected and amended text data is the pre-processed text data.

[0167] The working principle of the above technical solution is: retrieve the text data obtained after each speech data segment to be recognized is converted by the ASR (automatic speech recognition) system from the previous processing step or storage. These text data are the basis for subsequent preprocessing. Perform preliminary cleaning on the text data to remove blank characters (such as spaces, tabs, etc.) at the beginning and end, as well as redundant spaces in the text. At the same time, remove special symbols that are not Chinese characters, numbers, and English. These symbols may be recognition errors or insignificant marks, which may interfere with subsequent processing. After this step of processing, the first sample data is obtained. The first sample data is segmented using a word segmentation tool. Word segmentation is an important step in natural language processing, which divides continuous text into independent words or phrases. The result of word segmentation directly affects the accuracy of subsequent text analysis, understanding, and processing. After word segmentation, the second sample data is obtained. Error correction and correction processing is performed on the second sample data. This step may include grammatical correction, spelling check, semantic understanding, etc., with the aim of further improving the accuracy and readability of the text data. The error-corrected and amended text data is the preprocessed text data, which can be used for subsequent text analysis, information extraction, sentiment analysis and other applications.

[0168] The effect of the above technical solution is: by removing blank characters, extra spaces and irrelevant special symbols, as well as performing word segmentation and error correction, the technical solution can significantly improve the quality of text data. This helps to reduce noise and interference in subsequent processing and improve the accuracy and efficiency of processing. The preprocessed text data is neater, more standardized, and easier to read and understand. This is particularly important for scenarios that require manual participation or review, which can reduce the burden on staff and improve work efficiency. The preprocessed text data provides a reliable foundation for subsequent natural language processing tasks (such as text classification, sentiment analysis, information extraction, etc.). These tasks rely on high-quality text data, and the preprocessing steps ensure the accuracy and consistency of the data. In application scenarios such as speech recognition and text analysis, high-quality text data can enhance user experience. For example, in an intelligent customer service system, accurate text conversion and preprocessing can enable users to have a smoother and more accurate interactive experience.

[0169] In summary, this technical solution improves the quality, readability and accuracy of subsequent processing of text data through meticulous text preprocessing steps, provides a reliable foundation for natural language processing tasks, and helps improve user experience.

[0170] In one embodiment of the present invention, the deep learning model is a convolutional neural network, and the structure of the deep learning model is as follows:

[0171] Input layer, used to receive preprocessed text data;

[0172] An embedding layer for mapping the index of each word in the text data into a dense vector space of fixed dimension;

[0173] A convolutional layer, used to extract features from the text data and obtain a feature map corresponding to the text data;

[0174] A pooling layer, used for performing dimensionality reduction processing on the feature map;

[0175] The concatenation layer is used to concatenate the feature maps after dimensionality reduction to form feature vectors;

[0176] Fully connected layer, used to integrate the features of feature vectors;

[0177] The output layer is used to output emotion recognition results.

[0178] The working principle of the above technical solution is as follows: Input layer: This layer receives preprocessed text data. Preprocessing usually includes steps such as word segmentation, stop word removal, lemma restoration or stemming to ensure that the text data is suitable for model processing. The preprocessed text data is usually presented in the form of words or character sequences.

[0179] Embedding layer: The embedding layer maps the index of each word in the text data to a dense vector space of fixed dimension. This is usually achieved by looking up a pre-trained word embedding matrix (such as Word2Vec, GloVe, etc.), in which each word is represented as a vector of fixed length. These vectors can capture the semantic and grammatical relationships between words, laying the foundation for subsequent feature extraction.

[0180] Convolutional layer: The convolutional layer uses multiple convolution kernels of different sizes to perform convolution operations on the word vector matrix output by the embedding layer to extract local features of the text. The size of the convolution kernel determines the range of features captured, and multiple convolution kernels can extract a variety of different features. After the convolution operation, an activation function (such as ReLU) is usually used to increase the nonlinearity of the model.

[0181] Pooling layer: The pooling layer performs dimensionality reduction processing on the feature map output by the convolution layer to reduce the feature dimension and retain key information. Commonly used pooling operations include max pooling and average pooling. The pooling layer helps reduce the amount of calculation and improve the robustness of the model.

[0182] Concatenation layer: The concatenation layer concatenates multiple feature maps output by the pooling layer to form a feature vector. This feature vector contains the global features of the text data and provides strong support for subsequent classification tasks.

[0183] Fully connected layer: The fully connected layer integrates and classifies the feature vector output by the concatenation layer. This layer linearly transforms the feature vector through the weight matrix and bias term, and outputs the probability distribution of each category through an activation function (such as softmax).

[0184] Output layer: The output layer gives the final emotion recognition result based on the output of the fully connected layer. Usually, the output layer selects the category with the highest probability as the recognition result.

[0185] The effect of the above technical solution is: through the combination of convolutional layers and pooling layers, the model can efficiently extract local and global features of text data, providing strong support for emotion recognition. Since the convolutional neural network has a strong feature extraction capability, the model can process text data in different fields and has good generalization ability. Compared with some complex deep learning models, the structure of the convolutional neural network is relatively simple and clear, so its output results are easier to explain and understand. The model structure can be flexibly adjusted, such as changing the size and number of convolution kernels, adjusting the pooling method, etc., to adapt to different text classification tasks and data sets.

[0186] In summary, the deep learning model structure described achieves efficient feature extraction and emotion recognition of text data through the collaborative work of input layer, embedding layer, convolution layer, pooling layer, concatenation layer, fully connected layer and output layer. The model has the advantages of strong generalization ability, good interpretability and strong adaptability, and has broad application prospects in fields such as text classification and sentiment analysis.

[0187] The embodiment of the present invention proposes a user call emotion real-time recognition system based on ASR. Figure 2 As shown, the ASR-based user call emotion real-time recognition system includes:

[0188] A voice segmentation module is used to collect voice data of a user in a call in real time, and to segment the voice data using a comprehensive voice segmentation strategy combined with a first segmentation coefficient and a second segmentation coefficient to obtain a plurality of voice data segments to be recognized;

[0189] A text data acquisition module, used to convert the speech data segment to be recognized into text using the ASR automatic speech recognition system, and obtain text data corresponding to the speech data segment to be recognized;

[0190] A text data preprocessing module, used to preprocess the text data in combination with the speech data segment to be recognized, and obtain the preprocessed text data;

[0191] The emotion recognition module is used to input the preprocessed text data into the trained deep learning model for emotion analysis, and obtain the emotion recognition results corresponding to the preprocessed text data.

[0192] The working principle of the above technical solution is as follows: the method first collects the voice data of the user during the call in real time to ensure the timeliness and accuracy of the data. Then, the voice data is segmented by combining the first segmentation coefficient and the second segmentation coefficient using a comprehensive voice segmentation strategy. These coefficients may be set based on the volume, speech speed, pause and other characteristics of the voice to ensure that the segmented voice data segments can more accurately reflect the user's emotional changes. Through segmentation processing, multiple voice data segments to be recognized are obtained to provide basic data for subsequent steps. The ASR automatic speech recognition system is used to convert the voice data segments to be recognized into text. The ASR system can convert voice signals into text information, which is a basic step to achieve emotion recognition. In this process, the ASR system needs to overcome various challenges, such as noise interference, accent differences, etc., to ensure the accuracy and reliability of text conversion. The text data is preprocessed in combination with the voice data segments to be recognized. The preprocessing steps may include removing irrelevant characters, standardizing text formats, correcting spelling errors, etc. The preprocessed text data is clearer and more accurate, which is conducive to the subsequent emotion analysis steps. The preprocessed text data is input into the trained deep learning model for emotion analysis. The deep learning model can learn the emotional features in the text, classify and identify them. Through the calculation and analysis of the deep learning model, the emotion recognition results corresponding to the preprocessed text data are obtained. These results may include multiple emotion categories such as anger, sadness, happiness, etc.

[0193] The effect of the above technical solution is that the method can collect the voice data of the user during the call in real time, and perform segmentation processing and text conversion, so as to realize the real-time recognition of emotions. This helps to capture the emotional changes of users in a timely manner and provide a basis for subsequent decision-making and services. Through the application of comprehensive voice segmentation strategy and ASR automatic voice recognition system, the method can more accurately recognize the user's voice content and convert it into text information. At the same time, the application of deep learning model also improves the accuracy of emotion recognition. The method can be applied to a variety of scenarios, such as customer service, smart home, medical health and other fields. By adjusting the segmentation coefficient and the parameters of the deep learning model, it can adapt to the emotion recognition needs in different scenarios. In response to the needs of different user groups and application scenarios, the method can provide more personalized customized services. For example, in the field of customer service, the service strategy can be adjusted and the service process can be optimized according to the user's emotional state; in the field of smart home, the operation mode and settings of home appliances can be adjusted according to the user's emotional changes.

[0194] In summary, the real-time recognition method of user call emotions based on ASR has technical effects such as real-time, accuracy, flexibility and personalization, and can provide users with a more intelligent, convenient and efficient service experience.

[0195] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for real-time recognition of user conversation emotions based on ASR, characterized in that: include: Collecting voice data of a user in a call in real time, and performing segmentation processing on the voice data by using a comprehensive voice segmentation strategy in combination with a first segmentation coefficient and a second segmentation coefficient, to obtain a plurality of voice data segments to be recognized; Using an ASR automatic speech recognition system to convert the speech data segment to be recognized into text, and obtaining text data corresponding to the speech data segment to be recognized; Preprocessing the text data in combination with the speech data segment to be recognized to obtain preprocessed text data; Inputting the preprocessed text data into a trained deep learning model for sentiment analysis to obtain sentiment recognition results corresponding to the preprocessed text data; The comprehensive speech segmentation strategy includes: Retrieving the speech rate parameter, volume intensity and pitch change rate corresponding to the first voice data; Obtaining a first segmentation coefficient using the speech rate parameter, volume intensity, and pitch change rate corresponding to the first voice data, and adjusting a preset first segmentation speech rate threshold using the first segmentation coefficient to obtain an adjusted first segmentation speech rate threshold; Compare the speech rate in the first speech data with the adjusted first segmented speech rate threshold to obtain first target speech data and second target speech data; The first target voice data is the first voice data whose speech speed exceeds the adjusted first segmented speech speed threshold; the second target voice data is the first voice data whose speech speed does not exceed the adjusted first segmented speech speed threshold; Furthermore, the first target voice data and the second target voice data are the voice data segments to be recognized.

2. The method for real-time recognition of user conversation emotions based on ASR according to claim 1, characterized in that: The voice data of the user during the call is collected in real time, and the voice data is segmented by using a comprehensive voice segmentation strategy combined with a first segmentation coefficient and a second segmentation coefficient to obtain a plurality of voice data segments to be recognized, including: Collect voice data of users during calls in real time; Performing initial segmentation on the voice data using a preset initial speech rate threshold to obtain first voice data and second voice data; Wherein, the first voice data is voice data whose speed exceeds a preset initial speech speed threshold; the second voice data is voice data whose speed does not exceed a preset initial speech speed threshold; The first voice data and the second voice data are segmented respectively using the comprehensive voice segmentation strategy in combination with the first segmentation coefficient and the second segmentation coefficient to obtain the voice data segments to be recognized corresponding to the first voice data and the second voice data.

3. The method for real-time recognition of user conversation emotions based on ASR according to claim 1, characterized in that: The process of obtaining the first segment coefficient includes: Retrieving the speech rate parameter, volume intensity and pitch change rate corresponding to the first voice data; Acquire a first segmentation coefficient corresponding to the first speech data using a speech rate parameter, a volume intensity, and a pitch change rate corresponding to the first speech data; The first segment coefficient is obtained by the following formula: Among them, S 01 represents the first segment coefficient; n represents the number of unit time contained in the first speech data, and the value of the unit time is 5s-3min; K i represents the number of syllables contained in the first speech data of the i-th unit time; K c Indicates the preset reference value of the number of syllables; P si represents the pitch change rate at the start time of the i-th unit time; P zi represents the pitch change rate at the end time of the i-th unit time; T di represents the time length corresponding to the i-th unit time; P maxi represents the maximum value of the pitch change rate per unit time; P mini represents the minimum value of the pitch change rate per unit time; T mi represents the time interval between the maximum value of the pitch change rate and the minimum value of the pitch change rate corresponding to the i-th unit time; B maxi Indicates the maximum volume intensity that occurs in the i-th unit time; B mini Indicates the minimum volume intensity that occurs in the i-th unit time; B zpi represents the average volume intensity of all syllables that appear in the i-th unit time; K di represents the number of syllables that appear in the time period between the maximum volume intensity moment and the minimum volume intensity moment of the i-th unit time; Using the first segmentation coefficient to adjust the first segmentation speech rate threshold, and obtain the adjusted first segmentation speech rate threshold; The adjusted first segment speech rate threshold is obtained by the following formula: Among them, K t01 represents the adjusted speech rate threshold of the first segment; K 01 Indicates the speech rate threshold of the first segment; S 01 represents the first segment coefficient; X p01 Represents the average slope of the pitch change rate corresponding to n unit times of the first speech data.

4. The method for real-time recognition of user conversation emotions based on ASR according to claim 1, characterized in that: The comprehensive speech segmentation strategy also includes: Retrieving the speech speed parameter, volume intensity and pitch change rate corresponding to the second voice data; Obtaining a second segmentation coefficient using the speech rate parameter, volume intensity, and pitch change rate corresponding to the second voice data, and adjusting a preset second segmentation speech rate threshold using the second segmentation coefficient to obtain an adjusted second segmentation speech rate threshold; Compare the speech rate in the second speech data with the adjusted second segmented speech rate threshold to obtain third target speech data and fourth target speech data; The third target voice data is the second voice data whose speech speed exceeds the adjusted second segment speech speed threshold; the fourth target voice data is the second voice data whose speech speed does not exceed the adjusted second segment speech speed threshold; Furthermore, the third target voice data and the fourth target voice data are the voice data segments to be recognized.

5. The method for real-time recognition of user conversation emotions based on ASR according to claim 1 or 4, characterized in that: The process of obtaining the second segment coefficient includes: Retrieving the speech speed parameter, volume intensity and pitch change rate corresponding to the second voice data; Acquire a second segmentation coefficient corresponding to the second voice data using a speech rate parameter, a volume intensity, and a pitch change rate corresponding to the second voice data; The second segment coefficient is obtained by the following formula: Among them, S 02 represents the second segment coefficient; m represents the number of unit time contained in the second speech data, and the value of the unit time is 20s-3mjn; K j K represents the number of syllables contained in the second speech data of the jth unit time; c Indicates the preset reference value of the number of syllables; T dj represents the time length corresponding to the jth unit time; P maxj represents the maximum value of the pitch change rate per j-th unit time; P mjnj represents the minimum value of the pitch change rate per j-th unit time; T mj represents the time interval between the maximum pitch change rate moment and the minimum pitch change rate moment corresponding to the jth unit time; B maxj Indicates the maximum volume intensity that occurs in the jth unit time; B mjnj Indicates the minimum volume intensity that occurs in the jth unit time; B zpj represents the average volume intensity of all syllables that appear in the jth unit time; K dj represents the number of syllables that appear in the time period between the maximum volume intensity moment and the minimum volume intensity moment of the j-th unit time; Using the second segmentation coefficient to adjust the second segmentation speech rate threshold, and obtain the adjusted second segmentation speech rate threshold; The adjusted second segment speech rate threshold is obtained by the following formula: Among them, K t02 represents the adjusted speech rate threshold of the second segment; K 02 Indicates the speech rate threshold of the second segment; S 02 Represents the second segment coefficient; X p02 Represents the average slope of the pitch change rate corresponding to m unit times of the second speech data.

6. The method for real-time recognition of user conversation emotions based on ASR according to claim 1, characterized in that: Using an ASR automatic speech recognition system to convert the speech data segment to be recognized into text, and obtaining text data corresponding to the speech data segment to be recognized, including: Extracting multiple speech data segments to be recognized; Adjust the unit time, obtain the adjusted unit time length, and use the adjusted unit time as the unit time period; wherein the value range of the unit time period is 10s-1min; Performing speech preprocessing on each speech data segment to be recognized, and obtaining a plurality of speech data segments to be recognized after preprocessing; wherein the preprocessing includes noise suppression processing and echo cancellation processing; Retrieving the speech rate parameter corresponding to each speech data segment to be recognized after preprocessing; Extracting the time frame length corresponding to the preset frame processing; Obtaining the overlap ratio using the speech rate parameter corresponding to each speech data segment to be recognized; The overlap ratio is obtained by the following formula: Among them, P bI Indicates the overlap ratio; P bl0 represents the preset overlap ratio reference value; k represents the number of unit time periods contained in each speech data segment to be recognized; K xi represents the number of syllables corresponding to the i-th unit time period; K wp K represents the average number of syllables per unit time of the parent speech data segment corresponding to each speech data segment to be recognized; c Indicates the preset reference value of the number of syllables; Performing frame processing on each speech data segment to be recognized after preprocessing according to the preset time frame length corresponding to the frame processing and the overlap ratio, and obtaining an audio data frame corresponding to each speech data segment to be recognized; The audio data frame corresponding to each speech data segment to be recognized is input into the ASR automatic speech recognition system to generate text data corresponding to each speech data segment to be recognized.

7. The method for real-time recognition of user conversation emotions based on ASR according to claim 1, characterized in that: Preprocessing the text data in combination with the speech data segment to be recognized to obtain the preprocessed text data includes: Retrieving text data corresponding to each voice data segment to be recognized; Remove the blank characters at the beginning and several digits of the text data, redundant spaces, and special symbols that are not Chinese characters, numbers, or English, to obtain first sample data; Using a word segmentation tool to perform word segmentation processing on the first sample data to obtain second sample data after the word segmentation processing; Performing error correction and revision processing on the second sample data to obtain error-corrected and revised text data; The error-corrected and amended text data is the pre-processed text data.

8. The method for real-time recognition of user conversation emotions based on ASR according to claim 1, characterized in that: The deep learning model is a convolutional neural network, and the structure of the deep learning model is as follows: Input layer, used to receive preprocessed text data; An embedding layer for mapping the index of each word in the text data into a dense vector space of fixed dimension; A convolutional layer, used to extract features from the text data and obtain a feature map corresponding to the text data; A pooling layer, used for performing dimensionality reduction processing on the feature map; The concatenation layer is used to concatenate the feature maps after dimensionality reduction to form feature vectors; Fully connected layer, used to integrate the features of feature vectors; The output layer is used to output emotion recognition results.

9. A real-time user call emotion recognition system based on ASR, characterized in that: include: A voice segmentation module is used to collect voice data of a user in a call in real time, and to segment the voice data using a comprehensive voice segmentation strategy combined with a first segmentation coefficient and a second segmentation coefficient to obtain a plurality of voice data segments to be recognized; A text data acquisition module, used to convert the speech data segment to be recognized into text using the ASR automatic speech recognition system, and obtain text data corresponding to the speech data segment to be recognized; A text data preprocessing module, used to preprocess the text data in combination with the speech data segment to be recognized, and obtain the preprocessed text data; An emotion recognition module is used to input the preprocessed text data into a trained deep learning model for emotion analysis, and obtain emotion recognition results corresponding to the preprocessed text data; Wherein, the comprehensive speech segmentation strategy includes: Retrieving the speech rate parameter, volume intensity and pitch change rate corresponding to the first voice data; Obtaining a first segmentation coefficient using the speech rate parameter, volume intensity, and pitch change rate corresponding to the first voice data, and adjusting a preset first segmentation speech rate threshold using the first segmentation coefficient to obtain an adjusted first segmentation speech rate threshold; Compare the speech rate in the first speech data with the adjusted first segmented speech rate threshold to obtain first target speech data and second target speech data; The first target voice data is the first voice data whose speech speed exceeds the adjusted first segmented speech speed threshold; the second target voice data is the first voice data whose speech speed does not exceed the adjusted first segmented speech speed threshold; Furthermore, the first target voice data and the second target voice data are the voice data segments to be recognized.

Images