Speech processing method and related device

Abstract

The invention provides a voice processing method and a related device, and the method comprises the steps: determining a first sub-amplitude spectrum and a second sub-amplitude spectrum according to a target amplitude spectrum of target voice data; then, combining the first sub-amplitude spectrum and the second sub-amplitude spectrum into a fusion amplitude spectrum; then, processing the fusion amplitude spectrum through a first fusion network, and determining fusion feature data; and finally, processing the fused feature data through an emotion recognition network, and determining target emotion data corresponding to the target voice data. According to the method, the magnitude spectrum of the voice data can be split, and different sub-magnitude spectrums are fused, so that information related to emotion in different sub-magnitude spectrums can be fully utilized, and the accuracy of emotion recognition according to voice is greatly improved.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and in particular to a speech processing method and related devices. Background Art

[0002] Speech Emotion Recognition (SER) refers to the technology of identifying the emotional state of a speaker through the acoustic features of a speech. Speech emotion recognition can be applied in many scenarios. For example, in the scenario of customer service, it can accurately identify the emotions of customer service staff through speech emotion recognition, and provide communication guidance to the agents based on the identified emotions, which can improve communication efficiency. Current speech emotion recognition often lacks relevant information in the speech, and the recognition accuracy is low. Summary of the invention

[0003] In view of this, the present application provides a speech processing method and related devices, which can integrate effective information in speech data and improve the accuracy of emotion recognition based on speech.

[0004] In a first aspect, an embodiment of the present application provides a speech processing method, the method comprising:

[0005] Determine a first sub-amplitude spectrum and a second sub-amplitude spectrum according to a target amplitude spectrum of the target speech data;

[0006] Combining the first sub-amplitude spectrum and the second sub-amplitude spectrum into a fused amplitude spectrum;

[0007] Processing the fused amplitude spectrum through a first fusion network to determine fused feature data;

[0008] The fused feature data is processed through an emotion recognition network to determine target emotion data corresponding to the target speech data.

[0009] In a second aspect, an embodiment of the present application provides a speech processing device, the device comprising:

[0010] A splitting unit, used to determine a first sub-amplitude spectrum and a second sub-amplitude spectrum according to a target amplitude spectrum of the target speech data;

[0011] a merging unit, configured to merge the first sub-amplitude spectrum and the second sub-amplitude spectrum into a fused amplitude spectrum;

[0012] A feature unit, used for processing the fused amplitude spectrum through a first fusion network to determine fused feature data;

[0013] The recognition unit is used to process the fused feature data through an emotion recognition network to determine the target emotion data corresponding to the target speech data.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the program includes instructions for executing the steps of any method in the first aspect of the embodiment of the present application.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for electronic data exchange, wherein the computer program enables a computer to execute part or all of the steps described in any method of the first aspect of the embodiment of the present application.

[0016] In a fifth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute some or all of the steps described in any method of the first aspect of the embodiment of the present application. The computer program product may be a software installation package.

[0017] It can be seen that through the above-mentioned speech processing method and related devices, first, the first sub-amplitude spectrum and the second sub-amplitude spectrum are determined according to the target amplitude spectrum of the target speech data; then, the first sub-amplitude spectrum and the second sub-amplitude spectrum are merged into a fused amplitude spectrum; then, the fused amplitude spectrum is processed by the first fusion network to determine the fusion feature data; finally, the fusion feature data is processed by the emotion recognition network to determine the target emotion data corresponding to the target speech data. The amplitude spectrum of the speech data can be split, and different sub-amplitude spectra can be fused, so that the emotion-related information in different sub-amplitude spectra can be fully utilized, which greatly improves the accuracy of emotion recognition based on speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0019] Figure 1 A system architecture diagram of a speech processing method provided in an embodiment of the present application;

[0020] Figure 2 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0021] Figure 3A schematic diagram of a process for obtaining a fused amplitude spectrum provided in an embodiment of the present application;

[0022] Figure 4 A schematic diagram of the structure of a first fusion network provided in an embodiment of the present application;

[0023] Figure 5 A schematic diagram of a process for obtaining fused feature data provided in an embodiment of the present application;

[0024] Figure 6 A schematic diagram of the structure of an emotion recognition network provided in an embodiment of the present application;

[0025] Figure 7 A schematic diagram of a process for obtaining target emotion data provided in an embodiment of the present application;

[0026] Figure 8 A schematic diagram of the structure of another emotion recognition network provided in an embodiment of the present application;

[0027] Fig. 9 Another schematic diagram of a process for obtaining target emotion data provided in an embodiment of the present application;

[0028] Fig.10 A schematic diagram of the structure of a second fusion network provided in an embodiment of the present application;

[0029] Fig.11 A schematic diagram of a process for obtaining secondary fusion feature data provided in an embodiment of the present application;

[0030] Fig.12 A flowchart of a speech processing method provided in an embodiment of the present application;

[0031] Fig.13 A block diagram of the functional units of a speech processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0033] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0034] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article indicates that the associated objects before and after are in an "or" relationship. The "plurality" appearing in the embodiments of the present application refers to two or more.

[0035] In the embodiments of the present application, "at least one item" or similar expressions refer to any combination of these items, including any combination of single items or plural items, and refer to one or more, and multiple refers to two or more. For example, at least one item of a, b, or c can represent the following seven situations: a, b, c, a and b, a and c, b and c, a, b, and c. Among them, each of a, b, and c can be an element or a set containing one or more elements.

[0036] The "connection" that appears in the embodiments of the present application refers to various connection methods such as direct connection or indirect connection to achieve communication between devices, and the embodiments of the present application do not impose any limitations on this.

[0037] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0038] The following is an explanation of the relevant contents, concepts, meanings, technical issues, technical solutions, beneficial effects, etc. involved in the embodiments of the present application.

[0039] I. Related terms

[0040] Spectral characteristics: refers to the signal characteristics in the frequency domain, which is obtained by transforming the time domain signal to the frequency domain using short-time Fourier transform (STFT).

[0041] Amplitude spectrum: When using Fourier transform for analysis, the speech signal needs to be divided into short-time signals, and then Fourier transform is performed on each short-time signal to convert the time domain signal to frequency domain analysis. After the entire speech is divided into segments and Fourier transform is performed, a two-dimensional complex matrix (T, D) can be formed. Usually, when analyzing speech signals, the amplitude contains a lot of paralinguistic information, so the amplitude value of the two-dimensional complex matrix is ​​generally calculated to obtain a two-dimensional amplitude matrix, which is called the amplitude spectrum, where T represents the sum of the speech segments, also known as the number of speech frames, and D represents the frequency scale.

[0042] Sampling rate: refers to the number of times a signal is sampled within 1 second. For example, a sampling rate of 16000 (16kHz) means that the signal is sampled 16,000 times within 1 second, that is, 16,000 sampling points represent 1 second of speech.

[0043] 2. Application scenarios and electronic devices

[0044] 1. Application scenarios

[0045] The embodiments of the present application can be applied to the following application scenarios, including but not limited to: a neural network-based speech processing system deployed on an electronic device, for speech signal processing (such as speech emotion recognition, etc.). The embodiments of the present application can be adjusted and improved according to the specific application environment, and are not specifically limited here.

[0046] In an optional implementation, the speech processing method provided by the present application can be applied to scenarios where speech emotion recognition is required. For example, in an agent customer service scenario, when a customer and an agent are communicating, the customer's communication speech is obtained; the communication speech is then converted into an amplitude spectrum, and the amplitude spectrum is further divided into a first sub-amplitude spectrum and a second sub-amplitude spectrum using the speech processing method of the present application; the first sub-amplitude spectrum and the second sub-amplitude spectrum are then merged into a fused amplitude spectrum, and the fused amplitude spectrum is processed through a first fusion network to obtain fused feature data; the fused feature data is processed through an emotion recognition network to obtain emotion data corresponding to the communication speech.

[0047] Next, based on the emotional data, matching communication scripts are generated for the agents to instruct them to communicate with the customers based on the communication scripts. For example, if it is determined that the emotional data of the communication voice indicates that the customer is currently emotionally agitated and is likely to make complaints, the communication scripts generated for the agents can include some content to soothe the emotions; for another example, if it is determined that the emotional data of the communication voice indicates that the customer is more interested in the current topic of communication, the communication scripts generated for the agents can include more content to recommend the current topic.

[0048] It should be noted that the above is only one application scenario provided by the embodiment of the present application. In actual applications, the speech processing method provided by the embodiment of the present application can be used for any scenario requiring speech emotion recognition, which will not be described one by one here.

[0049] 2. Electronic equipment

[0050] The electronic device in the embodiment of the present application may be a portable electronic device that also includes other functions such as a personal digital assistant and / or a voice receiving function, such as a mobile phone, a tablet computer, a wearable electronic device with a wireless communication function (such as a smart watch), etc. Exemplary embodiments of portable electronic devices include but are not limited to portable electronic devices equipped with an IOS system, an Android system, a Microsoft system, or other operating systems. The above-mentioned portable electronic device may also be other portable electronic devices, such as a laptop computer (Laptop), etc. It should also be understood that in some other embodiments, the above-mentioned electronic device may not be a portable electronic device, but a desktop computer, a server, etc.

[0051] 3. Example

[0052] See also Figure 1 , Figure 1 This is a system architecture diagram of a speech processing method provided in an embodiment of the present application. Figure 1 As shown, the electronic device 100 includes a first fusion network 110 and an emotion recognition network 120. The electronic device 100 can receive voice data and process it to obtain an amplitude spectrum of the voice data, and then determine at least one first sub-amplitude spectrum and at least one second sub-amplitude spectrum of the amplitude spectrum, and input them into the first fusion network 110 for fusion. The fused data is then input into the emotion recognition network 120 for emotion recognition to obtain target emotion data. By determining multiple sub-amplitude spectra, the part of the voice data that is effective for emotion recognition can be determined. By fusion, the relevance and weight of each part can be determined, which can greatly improve the accuracy of recognizing the emotion of the voice data.

[0053] Among them, the first fusion network 110 and the emotion recognition network 120 can be obtained through iterative training. The first fusion network 110 and the emotion recognition network 120 can belong to the same neural network model or different neural network models, which is not specifically limited here.

[0054] Combine the following Figure 2 An electronic device in an embodiment of the present application is described. Figure 21 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application, wherein the electronic device 100 includes a processor 101 , a memory 102 , and a communication bus 103 for connecting the processor 101 and the memory 102 .

[0055] In some possible implementations, the memory 102 includes but is not limited to random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or portable read-only memory (CD-ROM), and the memory 102 is used to store the program code executed by the electronic device 100 and the transmitted data.

[0056] In some possible implementations, the electronic device 100 further includes a communication interface for receiving and sending data.

[0057] In some possible implementations, the processor 101 may be one or more central processing units (CPUs). When the processor 101 is a central processing unit (CPU), the central processing unit (CPU) may be a single-core central processing unit (CPU) or a multi-core central processing unit (CPU).

[0058] In some possible implementations, the processor 101 may be a baseband chip, a chip, a central processing unit (CPU), a general-purpose processor, a DSP, an ASIC, an FPGA or other programmable logic device, a transistor logic device, a hardware component or any combination thereof.

[0059] In a specific implementation, the processor 101 in the electronic device 100 is used to execute the program instructions 121 stored in the memory 102 and execute the instructions of any step in the subsequent method embodiment.

[0060] It is understandable that the electronic device 100 may include more or fewer structural elements than those in the above structural block diagram, for example, a power module, physical buttons, a Wi-Fi module, a speaker, a Bluetooth module, a sensor, a display module, etc., which are not specifically limited here.

[0061] It should be noted that the specific implementation of each operation can adopt the corresponding description of the method embodiment shown above, and the electronic device 100 can be used to execute the following method embodiments of the present application, which will not be described in detail.

[0062] 3. A Speech Processing Method

[0063] Among the current speech recognition emotion methods, the method of using deep learning for recognition is very common, but it often requires a large amount of data training and cannot guarantee the accuracy of the recognition results.

[0064] To solve the above problems, the embodiments of the present application provide a speech processing method and related devices, which can fuse the amplitude spectrum of speech data to determine the effective features therein, thereby greatly improving the accuracy of emotion recognition based on speech.

[0065] The technical solutions, beneficial effects, concepts, etc. involved in the embodiments of the present application are described in detail below.

[0066] 1. How to obtain the fused amplitude spectrum

[0067] In order to fully obtain the information of speech data that is useful for emotion recognition, this application introduces a fused amplitude spectrum, which can enhance the emotion representation of speech data.

[0068] Before obtaining the fused amplitude spectrum, the target amplitude spectrum needs to be determined. First, the target speech data can be pre-emphasized, framed, and windowed, and then divided into multiple short-time signals. Then, Fourier transform is performed on each short-time signal to convert the time domain signal into the frequency domain. A whole segment of target speech data can be transformed into a two-dimensional complex matrix (T, D) after Fourier transform. The target amplitude spectrum of the target speech data can be obtained by calculating the amplitude value of the two-dimensional complex matrix, where T represents the sum of each segment of speech, also known as the number of speech frames, and D represents the frequency scale.

[0069] For example, the sampling rate of the target speech data is 16kHz, which corresponds to 16,000 sampling points, and the length of the target speech data is 1 second. At this time, the window length is set to 50ms (corresponding to 800 sampling points), and the frame shift is 12.5ms (corresponding to 200 sampling points). At this time, the length of the sequence to be Fourier transformed is 1024, which is greater than the 800 sampling points corresponding to the window length, and the length of the sequence to be Fourier transformed is 2 to the 10th power. At this time, the number of speech frames T can be obtained, which is 80 obtained by dividing the length of the target speech data by the frame shift. Due to the characteristics of Fourier transform, when processing a one-dimensional signal such as a speech signal, half of it can reflect all the information, so the frequency scale D can be set to 512. When the frequency scale D takes different values, it represents different frequencies. At this time, the dimension size of the two-dimensional complex matrix can be obtained to be (80,512), that is, the dimension size of the target amplitude spectrum is (80,512).

[0070] It can be seen that by determining the target amplitude spectrum of the target speech data, a reference can be provided for the subsequent sub-amplitude spectrum.

[0071] After obtaining the target amplitude spectrum of the target speech data, it is necessary to determine at least one first sub-amplitude spectrum and at least one second sub-amplitude spectrum. The target amplitude spectrum can be divided into the at least one first sub-amplitude spectrum according to the first filtering range and the first filtering direction, and the target amplitude spectrum can be divided into the at least one second sub-amplitude spectrum according to the second filtering range and the second filtering direction, and the second filtering direction is opposite to the first filtering direction. Among them, the first filtering direction can be the direction from 0 Hz to the maximum Hz, and the second filtering direction is the direction from the maximum Hz to 0 Hz; the first filtering direction can also be the direction from the maximum Hz to 0 Hz, and the second filtering direction is the direction from 0 Hz to the maximum Hz. The first filtering range can be one range or multiple ranges, and the second filtering range can be one range or multiple ranges. There is a corresponding relationship between the first filtering range and the second filtering range. The minimum value of the first filtering range and the maximum value of the first filtering range can be different from the minimum value of the second filtering range and the maximum value of the second filtering range, but the absolute value of the difference between the minimum value of the first filtering range and the maximum value of the first filtering range is the same as the absolute value of the difference between the minimum value of the second filtering range and the maximum value of the second filtering range. For example, the target amplitude spectrum is 0 Hz to 16000 Hz, the first filtering range may be 4000 Hz to 10000 Hz, and the second filtering range may be 12000 Hz to 6000 Hz.

[0072] In a possible embodiment, a first sub-amplitude spectrum and a second sub-amplitude spectrum may be determined. It should be noted that the dimension size of the first sub-amplitude spectrum needs to be the same as the dimension size of the second sub-amplitude spectrum. When splitting the first sub-amplitude spectrum and the second sub-amplitude spectrum, it is sufficient to ensure that the frequency bands split along the corresponding filtering direction correspond. For example, the first sub-amplitude spectrum may correspond to 1000 Hz to 13000 Hz, and the second sub-amplitude spectrum may correspond to 15000 Hz to 3000 Hz; the first sub-amplitude spectrum may correspond to 2000 Hz to 12000 Hz, and the second sub-amplitude spectrum may correspond to 14000 Hz to 4000 Hz; it is convenient for subsequent fusion. At the same time, since a large amount of language information is contained in the low-frequency band of the speech, and a large amount of paralinguistic information (such as emotion, style and rhythm information) is contained in the middle and high-frequency bands of the speech, the language information and the paralinguistic information can be taken into account when splitting the first sub-amplitude spectrum and the second sub-amplitude spectrum. After fusion, the accuracy of subsequent emotion recognition can be higher. The first sub-amplitude spectrum and the second sub-amplitude spectrum can be combined to obtain a fused amplitude spectrum. It can be understood that the number of speech frames T of the first sub-amplitude spectrum and the second sub-amplitude spectrum is the same, the first dimension of the first sub-amplitude spectrum can be (T, D1), and the second dimension of the second sub-amplitude spectrum is also (T, D1). The dimension of the fused amplitude spectrum can be (B, 2, T, D1), and B represents the batch size. It can take a complex value in the neural network training stage. At this time, the neural network is already trained and can take a value of 1. In this way, it can be ensured that the first sub-amplitude spectrum and the second sub-amplitude spectrum can be paid attention to at the same time when the fused feature data is obtained later.

[0073] In a possible embodiment, at least two first sub-amplitude spectra and at least two second sub-amplitude spectra may be determined. During the splitting process, it is necessary to ensure that the dimension size of one of the first sub-amplitude spectra is the same as the dimension size of one of the second sub-amplitude spectra, and the dimension size of another first sub-amplitude spectrum is the same as the dimension size of another second sub-amplitude spectrum. At the same time, the dimensions of the two groups are different. The first dimension of each first sub-amplitude spectrum may be determined, and the second dimension of each second sub-amplitude spectrum may be determined. Then, the first sub-amplitude spectra and the second sub-amplitude spectra having the same first dimension and the same second dimension may be screened out. Finally, each group of first sub-amplitude spectra and second sub-amplitude spectra having the same first dimension and the same second dimension may be merged into the fused amplitude spectrum.

[0074] For example, see Figure 3 , Figure 3A schematic diagram of a process for obtaining a fused amplitude spectrum provided in an embodiment of the present application, wherein three first sub-amplitude spectra are divided from 0 Hz to 16 kHz, namely, a first low-frequency sub-amplitude spectrum, a first intermediate-frequency sub-amplitude spectrum and a first high-frequency sub-amplitude spectrum, wherein the first low-frequency sub-amplitude spectrum corresponds to (0, 4000) Hz, the first intermediate-frequency sub-amplitude spectrum corresponds to (2000, 12000) Hz, and the first high-frequency sub-amplitude spectrum corresponds to (8000, 16000) Hz. Setting an overlapping area during division can make subsequent feature extraction more sufficient. When the number of speech frames T is constant and the frequency scale corresponding to 16 kHz is 512, the dimension size of the first low-frequency sub-amplitude spectrum is (T, 0: 128) = (T, 128), the dimension size of the first intermediate-frequency sub-amplitude spectrum is (T, 64: 384) = (T, 320), and the dimension size of the first high-frequency sub-amplitude spectrum is (T, 256: 512) = (T, 256). Three second sub-amplitude spectra are divided from 16kHz to 0Hz, namely the second high-frequency sub-amplitude spectrum, the second intermediate-frequency sub-amplitude spectrum and the second low-frequency sub-amplitude spectrum. The second high-frequency sub-amplitude spectrum corresponds to (16000, 12000) Hz, the second intermediate-frequency sub-amplitude spectrum corresponds to (14000, 4000) Hz, and the second low-frequency sub-amplitude spectrum corresponds to (8000, 0) Hz. Similarly, when the number of speech frames T is constant and the frequency scale corresponding to 16kHz is 512, the dimension of the second high-frequency sub-amplitude spectrum is (T, 512: 384) = (T, 128), and the second intermediate-frequency sub-amplitude spectrum corresponds to (14000, 4000) Hz. The dimension size of the spectrum is (T, 448: 128) = (T, 320), and the dimension size of the second low-frequency sub-amplitude spectrum is (T, 256: 0) = (T, 256). It can be determined that the dimension size of the first low-frequency sub-amplitude spectrum and the dimension size of the second high-frequency sub-amplitude spectrum are both (T, 128), defined as (T, D1); the dimension size of the first intermediate frequency sub-amplitude spectrum and the dimension size of the second intermediate frequency sub-amplitude spectrum are both (T, 320), defined as (T, D2); the dimension size of the first high frequency sub-amplitude spectrum and the dimension size of the second low frequency sub-amplitude spectrum are both (T, 256), defined as (T, D3). The first low-frequency sub-amplitude spectrum and the second high-frequency sub-amplitude spectrum are combined to obtain a first fused amplitude spectrum, whose dimension size is (B, 2, T, D1); the first intermediate frequency sub-amplitude spectrum and the second intermediate frequency sub-amplitude spectrum are combined to obtain a second fused amplitude spectrum, whose dimension size is (B, 2, T, D2); the first high-frequency sub-amplitude spectrum and the second low-frequency sub-amplitude spectrum are combined to obtain a third fused amplitude spectrum, whose dimension size is (B, 2, T, D3); it should be noted that B here represents the batch size, which can be 1 at this time. In this way, multiple fused amplitude spectra can be obtained, which respectively focus on the speech content and energy information corresponding to the low frequency, the speech content, part of the emotion, part of the rhythm and other information corresponding to the intermediate frequency, and the emotion, style, rhythm and other information corresponding to the high frequency, so as to ensure that the subsequent feature extraction can obtain more accurate and rich speech-related information and improve the accuracy of speech emotion recognition.

[0075] The above examples are possible ways of dividing the sub-amplitude spectra, and do not represent specific limitations on the embodiments of the present application. Other ways of dividing can be adopted according to actual needs, and are not specifically limited here.

[0076] 2. The first fusion network

[0077] In order to obtain relevant information of speech data that is effective for emotion recognition, the embodiment of the present application introduces the concept of a first fusion network, which can enhance the emotion representation of speech data.

[0078] Each input of the first fusion network is a fused amplitude spectrum, and each output of the first fusion network is fused feature data. Before explaining how to process the fused amplitude spectrum after it is input into the first fusion network, the structure of the first fusion network is first explained.

[0079] The first fusion network may include N first convolution modules, a first weight module, and a first output module, where N is a natural number greater than 2, wherein the N first convolution modules are used to perform convolution processing on the fused amplitude spectrum to output N first eigenvectors, the first weight module is used to perform a first operation on M eigenvectors of the N first eigenvectors, and process the first operation result to obtain a first weighted eigenvector, where M is a natural number less than N and greater than 1, and the first output module is used to perform a second operation on the first weighted eigenvector and the NM first eigenvectors, and process the second operation result to obtain fused feature data, so that fused feature data containing more accurate and rich speech-related information can be obtained, so that subsequent speech emotion recognition is more accurate.

[0080] The first operation and the second operation may both refer to multiplication operations. For ease of understanding in this application, it is assumed that N is equal to 3 and M is equal to 2. Figure 4 , Figure 4 A structural schematic diagram of a first fusion network provided for an embodiment of the present application, the first fusion network includes a first convolution module 410, a first convolution module 420, a first convolution module 430, a first weight module 440, and a first output module 450, wherein the first convolution module 410, the first convolution module 420 and the first convolution module 430 have the same structure and can include a one-dimensional convolution kernel, the first weight module 440 can be used to obtain the respective weights of the two sub-amplitude spectra in the fused amplitude spectrum, and the first output module 450 can be used to obtain fused feature data in a target dimensional format.

[0081] The following processing may be performed on each fused amplitude spectrum to determine each fused feature data:

[0082] First, the fused amplitude spectrum is transformed into a preset dimensional format to obtain a fused amplitude spectrum in the preset dimensional format, and then the fused amplitude spectrum in the preset dimensional format is input into the first convolution module 410 to obtain a first eigenvector t1; the fused amplitude spectrum in the preset dimensional format is input into the first convolution module 420 to obtain a first eigenvector t2; the fused amplitude spectrum in the preset dimensional format is input into the first convolution module 430 to obtain a first eigenvector t3; then, the first eigenvector t1 and the first eigenvector t2 are matrix multiplied and input into the first weight module 440 to obtain a first weighted eigenvector; finally, the first weighted eigenvector and the first eigenvector t3 are multiplied and input into the first output module 450 to obtain fused feature data in the target dimensional format.

[0083] In a possible embodiment, the first weight module 440 may include a first linear layer and a first classification softmax layer, and the first output module 450 may include a first pooling layer and a second linear layer.

[0084] For ease of understanding, the following is an exemplary description of how to obtain fusion feature data, see Figure 5 , Figure 5A flow chart of obtaining fused feature data is provided for an embodiment of the present application. The feature dimension of the fused amplitude spectrum is (B, 2, T, D1). First, the feature dimension of the fused amplitude spectrum is changed to a preset dimension format, namely (B, 2, T*D1). Then, the fused amplitude spectrum of (B, 2, T*D1) is input into the first convolution kernel, the second convolution kernel and the third convolution kernel respectively. The first convolution kernel outputs the first feature vector t1, the second convolution kernel outputs the first feature vector t2, and the third convolution kernel outputs the first feature vector t3. At this time, the dimensions of the first feature vector t1, the first feature vector t2 and the first feature vector t3 remain unchanged, and are still (B, 2, T*D1). Then, the first feature vector t1 and the first feature vector t2 are matrix multiplied to obtain the first intermediate data with a dimension of (B, 2, 2). This is to determine the correlation between the first feature vector t1 and the first feature vector t2 in the future. Then, The first intermediate data is input into the first linear layer to obtain the second intermediate data with a dimension of (B, 2, 1), and then the second intermediate data is input into the first softmax layer to obtain the first weighted feature vector. At this time, the dimension remains unchanged and is still (B, 2, 1). Then, the first weighted feature vector is multiplied by the first feature vector t3, so that the degree of attention to the first sub-amplitude spectrum and the second sub-amplitude spectrum can be determined. Then, the first pooling layer is summed to convert the dimension into (B, T*D1), and then the dimension is restored to obtain (B, T, D1). Finally, the second linear layer is used to map the dimension D1 to Dx to obtain the fused feature data of the target dimension format (B, T, Dx). It should be noted that Dx can be D1 or other values. The purpose of unifying it as Dx is to facilitate subsequent processing when there are multiple fused feature data.

[0085] It can be understood that the above example is a one-time input and output process. When there are multiple fused amplitude spectra, the process can be executed multiple times, either in a serial manner or in a parallel manner, and no specific limitation is made here.

[0086] It can be seen that by obtaining the fused feature data through the first fusion network, we can focus on the different amplitude spectrum parts in the speech data, and focus on the parts with higher weights, which greatly improves the accuracy of identifying the emotions corresponding to the speech data.

[0087] 2. Emotion Recognition Network

[0088] In order to determine the target emotion data corresponding to the target speech data, the embodiment of the present application introduces an emotion recognition network, the input of which is at least one fusion feature data, and the output is the target emotion data.

[0089] (1) A possible emotion recognition network

[0090] In one possible embodiment, see Figure 6, Figure 6 A structural diagram of an emotion recognition network provided in an embodiment of the present application, the emotion recognition network may include a dimension processing module 610 and an emotion classification module 620. Among them, the dimension processing module 610 may include a maximum pooling layer for performing summation processing of the time dimension to obtain fused feature data in a preset dimensional format, and the emotion classification module 620 may include a fully connected layer and a second classification softmax layer for obtaining the confidence of each preset emotion category, and finally the target emotion data may be determined according to the preset emotion category whose confidence meets the preset condition. Optionally, the preset condition may be to select the confidence with the highest confidence, or the preset condition may also be that the confidence meets a certain threshold.

[0091] For ease of understanding, the following Figure 7 An exemplary explanation of how to obtain the target emotion data in the embodiment of the present application is given, see the description in 1, the dimension of the fused feature data is (B, T, Dx), the maximum pooling layer is input and the dimension is transformed to (B, Dx), and then the dimension is transformed to (B, 7) through the fully connected layer, where 7 represents the preset emotion category, including anger, boredom, anxiety, happiness, sadness, disgust, and neutrality. It can be understood that the number and type of preset emotion categories can be flexibly adjusted. This is only an exemplary explanation and does not represent a specific limitation on the embodiment of the present application. Finally, the confidence of each preset emotion category can be output through the second classification softmax layer, and the one with the highest confidence is used as the target emotion data (B, 1).

[0092] It can be seen that when there is a fused feature data, the emotion fusion network can be used to directly output the target emotion data. The fused feature data has already focused on the parts of the speech data that are strongly related to emotion recognition, and has determined the degree of attention of different parts based on weights, which greatly improves the accuracy of speech emotion recognition.

[0093] (2) Another possible emotion recognition network

[0094] In one possible embodiment, see Figure 8 , Figure 8A schematic diagram of the structure of another emotion recognition network provided in an embodiment of the present application, the emotion recognition network may include a dimension processing module 610, an emotion classification module 620, and a second fusion network 630. Among them, the dimension processing module 610 may include a maximum pooling layer for performing summation processing of the time dimension to obtain fused feature data in a preset dimensional format, and the emotion classification module 620 may include a fully connected layer and a second classification softmax layer for obtaining the confidence of each preset emotion category, and finally the target emotion data may be determined according to the preset emotion category with the highest confidence. The second fusion network 630 may be used to fuse at least two fused feature data again to obtain secondary fused feature data.

[0095] The following processing may be performed on at least two fused feature data to determine the target emotion data:

[0096] First, the at least two fused feature data are merged into spliced ​​feature data; then, the spliced ​​feature data is preprocessed and input into the second fusion network 630 to obtain secondary fused feature data; then, the secondary fused feature data in the target dimensional format is input into the dimension processing module 610 to obtain secondary fused feature data in a preset dimensional format; then, the secondary fused feature data in the preset dimensional format is input into the sentiment classification module 620 to obtain the confidence of each preset sentiment category; finally, the target sentiment data is determined according to the preset sentiment category with the highest confidence.

[0097] For ease of understanding, the following Fig. 9An exemplary description of how to obtain the target emotion data in the embodiment of the present application is given, referring to the description in 1, the dimensions of at least two fused feature data are (B, T, Dx), and the fused feature data in the at least two target dimension formats are merged into the spliced ​​feature data, and the dimensions of the spliced ​​feature data are (B, S, T, Dx), where S is the number of fused feature data, that is, S is 2 when there are two fused feature data, S is 3 when there are three fused feature data, and S is a natural number greater than or equal to 2, which will not be described in detail here; then, the spliced ​​feature data is preprocessed and input into the second fused feature data. Network 630, to obtain the secondary fusion feature data of the target dimensional format (B, T, Dx); then, the secondary fusion feature data of the target dimensional format (B, T, Dx) is input into the maximum pooling layer and deformed to convert the dimension into (B, Dx), and then the dimension is transformed into (B, 7) through the fully connected layer, where 7 represents the preset emotion category, including anger, boredom, anxiety, happiness, sadness, disgust, and neutrality. It can be understood that the number and type of preset emotion categories can be flexibly adjusted. This is only an exemplary explanation and does not represent a specific limitation on the embodiments of the present application. Finally, the confidence of each preset emotion category can be output through the second softmax layer, and the one with the highest confidence is used as the target emotion data.

[0098] The structure of the second fusion network is the same as that of the first fusion network. The second fusion network can be the first fusion network, that is, the first fusion network is reused, or a new second fusion network with the same structure as the first fusion network can be created.

[0099] For details, see Fig.10 , Fig.10 A structural schematic diagram of a second fusion network provided for an embodiment of the present application, the second fusion network may include a second convolution module 1010, a second convolution module 1020, a second convolution module 1030, a second weight module 1040, and a second output module 1050. The second weight module 1040 can be used to obtain the respective weights of each fused feature data in the spliced ​​feature data, and the second output module 1050 can be used to obtain secondary fused feature data in a target dimensional format.

[0100] The following processing may be performed on each spliced ​​feature data to determine each secondary fusion feature data:

[0101] First, the splicing feature data is transformed into the preset dimensional format to obtain the splicing feature data in the preset dimensional format; then, the splicing feature data in the preset dimensional format is input into the second convolution module 1010 to obtain the second feature vector t4; the splicing feature data in the preset dimensional format is input into the second convolution module 1020 to obtain the second feature vector t5; the splicing feature data in the preset dimensional format is input into the second convolution module 1030 to obtain the second feature vector t6; the second feature vector t4 and the second feature vector t5 are matrix multiplied and input into the second weight module 1040 to obtain the second weight feature vector; the second weight feature vector and the second feature vector t6 are multiplied and input into the second output module 1050 to obtain the secondary fusion feature data in the target dimensional format.

[0102] In a possible embodiment, the second weight module 1040 may include a third linear layer and a third classification softmax layer, and the second output module 1050 may include a second pooling layer. For ease of understanding, the following is an exemplary description of how to obtain the secondary fusion feature data, see Fig.11 , Fig.11 A flow chart of obtaining secondary fusion feature data provided in an embodiment of the present application. If the feature dimension of the spliced ​​feature data is (B, 3, T, Dx), first, the feature dimension of the spliced ​​feature data is changed to a preset dimension format, namely (B, 3, T*Dx), then the spliced ​​feature data of (B, 3, T*Dx) is input into the fourth convolution kernel, the fifth convolution kernel and the sixth convolution kernel respectively, the fourth convolution kernel outputs the second feature vector t4, the fifth convolution kernel outputs the second feature vector t5, and the sixth convolution kernel outputs the second feature vector t6. At this time, the dimensions of the second feature vector t4, the second feature vector t5 and the second feature vector t6 remain unchanged, and are still (B, 3, T*Dx). Then, the second feature vector t4 and the second feature vector t5 are matrix multiplied to obtain a dimension of (B, 3, 2 ), here is to determine the correlation between the second eigenvector t4 and the second eigenvector t5 later, then input the third intermediate data into the third linear layer to obtain the fourth intermediate data with dimension (B, 3, 1), then input the fourth intermediate data into the third classification softmax layer to obtain the second weighted feature vector, at this time the dimension remains unchanged, still (B, 3, 1), then, multiply the second weighted feature vector by the second eigenvector t6, so that the degree of attention to each fused feature data can be determined, then the second pooling layer is used to sum the dimension to convert it to (B, T*Dx), and then the dimension is restored to obtain the secondary fused feature data with dimension (B, T, Dx).

[0103] It can be seen that when there are at least two fused feature data, the emotion fusion network can be used to perform secondary fusion to obtain secondary fused feature data, and then output the target emotion data. Since each fused feature data has already focused on the part of the speech data that is strongly relevant to emotion recognition, and has determined the degree of attention to different parts based on the weight, the secondary fusion can again determine the degree of attention to each fused feature data, greatly improving the accuracy of speech emotion recognition.

[0104] IV. Exemplary Description of a Speech Processing Method

[0105] In combination with the above content, a speech processing method according to an embodiment of the present application is introduced as an example below.

[0106] like Fig.12 As shown, Fig.12 A flow chart of a speech processing method provided in an embodiment of the present application specifically includes the following steps:

[0107] Step 1201, determining a first sub-amplitude spectrum and a second sub-amplitude spectrum according to a target amplitude spectrum of target speech data.

[0108] In a possible embodiment, the target amplitude spectrum can be divided into a first sub-amplitude spectrum according to a first filtering range and a first filtering direction; the target amplitude spectrum can be divided into a second sub-amplitude spectrum according to a second filtering range and a second filtering direction, and the second filtering direction is opposite to the first filtering direction.

[0109] In this way, relevant information that is effective during speech emotion recognition can be extracted as much as possible, thereby improving the accuracy of speech emotion recognition.

[0110] Step 1202: merge the first sub-amplitude spectrum and the second sub-amplitude spectrum into a fused amplitude spectrum.

[0111] In a possible embodiment, the first dimension of each first sub-amplitude spectrum may be determined, and the second dimension of each second sub-amplitude spectrum may be determined; then, the first sub-amplitude spectrum and the second sub-amplitude spectrum having the same first dimension and second dimension may be merged into the fused amplitude spectrum.

[0112] Step 1203: Process the fused amplitude spectrum through a first fusion network to determine fused feature data.

[0113] The first fusion network includes N first convolution modules, a first weight module, and a first output module, where N is a natural number greater than 2. The following processing can be performed on each fusion amplitude spectrum to determine each fusion feature data:

[0114] Inputting the fused amplitude spectrum into the N first convolution modules for convolution processing to obtain N first eigenvectors;

[0115] Performing a first operation on M feature vectors of the N first feature vectors, and processing the first operation result by the first weight module to obtain a first weighted feature vector, where M is a natural number less than N and greater than 1;

[0116] A second operation is performed on the first weighted feature vector and the NM first feature vectors, and the second operation result is processed by the first output module to obtain fused feature data.

[0117] It can be seen that in this way, the degree of attention to each sub-amplitude spectrum can be determined, thereby improving the accuracy of speech emotion recognition.

[0118] Step 1204: Process the fused feature data through an emotion recognition network to determine target emotion data corresponding to the target speech data.

[0119] In a possible embodiment, the fused feature data is summed in the time dimension through the emotion recognition network to obtain the fused feature data in a preset dimensional format; the fused feature data in the preset dimensional format is processed through the emotion recognition network to obtain the confidence of each preset emotion category; and the target emotion data is determined according to the preset emotion category with the highest confidence.

[0120] It can be seen that when there is a fusion feature data, the emotion fusion network can be used to directly output the target emotion data. The fusion feature data has focused on the part of the speech data that is strongly related to emotion recognition, and has determined the degree of attention of different parts based on weights, which greatly improves the accuracy of speech emotion recognition.

[0121] In a possible embodiment, the number of the fused feature data is at least two; at least two fused feature data are merged into spliced ​​feature data;

[0122] The spliced ​​feature data is subjected to secondary fusion processing through the emotion recognition network to obtain secondary fusion feature data, and the secondary fusion feature data is processed to obtain the confidence of each preset emotion category; the target emotion data is determined according to the preset emotion category with the highest confidence.

[0123] Furthermore, the second fusion network includes N second convolution modules, a second weight module, and a second output module, where N is a natural number greater than 2; the spliced ​​feature data can be input into the N second convolution modules for convolution processing to obtain N second feature vectors; M feature vectors of the N second feature vectors are subjected to a second operation, and the second operation result is processed by the second weight module to obtain a second weighted feature vector, where M is a natural number less than N and greater than 1; the second weighted feature vector and the NM second feature vectors are subjected to a third operation, and the third operation result is processed by the second output module to obtain secondary fusion feature data.

[0124] It can be seen that when there are at least two fused feature data, the emotion fusion network can be used to perform secondary fusion to obtain secondary fused feature data, and then output the target emotion data. Since each fused feature data has already focused on the part of the speech data that is strongly relevant to emotion recognition, and has determined the degree of attention to different parts based on the weight, the secondary fusion can again determine the degree of attention to each fused feature data, greatly improving the accuracy of speech emotion recognition.

[0125] Through the above-mentioned speech processing method, first, the first sub-amplitude spectrum and the second sub-amplitude spectrum are determined according to the target amplitude spectrum of the target speech data; then, the first sub-amplitude spectrum and the second sub-amplitude spectrum are merged into a fused amplitude spectrum; then, the fused amplitude spectrum is processed by the first fusion network to determine fusion feature data; finally, the fused feature data is processed by the emotion recognition network to determine the target emotion data corresponding to the target speech data. The amplitude spectrum of the speech data can be split, and different sub-amplitude spectra can be fused, so that the emotion-related information in different sub-amplitude spectra can be fully utilized, which greatly improves the accuracy of emotion recognition based on speech.

[0126] For the parts not described in detail above, please refer to the description of 1-2 and will not be repeated here.

[0127] 5. Exemplary Description of a Speech Processing Device

[0128] The above mainly introduces the scheme of the embodiment of the present application from the perspective of the execution process on the method side. It is understandable that, in order to realize the above functions, the electronic device includes a hardware structure and / or software module corresponding to the execution of each function. It should be easily appreciated by those skilled in the art that, in combination with the units and algorithm steps of each example described in the embodiments provided herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.

[0129] The embodiment of the present application can divide the electronic device into functional units according to the above method example. For example, each functional unit can be divided according to each function, or two or more functions can be integrated into one processing unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of units in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0130] In the case of dividing each functional module into corresponding functional modules, Fig.13 This is a block diagram of the functional units of a speech processing device provided in an embodiment of the present application. The speech processing device 1300 includes:

[0131] A splitting unit 1310 is used to determine a first sub-amplitude spectrum and a second sub-amplitude spectrum according to a target amplitude spectrum of the target speech data;

[0132] A merging unit 1320, configured to merge the first sub-amplitude spectrum and the second sub-amplitude spectrum into a fused amplitude spectrum;

[0133] A feature unit 1330 is used to process the fused amplitude spectrum through a first fusion network to determine fused feature data;

[0134] The recognition unit 1340 is used to process the fused feature data through an emotion recognition network to determine the target emotion data corresponding to the target speech data.

[0135] It can be seen that through the above-mentioned speech processing method and related devices, first, the first sub-amplitude spectrum and the second sub-amplitude spectrum are determined according to the target amplitude spectrum of the target speech data; then, the first sub-amplitude spectrum and the second sub-amplitude spectrum are merged into a fused amplitude spectrum; then, the fused amplitude spectrum is processed by the first fusion network to determine the fusion feature data; finally, the fusion feature data is processed by the emotion recognition network to determine the target emotion data corresponding to the target speech data. The amplitude spectrum of the speech data can be split, and different sub-amplitude spectra can be fused, so that the emotion-related information in different sub-amplitude spectra can be fully utilized, which greatly improves the accuracy of emotion recognition based on speech.

[0136] It should be noted that the specific implementation of each operation can adopt the corresponding description of the method embodiment shown above, and the speech processing device 1300 can be used to execute the above method embodiment of the present application, which will not be described in detail.

[0137] 6. Other examples

[0138] An embodiment of the present application also provides a chip, including a processor, a memory, and a computer program or instructions stored in the memory, wherein the processor executes the computer program or instructions to implement the steps described in the above method embodiment.

[0139] An embodiment of the present application also provides a chip module, including a transceiver component and a chip, the chip including a processor, a memory and a computer program or instructions stored in the memory, wherein the processor executes the computer program or instructions to implement the steps described in the above method embodiment.

[0140] An embodiment of the present application also provides a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, wherein the computer program enables a computer to execute part or all of the steps of any method recorded in the above method embodiment, and the above computer includes an electronic device.

[0141] The embodiment of the present application also provides a computer program product, the computer program product includes a non-transitory computer-readable storage medium storing a computer program, the computer program is operable to cause a computer to execute some or all of the steps of any method described in the method embodiment. The computer program product may be a software installation package, and the computer includes an electronic device.

[0142] It should be noted that, for the above-mentioned various embodiments, for the sake of simple description, they are all expressed as a series of action combinations. Those skilled in the art should be aware that the present application is not limited by the described order of actions, because some steps in the embodiments of the present application can be performed in other orders or simultaneously. In addition, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions, steps, modules or units involved are not necessarily required by the embodiments of the present application.

[0143] In the above embodiments, the embodiments of the present application have different focuses on the description of each embodiment. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0144] The steps of the method or algorithm described in the embodiments of the present application can be implemented in hardware or by executing software instructions by a processor. The software instructions can be composed of corresponding software modules, and the software modules can be stored in RAM, flash memory, ROM, EPROM, electrically erasable programmable read-only memory (electrically EPROM, EEPROM), registers, hard disks, mobile hard disks, read-only compact disks (CD-ROMs) or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a terminal device or a management device. Of course, the processor and the storage medium can also be present in a terminal device or a management device as discrete components.

[0145] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiments of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server, or data center to another website site, computer, server, or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital video disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0146] The modules / units included in the devices and products described in the above embodiments may be software modules / units or hardware modules / units, or may be partially software modules / units and partially hardware modules / units. For example, for the devices and products applied to or integrated in the chip, the modules / units included therein may all be implemented in the form of hardware such as circuits, or at least some of the modules / units may be implemented in the form of software programs, which run on the processor integrated inside the chip, and the remaining (if any) modules / units may be implemented in the form of hardware such as circuits; for the devices and products applied to or integrated in the chip module, the modules / units included therein may all be implemented in the form of hardware such as circuits, and different modules / units may be located in the same component (such as a chip, circuit module, etc.) or in different components of the chip module, or at least some of the modules / units may be implemented in the form of software programs. The software programs run on the processor integrated inside the chip, and the remaining (if any) modules / units may be implemented in the form of hardware such as circuits. It is implemented in the form of a software program, which runs on a processor integrated inside the chip module, and the remaining (if any) modules / units can be implemented in hardware such as circuits; for various devices and products applied to or integrated in the terminal equipment, the various modules / units contained therein can be implemented in hardware such as circuits, and different modules / units can be located in the same component (for example, chip, circuit module, etc.) or in different components in the terminal equipment, or, at least some modules / units can be implemented in the form of a software program, which runs on a processor integrated inside the terminal equipment, and the remaining (if any) modules / units can be implemented in hardware such as circuits.

[0147] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the embodiments of the present application. It should be understood that the above description is only the specific implementation method of the embodiments of the present application and is not intended to limit the protection scope of the embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the embodiments of the present application should be included in the protection scope of the embodiments of the present application.

Claims

1. A speech processing method, characterized in that: The method comprises: Determine a first sub-amplitude spectrum and a second sub-amplitude spectrum according to a target amplitude spectrum of the target speech data; Combining the first sub-amplitude spectrum and the second sub-amplitude spectrum into a fused amplitude spectrum; Processing the fused amplitude spectrum through a first fusion network to determine fused feature data; The fused feature data is processed through an emotion recognition network to determine target emotion data corresponding to the target speech data.

2. The method according to claim 1, characterized in that The determining of the first sub-amplitude spectrum and the second sub-amplitude spectrum according to the target amplitude spectrum of the target speech data comprises: Dividing the target amplitude spectrum into a first sub-amplitude spectrum according to a first filtering range and a first filtering direction; The target amplitude spectrum is divided into a second sub-amplitude spectrum according to a second filtering range and a second filtering direction, wherein the second filtering direction is opposite to the first filtering direction.

3. The method according to claim 1, characterized in that The step of merging the first sub-amplitude spectrum and the second sub-amplitude spectrum into a fused amplitude spectrum comprises: determining a first dimension of each first sub-amplitude spectrum, and determining a second dimension of each second sub-amplitude spectrum; The first sub-amplitude spectrum and the second sub-amplitude spectrum having the same first dimension and second dimension are combined to obtain a fused amplitude spectrum.

4. The method according to claim 1, characterized in that: The first fusion network includes N first convolution modules, a first weight module, and a first output module, where N is a natural number greater than 2; the fusion amplitude spectrum is processed by the first fusion network to determine the fusion feature data, including: Inputting the fused amplitude spectrum into the N first convolution modules for convolution processing to obtain N first eigenvectors; Performing a first operation on M feature vectors of the N first feature vectors, and processing the first operation result by the first weight module to obtain a first weighted feature vector, where M is a natural number less than N and greater than 1; A second operation is performed on the first weighted feature vector and the NM first feature vectors, and the second operation result is processed by the first output module to obtain fused feature data.

5. The method according to claim 4, characterized in that The step of processing the fused feature data through an emotion recognition network to determine target emotion data corresponding to the target speech data includes: Performing time dimension summation processing on the fused feature data through the emotion recognition network to obtain fused feature data in a preset dimensional format; Processing the fused feature data in the preset dimensional format through the emotion recognition network to obtain the confidence of each preset emotion category; The target emotion data is determined according to a preset emotion category whose confidence meets a preset condition.

6. The method according to claim 1, characterized in that The number of the fused feature data is at least two; and the processing of the fused feature data by the emotion recognition network to determine the target emotion data corresponding to the target speech data includes: Merging at least two fused feature data into concatenated feature data; The spliced ​​feature data is subjected to secondary fusion processing through the emotion recognition network to obtain secondary fusion feature data, and the secondary fusion feature data is processed to obtain the confidence of each preset emotion category; The target emotion data is determined according to the preset emotion category with the highest confidence.

7. A speech processing device, characterized in that: The device comprises: A splitting unit, used to determine a first sub-amplitude spectrum and a second sub-amplitude spectrum according to a target amplitude spectrum of the target speech data; a merging unit, configured to merge the first sub-amplitude spectrum and the second sub-amplitude spectrum into a fused amplitude spectrum; A feature unit, used for processing the fused amplitude spectrum through a first fusion network to determine fused feature data; The recognition unit is used to process the fused feature data through an emotion recognition network to determine the target emotion data corresponding to the target speech data.

8. An electronic device, characterized in that: include: A processor, a memory, and one or more programs; the one or more programs are stored in the memory and are configured to be executed by the processor, the programs comprising instructions for executing the steps in the method according to any one of claims 1 to 7.

9. A computer storage medium, characterized in that The computer storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that The computer program product comprises a non-transitory computer-readable storage medium storing a computer program, wherein the computer program is operable to cause a computer to execute the method according to any one of claims 1-7.

Images