Speech processing method and related device

Abstract

The embodiment of the invention provides a voice processing method and a related device, which are at least applied to the field of artificial intelligence, and the method comprises the steps: obtaining a time-frequency feature map of a to-be-processed voice signal; performing feature extraction on the time-frequency feature map under different feature scales to obtain time-frequency features under N feature scales; n is an integer greater than 1; based on the time-frequency feature map, performing time-frequency feature extraction on the time-frequency features under the N feature scales to obtain time-frequency features of the to-be-processed voice signal; and performing voice processing on the to-be-processed voice signal according to the time-frequency characteristics. According to the method, the time-frequency features of the to-be-processed voice signals are extracted and analyzed, the voice signals can be processed from multiple dimensions, and thus the voice processing precision is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence, and are related to but not limited to a speech processing method and related devices. Background Art

[0002] With the development of artificial intelligence technology, the processing of speech signals has become an important research topic. In related technologies, when processing speech signals, only the extraction and analysis of time domain features are considered, while the importance of frequency domain features is ignored. As a result, related technologies can only process speech signals from a single dimension, reducing the accuracy of speech signal processing. Summary of the invention

[0003] The embodiments of the present application provide a speech processing method and related devices, which can be applied at least in the field of artificial intelligence. By extracting and analyzing the time-frequency features of the speech signal to be processed, the speech signal can be processed from multiple dimensions to improve the accuracy of speech processing.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The embodiment of the present application provides a speech processing method, including: obtaining a time-frequency feature map of a speech signal to be processed; performing feature extraction at different feature scales on the time-frequency feature map to obtain time-frequency features at N feature scales; N is an integer greater than 1; based on the time-frequency feature map, performing time-frequency feature extraction on the time-frequency features at the N feature scales to obtain the time-frequency features of the speech signal to be processed; and performing speech processing on the speech signal to be processed according to the time-frequency features.

[0006] The embodiment of the present application provides a speech processing device, including: an acquisition module, used to acquire a time-frequency feature map of a speech signal to be processed; a feature extraction module, used to perform feature extraction at different feature scales on the time-frequency feature map to obtain time-frequency features at N feature scales; N is an integer greater than 1; the feature extraction module is also used to perform time-frequency feature extraction on the time-frequency features at the N feature scales based on the time-frequency feature map to obtain the time-frequency features of the speech signal to be processed; and a speech processing module is used to perform speech processing on the speech signal to be processed according to the time-frequency features.

[0007] An embodiment of the present application provides an electronic device, comprising: a memory for storing executable instructions; and a processor for implementing the above-mentioned voice processing method when executing the executable instructions stored in the memory.

[0008] An embodiment of the present application provides a computer program product, which includes executable instructions stored in a computer-readable storage medium; wherein a processor of an electronic device reads the executable instructions from the computer-readable storage medium and implements the above-mentioned speech processing method when executing the executable instructions.

[0009] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute the executable instructions to implement the above-mentioned speech processing method.

[0010] The above scheme has the following beneficial effects:

[0011] When performing speech processing, first, obtain the time-frequency feature graph of the speech signal to be processed; then, perform feature extraction on the time-frequency feature graph at different feature scales to obtain time-frequency feature graphs at N feature scales; N is an integer greater than 1; then, based on the time-frequency feature graph, perform time-frequency feature extraction on the time-frequency feature graph at N feature scales to obtain the time-frequency features of the speech signal to be processed; finally, perform speech processing on the speech signal to be processed according to the time-frequency features. It can be seen that the speech processing method of the embodiment of the present application realizes the processing of the speech signal by extracting and analyzing the time-frequency features of the speech signal to be processed. Since the time-frequency features are features extracted from the speech signal in the time domain and the frequency domain, the speech processing method of the embodiment of the present application can process the speech signal from multiple dimensions, thereby improving the accuracy of speech processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is an optional architectural diagram of a speech processing system provided in an embodiment of the present application;

[0013] Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0014] Figure 3 This is an optional flowchart of the speech processing method provided in the embodiment of the present application;

[0015] Figure 4 is another optional flow chart of the speech processing method provided in the embodiment of the present application;

[0016] Figure 5 It is a schematic diagram of a flow chart of determining the time-frequency characteristics of a speech signal to be processed provided by an embodiment of the present application;

[0017] Figure 6 is a schematic diagram of a process for determining a first time-frequency sub-feature provided in an embodiment of the present application;

[0018] Figure 7is a detailed architecture diagram of the speech processing method provided in the embodiment of the present application;

[0019] Figure 8 It is a simplified architecture diagram of the feature extraction module and the time-frequency feature extraction module provided in the embodiment of the present application;

[0020] Fig. 9 It is a detailed architecture diagram of the time-frequency feature extraction module provided in the embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0022] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meaning as those commonly understood by those skilled in the art of the technical field of the embodiments of the present application. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0023] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0024] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0025] Before describing the speech processing method provided in the embodiment of the present application, the professional terms involved in the embodiment of the present application are first described:

[0026] 1) Time-frequency features: refers to the features extracted and analyzed from speech signals in the time domain and frequency domain. The time-frequency features combine the time domain features and frequency domain features of speech signals, allowing computers to analyze and understand speech signals in both time and frequency dimensions.

[0027] 2) Time-frequency feature diagram: It is an image used to display the time-frequency characteristics of speech signals, usually used to show the changes in the frequency distribution of speech signals in different time periods.

[0028] 3) Time domain features: refers to the features obtained by extracting and analyzing speech signals in the time domain. Time domain features describe the changes and waveform shapes of speech signals on the time axis, and are usually used to represent the timing characteristics and dynamic behaviors of speech signals.

[0029] 4) Frequency domain features: refers to the features obtained by extracting and analyzing speech signals in the frequency domain. Frequency domain features describe the components and spectral characteristics of speech signals on the frequency axis, and are usually used to represent the frequency composition and spectral distribution of speech signals.

[0030] Before explaining the speech processing method of the embodiment of the present application, here, first explain the exemplary application of the speech processing device of the embodiment of the present application, which is an electronic device for implementing the speech processing method. In one implementation, the speech processing device (i.e., electronic device) provided in the embodiment of the present application can be implemented as a terminal or a server. In one implementation, the electronic device provided in the embodiment of the present application can be implemented as any terminal with speech processing function such as a laptop, a tablet computer, a desktop computer, an intelligent robot, an intelligent home appliance, and an intelligent vehicle-mounted device; in another implementation, the speech processing device provided in the embodiment of the present application can also be implemented as a server, wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN, Content Delivery Network), and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected by wired or wireless communication, which is not limited in the embodiment of the present application. Below, the exemplary application of the speech processing device when it is implemented as a server will be described.

[0031] See also Figure 1 , Figure 1It is an optional architecture diagram of the speech processing system provided by the embodiment of the present application. The speech processing system 10 of the embodiment of the present application at least includes a terminal 100, a network 200 and a server 300. A speech processing application is deployed on the terminal 100, and the server 300 can be a background server of the speech processing application. The server 300 can constitute the speech processing device of the embodiment of the present application, that is, the speech processing method of the embodiment of the present application is implemented through the server 300. The terminal 100 is connected to the server 300 through the network 200, and the network 200 can be a wide area network or a local area network, or a combination of the two.

[0032] See also Figure 1 , the terminal 100 receives the user's voice processing operation, and the terminal 100 generates a voice processing request in response to the voice processing operation; then, the terminal 100 sends the voice processing request to the server 300 through the network 200. After receiving the voice processing request, the server 300 responds to the voice processing request and obtains the time-frequency feature map of the voice signal to be processed; then, the server 300 extracts features at different feature scales on the time-frequency feature map to obtain time-frequency features at N feature scales; N is an integer greater than 1; then, based on the time-frequency feature map, the server 300 extracts time-frequency features at N feature scales to obtain time-frequency features of the voice signal to be processed; then, the server 300 performs voice processing on the voice signal to be processed according to the time-frequency features; finally, the server 300 sends the voice processing result to the terminal 100 through the network 200, and displays the voice processing result on the display interface of the terminal 100.

[0033] In some embodiments, the above-mentioned voice processing method can also be executed by the terminal, that is, after receiving the user's voice processing operation, the terminal 100 can respond to the voice processing operation and obtain the time-frequency feature map of the voice signal to be processed; then, the terminal 100 performs feature extraction at different feature scales on the time-frequency feature map to obtain time-frequency features at N feature scales; N is an integer greater than 1; then, based on the time-frequency feature map, the terminal 100 performs time-frequency feature extraction on the time-frequency features at N feature scales to obtain the time-frequency features of the voice signal to be processed; then, the terminal 100 performs voice processing on the voice signal to be processed according to the time-frequency features; finally, the terminal 100 displays the voice processing result on the display interface.

[0034] Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, Figure 2The electronic device shown may be a voice processing device, which includes: at least one processor 210, a memory 250, at least one network interface 220 and a user interface 230. The various components in the voice processing device are coupled together via a bus system 240. It is understood that the bus system 240 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 240 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 240 is not described in detail. Figure 2 Various buses are labeled as bus system 240 .

[0035] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0036] The user interface 230 includes one or more output devices 231 that enable presentation of media content, and one or more input devices 232 .

[0037] The memory 250 may be removable, non-removable or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drive, optical disk drive, etc. The memory 250 may optionally include one or more storage devices physically away from the processor 210. The memory 250 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 250 described in the embodiment of the present application is intended to include any suitable type of memory. In some embodiments, the memory 250 can store data to support various operations, and examples of these data include programs, modules, and data structures or subsets or supersets thereof, as exemplarily described below.

[0038] An operating system 251 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic businesses and processing hardware-based tasks; a network communication module 252 is used to reach other computing devices via one or more (wired or wireless) network interfaces 220. Exemplary network interfaces 220 include: Bluetooth, wireless compatibility certification (WiFi), and Universal Serial Bus (USB), etc.; an input processing module 253 is used to detect one or more user inputs or interactions from one of the one or more input devices 232 and translate the detected inputs or interactions.

[0039] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 A speech processing device 254 stored in the memory 250 is shown. The speech processing device 254 may be a speech processing device in an electronic device, which may be software in the form of a program or a plug-in, and includes the following software modules: an acquisition module 2541, a feature extraction module 2542, and a speech processing module 2543. These modules are logical, and therefore may be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0040] In some embodiments, the device provided in the embodiments of the present application can be implemented in hardware. As an example, the device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the speech processing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs) or other electronic components.

[0041] The speech processing methods provided in the embodiments of the present application can be executed by an electronic device, wherein the electronic device can be a server or a terminal, that is, the speech processing methods in the embodiments of the present application can be executed by a server, or by a terminal, or by interaction between a server and a terminal.

[0042] Figure 3This is an optional flow chart of the speech processing method provided in the embodiment of the present application. Figure 3 The steps shown are explained as Figure 3 As shown, the voice processing method is described by taking the server as an example, and the method includes the following steps S101 to S104:

[0043] Step S101, obtaining a time-frequency feature map of a speech signal to be processed.

[0044] Here, the speech signal to be processed refers to the original speech data that has not been processed or analyzed. These original speech data can be sound signals collected from a microphone or a recording device, and the speech signal to be processed usually contains various characteristics and information of the speech. The speech signal to be processed is usually represented in digital form, which can be continuous waveform data or a discrete sequence of sampling points; in addition, the speech signal to be processed can be a single speech sample or a continuous speech stream. The speech signal to be processed contains information about the speaker's speech content, speaking speed, intonation, pitch, volume, etc. The form and content of the speech signal to be processed are not limited in this application.

[0045] The time-frequency feature graph is a feature graph obtained after the speech signal to be processed is subjected to digital signal processing, wherein the digital signal processing method may include Mel-frequency cepstrum coefficients, linear predictive coding, short-time Fourier transform and power spectral density, etc. The specific digital signal processing method is not limited in this application.

[0046] It should be noted that the embodiments of the present application do not limit the digital signal processing method, and appropriate digital signal processing methods and time-frequency feature maps can be selected according to specific tasks and application scenarios, thereby improving the flexibility and generalization ability of the speech processing method. For example, for normal speech signals, Mel frequency cepstral coefficients can be used, and for singing speech signals, a constant Q transform method can be used.

[0047] Step S102, extracting features at different feature scales from the time-frequency feature graph to obtain time-frequency features at N feature scales; N is an integer greater than 1.

[0048] Here, the time-frequency features at N feature scales refer to a series of lower or higher resolution time-frequency features generated by extracting features from the time-frequency feature map through N feature extraction modules. Resolution refers to the number of pixels in the feature map, that is, the size of the feature map, and pixels represent the energy value of the corresponding position in the feature map; resolution can describe the detail and clarity of the feature map. The higher the resolution, the more pixels there are in the feature map, and the richer the details that can be represented. For example, a 50×50 pixel feature map has a higher resolution than a 25×25 pixel feature map because it contains more pixels and can provide more information.

[0049] In some embodiments, the server extracts features from the time-frequency feature graph through N feature extraction modules to obtain time-frequency features at N feature scales; wherein the network structures and network parameters of different feature extraction modules are different. For example: the first feature extraction module may include a fully connected layer, the second feature extraction module may include a fully connected layer and a convolutional layer (the convolution kernel size is 3×3, and the step size is 1), and the third feature extraction module may include a fully connected layer and a convolutional layer (the convolution kernel size is 1×1, and the step size is 2). It can be seen that the network structure of the first feature extraction module is different from that of the second feature extraction module and the third feature extraction module. Although the second feature extraction module and the third feature extraction module have the same network structure, the network parameters are different. The specific network structure and network parameters of the feature extraction module are not limited in this application.

[0050] It should be noted that, through step S102, the extraction of time-frequency features at multiple feature scales can be achieved, and the time-frequency features at different scales can complement and enhance each other, so as to better capture the time-frequency features of the speech signal to be processed. In addition, the applicability and generalization ability of the speech processing system can be improved by appropriately selecting and combining time-frequency features at different scales to adapt to different application scenarios and data features.

[0051] Step S103: based on the time-frequency feature graph, extract the time-frequency features at N feature scales to obtain the time-frequency features of the speech signal to be processed.

[0052] In some embodiments, the server may call a time-frequency feature extraction module to perform time-frequency feature extraction. The server may perform time-frequency feature extraction through the time-frequency feature extraction module through the following method: first, the server performs time-frequency feature extraction on the time-frequency feature graph and the time-frequency features at the first feature scale through the first time-frequency module in the time-frequency feature extraction module corresponding to the first feature scale to obtain a first time-frequency sub-feature; then, the server performs convolution processing on the first time-frequency sub-feature through the convolution layer in the time-frequency feature extraction module corresponding to the first feature scale to obtain the first time-frequency sub-feature after convolution processing; finally, the server performs average pooling processing on the first time-frequency sub-feature after convolution processing through the average pooling layer in the time-frequency feature extraction module corresponding to the first feature scale to obtain the first time-frequency feature; wherein, average pooling processing refers to the operation of reducing the size of the first time-frequency sub-feature after convolution processing and extracting important features.

[0053] Next, the server uses a time-frequency feature extraction module corresponding to the second feature scale to extract time-frequency features for the first time-frequency features and the time-frequency features at the second feature scale in the same way as the first time-frequency feature extraction method to obtain a second time-frequency feature; finally, the server uses a time-frequency feature extraction module corresponding to the third feature scale to extract time-frequency features for the second time-frequency features and the time-frequency features at the third feature scale in the same way as the first time-frequency feature extraction method to obtain a third time-frequency feature; and determines the third time-frequency feature as the time-frequency feature of the speech signal to be processed.

[0054] It should be noted that, as can be seen from step S103, the output of the first time-frequency feature extraction module is used as the input of the second feature extraction module, and the output of the second feature extraction module is used as the input of the third feature extraction module. In a serial manner, time-frequency feature reuse can be achieved, thereby improving the utilization and representation capability of time-frequency features.

[0055] Step S104: performing speech processing on the speech signal to be processed according to the time-frequency characteristics.

[0056] In some embodiments, after extracting the time-frequency features of the speech signal to be processed, the server first calls a dimension conversion function to perform dimension conversion on the time-frequency features; then, the time-frequency features after dimension conversion are reduced in dimension through the average pooling layer; then, the time-frequency features after dimension reduction are input into the fully connected layer, and the time-frequency features after dimension reduction are linearly combined through the linear function in the fully connected layer; finally, the time-frequency features after linear combination are normalized through the normalization function to complete the processing of the speech signal to be processed, wherein normalization refers to the operation of linearly transforming the feature data so that the numerical range of the feature data is between 0 and 1 or other specific ranges.

[0057] The speech processing method of the embodiment of the present application can select appropriate digital signal processing methods and time-frequency feature graphs according to specific tasks and application scenarios, which can improve the flexibility and generalization ability of the speech processing method; moreover, the embodiment of the present application extracts time-frequency features at multiple feature scales through multiple feature extraction modules, and the time-frequency features at different scales can complement and enhance each other, so as to better capture the time-frequency features of the speech signal to be processed; in addition, in the embodiment of the present application, the output of the first time-frequency feature extraction module is used as the input of the second feature extraction module, and the output of the second feature extraction module is used as the input of the third feature extraction module. In a serial manner, feature reuse can be achieved, thereby improving the utilization rate and representation ability of the time-frequency features; more importantly, the embodiment of the present application realizes the processing of the speech signal by extracting and analyzing the time-frequency features of the speech signal to be processed. Since the time-frequency features are features extracted from the speech signal in the time domain and the frequency domain, the speech processing method of the embodiment of the present application can process the speech signal from multiple dimensions, thereby improving the accuracy of speech processing.

[0058] The following is an example of an application scenario of the speech processing method provided in the embodiment of the present application. The embodiment of the present application can be applied to at least the following exemplary scenarios:

[0059] User A needs to perform speech recognition or speech synthesis on the collected speech data. In this way, before performing speech recognition or speech synthesis, the speech processing method provided in the embodiment of the present application can be used for speech processing. First, after the terminal receives the speech processing operation triggered by user A, it generates a speech processing request and sends the speech processing request to the server; then, the server responds to the speech processing request and obtains the time-frequency feature graph of the speech signal to be processed; then, the server extracts features at different feature scales from the time-frequency feature graph to obtain time-frequency features at N feature scales; N is an integer greater than 1; then, based on the time-frequency feature graph, the server extracts time-frequency features at N feature scales to obtain time-frequency features of the speech signal to be processed; finally, speech processing is performed on the speech signal to be processed according to the time-frequency features.

[0060] Based on the above scenario, the speech processing method of the embodiment of the present application is explained. Figure 4 is another optional flow chart of the speech processing method provided in the embodiment of the present application, such as Figure 4 As shown, the method includes the following steps S201 to S209:

[0061] Step S201: The terminal receives a voice processing operation.

[0062] Here, a voice processing application may be running on the terminal, and the server constitutes a background server of the voice processing application. The voice processing operation may be a selection operation or an input operation inputted through a client of the voice processing application running on the terminal, for example, the selection operation may select a storage path of a data set for voice processing, or the input operation may be a user inputting a storage path of a data set for voice processing on the client, or the user may directly input a data set for voice processing on the client.

[0063] In some embodiments, the speech processing application may provide an input interface or input box to allow the user to select or input the storage path of the data set used for speech processing. The input interface may be in the form of a form, a text box, or a drop-down menu, and the specific form is not limited in this application. The user may select the storage path of the data set used for speech processing from a predetermined option, or manually input the storage path of the data set used for speech processing.

[0064] Step S202: The terminal generates a voice processing request in response to the voice processing operation.

[0065] Here, the terminal may encapsulate the storage path of the data set for speech processing and the model parameters or model name of the speech processing model selected or input by the user into the speech processing request.

[0066] In some embodiments, in order to ensure that the storage path of the data set selected or input by the user for speech processing is accurately included in the request data, first, it is necessary to check the integrity of the storage path information. In the process of filling the storage path information into the speech processing request, ensure that the storage path information is complete and accurate. The method for checking the integrity of the storage path includes checking whether the storage path contains the file name, file extension, and necessary characters such as slash or backslash; then, select a suitable data structure (such as JSON) to ensure that the storage path information is correctly embedded in the data structure; finally, special characters are processed. For special characters such as slash, backslash, quotation marks, etc., the special characters can be converted into a specific encoding format for correct transmission and parsing.

[0067] Step S203: The terminal sends a voice processing request to the server.

[0068] In some embodiments, the terminal sends the encapsulated voice processing request to the server and requests the server to perform a voice processing operation. The voice processing request is usually sent using protocols such as HTTP or Web Socket.

[0069] Step S204: the server obtains a time-frequency feature graph of the speech signal to be processed in response to the speech processing request.

[0070] Here, after receiving the speech processing request, the server will parse the speech processing request. For example, for an HTTP request, the server can parse the request header and request body. The request header is parsed to obtain relevant information of the request; the request body is parsed to obtain the main body data of the request, that is, the storage path of the data set used for speech processing. Specific fields or parameters in the request body are parsed, and these fields or parameters contain the storage path of the data set used for speech processing. A specific data format, such as JSON or XML, is extracted from the request body, and then the data format is parsed to obtain the storage path of the data set used for speech processing.

[0071] In some embodiments, after obtaining the storage path of the data set used for speech processing and before obtaining the time-frequency feature map of the speech signal to be processed, the speech data needs to be digitally processed. The digital signal processing method may include Mel-frequency cepstral coefficients, linear prediction coding, short-time Fourier transform and power spectral density, etc. The specific method is not limited in this application. Taking the Mel-frequency cepstral coefficient as an example, first, the server will pre-process the voice signal, including pre-emphasis and framing. Pre-emphasis refers to emphasizing the high-frequency part of the voice signal through a high-pass filter to reduce the impact of the attenuation of the high-frequency components in the voice signal; framing refers to dividing the voice signal into short time periods according to a preset time length. The voice signal in each short time period is called a frame, and usually every 20-30 milliseconds is divided into a frame; then, Fourier transform is performed on each frame of the voice signal to convert the time domain signal into a frequency domain signal to generate a spectrum frame; then, each spectrum frame is convolved with the Mel filter group to calculate the energy of each filter channel; then, the logarithm of the energy of each filter group is taken to obtain the logarithmic energy spectrum, and the logarithmic energy spectrum is discrete cosine transformed to obtain the Mel-frequency cepstral coefficient; some Mel-frequency cepstral coefficients are retained as the time-frequency characteristics of the voice signal; finally, through visualization methods, the representation of the voice signal in the time-frequency domain can be obtained to generate a time-frequency feature map.

[0072] Step S205: The server extracts features at different feature scales on the time-frequency feature graph to obtain time-frequency features at N feature scales; N is an integer greater than 1.

[0073] Here, it is necessary to extract features from the time-frequency feature graph through multiple feature extraction modules with different network structures and network parameters in order to obtain time-frequency features at multiple feature scales.

[0074] In some embodiments, step S205 can be implemented by the following method: the server extracts features from the time-frequency feature graph through N feature extraction modules to obtain time-frequency features at N feature scales; wherein the network structures and network parameters of different feature extraction modules are different.

[0075] In some examples, there are three feature extraction modules with different network structures and network parameters. The server calls the first feature extraction module to perform feature extraction on the time-frequency feature graph to obtain the time-frequency features at a first feature scale, calls the second feature extraction module to perform feature extraction on the time-frequency feature graph to obtain the time-frequency features at a second feature scale, and calls the third feature extraction module to perform feature extraction on the time-frequency feature graph to obtain the time-frequency features at a third feature scale; wherein the first feature extraction module, the second feature extraction module and the third feature extraction module only represent different feature extraction modules, and do not represent the order of server calls.

[0076] Step S206: The server extracts time-frequency features of the time-frequency features at N feature scales based on the time-frequency feature graph to obtain time-frequency features of the speech signal to be processed.

[0077] Here, the server will use the time-frequency feature extraction module to extract the time-frequency features at each feature scale according to the time-frequency feature graph, so as to obtain the time-frequency features of the speech signal to be processed.

[0078] In some embodiments, see Figure 5 , Figure 5 It is shown that in step S206, the server extracts the time-frequency features at each feature scale based on the time-frequency feature graph to obtain the time-frequency features of the speech signal to be processed, which can be achieved by the following steps S2061 to S2064:

[0079] Step S2061: The server extracts time-frequency features from the time-frequency feature graph and the time-frequency features at the first feature scale through a time-frequency feature extraction module corresponding to the first feature scale to obtain a first time-frequency feature.

[0080] In some embodiments, the time-frequency feature extraction module corresponding to the first characteristic scale includes a first time-frequency module, a convolution layer and a pooling layer; step S2061 can be implemented by the following method: first, the server performs time-frequency feature extraction on the time-frequency feature graph and the time-frequency features at the first characteristic scale through the first time-frequency module to obtain the first time-frequency sub-feature; then, the server performs convolution processing on the first time-frequency sub-feature through the convolution layer to obtain the first time-frequency sub-feature after convolution processing; and performs average pooling processing on the first time-frequency sub-feature after convolution processing through the pooling layer to obtain the first time-frequency feature.

[0081] Here, the main function of the convolution layer is to extract local features by sliding a small window called a convolution kernel (or filter) on the input data; the main components of the convolution layer include the convolution kernel, stride, and padding. The convolution kernel is a small matrix that determines the size of the window sliding on the input data; stride refers to the length of the convolution kernel sliding on the input data. By adjusting the stride, the spatial size of the output feature map can be controlled. Generally, a larger stride will result in a smaller spatial size of the output feature map, while a smaller stride will result in an increase in the spatial size of the output feature map; padding refers to adding additional values ​​(usually zero) to the boundaries of the feature matrix to control the output size of the convolution operation. Average pooling refers to averaging the values ​​of each small area (usually a rectangular area) of the input feature map, and then using this average value as the value of the corresponding position in the output feature map. Average pooling can reduce the spatial size of the feature map, thereby reducing the number of parameters and the amount of calculation of the model, while improving the stability and generalization ability of the model.

[0082] In some embodiments, see Figure 6 , Figure 6 It is shown that the server extracts time-frequency features from the time-frequency feature graph and the time-frequency features at the first feature scale through the first time-frequency module in the first time-frequency feature extraction module to obtain the first time-frequency sub-feature, which can be implemented by the following steps S301 to S304:

[0083] Step S301: The server extracts time-domain features from the time-frequency feature graph and the time-frequency features at the first feature scale through the feature extraction submodule in the first time-frequency module to obtain time-domain features.

[0084] Here, the feature extraction submodule is a module for feature extraction composed of multiple convolutional layers with different parameters; the time domain feature refers to the feature obtained by analyzing and extracting the speech signal in the time domain.

[0085] In some embodiments, the feature extraction submodule includes a first feature extraction submodule and a second feature extraction submodule; step S301 can be implemented by the following method: first, the server inputs the time-frequency features at the first feature scale into the first feature extraction submodule in the first time-frequency module to obtain a first feature vector; then, the server inputs the time-frequency feature map into the second feature extraction submodule to obtain a second feature vector; finally, the server determines the time domain features based on the first feature vector and the second feature vector.

[0086] Here, the convolution layer parameters in the first feature extraction submodule and the second feature extraction submodule are different; wherein, the number of second feature extraction submodules is at least two, and the convolution layer parameters in any two feature extraction submodules are different; by inputting the time-frequency feature map into multiple second feature extraction submodules, multiple second feature vectors can be obtained, for example, by inputting the time-frequency feature map into 5 second feature extraction submodules, 5 different second feature vectors can be obtained.

[0087] In some embodiments, when the number of second feature extraction sub-modules is M and the number of second feature vectors is M, the server can determine the time domain features based on the first feature vector and the second feature vector by the following method: first, the server divides the M second feature vectors into a first key vector and a first numerical vector; then, the server operates the first feature vector and the first key vector to obtain a first operation result; and performs convolution processing on the first operation result to obtain a second operation result; then, the server normalizes the second operation result to obtain a first normalized result; finally, the server determines the first normalized result and the first preset operation result of the first numerical vector as the time domain feature.

[0088] Here, the key vector refers to a key vector with the same dimension as the input data, which is used to calculate the attention weight; the numerical vector refers to a vector with the same dimension as the input data, which is used to calculate the weighted sum; and the preset operation can be an inner product operation.

[0089] Step S302: the server performs a transposition operation on the time domain features to obtain transposed time domain features.

[0090] The purpose of transposing features is to change the arrangement order or dimension order within the features to meet the requirements of subsequent processing. The transposition operation of features is usually implemented through the transposition function. For example, the arrangement order of the input features is (B, T, D), where B represents the number of samples, T represents time, and D represents frequency; the calculation is performed in the order of (B, T, D), mainly to extract time domain features. After the transposition operation is performed through the transposition function, the arrangement order of the features is transformed into (B, D, T). The calculation is performed in the order after the transformation, mainly to extract frequency domain features.

[0091] Step S303: the server performs a transposition operation on the time-frequency features at the first feature scale to obtain the transposed time-frequency features at the first feature scale.

[0092] The transposition operation process has been explained in detail in step S302, and this application will not repeat it here. In addition, the execution order of step S302 and step S303 can be swapped, or they can be executed simultaneously.

[0093] Step S304: The server extracts features from the transposed time domain features and the transposed time-frequency features at the first feature scale through the feature processing submodule in the first time-frequency module to obtain first time-frequency sub-features.

[0094] The first time-frequency sub-feature is obtained by further extracting the transposed time domain feature and the transposed time-frequency feature at the first feature scale.

[0095] In some embodiments, when the feature processing submodule includes a first feature processing submodule and a second feature processing submodule, and the number of second feature processing submodules is K, step S304 can be implemented by the following method: first, the server inputs the transposed time-frequency features at the first feature scale into the first feature processing submodule in the first time-frequency module to obtain a third feature vector; then, the server inputs the transposed time domain features into the second feature processing submodule to obtain K fourth feature vectors; then, the server divides the K fourth feature vectors into a second key vector and a second numerical vector; then, the server normalizes the second preset operation results of the third feature vector and the second key vector to obtain a second normalized result; finally, the server determines the first time-frequency sub-feature based on the second normalized result.

[0096] Here, the convolution layer parameters in the first feature processing submodule and the second feature processing submodule are different; wherein the number K of the second feature processing submodules is at least 2, and the convolution layer parameters in any two second feature processing submodules are different.

[0097] In some embodiments, the server determines the first time-frequency sub-feature based on the second normalization result, which can be achieved by the following method: first, the server performs convolution processing on the second normalization result to obtain a third operation result; then, the server performs normalization processing on the third operation result to obtain a third normalized result; and the third normalization result and the third preset operation result of the second numerical vector are determined as frequency domain features; then, the server performs a transposition operation on the frequency domain features to obtain the transposed frequency domain features; finally, the server determines the first time-frequency sub-feature based on the transposed frequency domain features.

[0098] Here, the frequency domain features are transposed in order to keep them consistent with the input dimension of the module in subsequent processing. After obtaining the transposed frequency domain features, the server normalizes the transposed frequency domain features through the layer normalization function, and then inputs the normalized frequency domain features into the fully connected layer, learns the more complex features in the frequency domain features through the fully connected layer, and finally determines the frequency domain features processed by the fully connected layer as the first time-frequency sub-feature.

[0099] Step S2062: The server extracts time-frequency features from the i-1th time-frequency feature and the time-frequency feature at the i-th feature scale through the time-frequency feature extraction module corresponding to the i-th feature scale to obtain the i-th time-frequency feature; i is greater than 2 and less than or equal to N.

[0100] The process of time-frequency feature extraction has been described in detail in step S2061, and this application will not repeat it here.

[0101] Step S2063, when i is less than N, the server replaces i with i+1 and triggers the step of extracting time-frequency features from the i-1th time-frequency features and the time-frequency features at the i-th feature scale through the time-frequency feature extraction module corresponding to the i-th feature scale;

[0102] Here, when N=5 and i=3, i is less than N, and the server will continue to perform feature extraction operations, that is, use the time-frequency feature extraction module corresponding to the fourth feature scale to extract time-frequency features from the third time-frequency features and the time-frequency features under the fourth feature scale.

[0103] Step S2064: when i is equal to N, the server determines the i-th time-frequency feature as the time-frequency feature of the speech signal to be processed.

[0104] Here, when N=5, i=5, i is equal to N, and the server determines the fifth time-frequency feature as the time-frequency feature of the speech signal to be processed. It can be seen that when the server extracts the time-frequency feature, it uses the time-frequency feature extracted by the time-frequency feature extraction module of the previous layer, that is, the server extracts the input time-frequency feature in a serial manner, and uses the time-frequency feature extracted by the time-frequency feature extraction module corresponding to the last feature scale as the time-frequency feature of the speech signal to be processed.

[0105] Step 207: The server performs speech processing on the speech signal to be processed according to the time-frequency characteristics.

[0106] The specific process of performing speech processing on the speech signal to be processed according to the time-frequency characteristics has been described in detail in step S104, which will not be repeated here.

[0107] Step S208: The server sends the voice processing result to the terminal.

[0108] Step S209: the terminal displays the voice processing result on the current interface.

[0109] After obtaining the speech processing results, the next step can be performed based on the speech processing results, such as speech recognition, speech synthesis, etc.

[0110] The speech processing method provided in the embodiment of the present application can select appropriate digital signal processing methods and time-frequency feature graphs according to specific tasks and application scenarios, which can improve the flexibility and generalization ability of the speech processing method; moreover, the embodiment of the present application extracts time-frequency features at multiple feature scales through multiple feature extraction modules, and the time-frequency features at different scales can complement and enhance each other, thereby better capturing the time-frequency features of the speech signal to be processed; in addition, in the embodiment of the present application, the output of the first time-frequency feature extraction module is used as the input of the second feature extraction module, and the output of the second feature extraction module is used as the input of the third feature extraction module. In a serial manner, feature reuse can be achieved, thereby improving the utilization and representation ability of the time-frequency features; more importantly, the embodiment of the present application can extract and analyze the time-frequency features of the speech signal to be processed, combine the time domain features and frequency domain features of the speech signal to be processed, and process the speech signal from multiple dimensions, thereby improving the accuracy of speech processing.

[0111] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0112] The speech processing method of the embodiment of the present application can be applied to a time-frequency multi-feature discriminator based on an adversarial generative network to process the speech signal generated by the generator. Figure 7 , Figure 7 It is a detailed architecture diagram of the speech processing method provided in the embodiment of the present application. First, the speech signal generated by the generator is feature extracted through a digital signal processing method to obtain a time-frequency feature graph. It should be noted that the speech processing method in the embodiment of the present application can receive any time-frequency feature graph; then, the time-frequency feature graph is input into the speech processing model for processing. The speech processing model includes a feature extraction module and a time-frequency feature extraction module. Below, the feature extraction module and the time-frequency feature extraction module in the speech processing model are described in detail.

[0113] See also Figure 8 , Figure 8 This is a simplified architecture diagram of the feature extraction module and the time-frequency feature extraction module provided in the embodiment of the present application. First, the time-frequency feature map B' of the speech signal to be processed is input into the first feature extraction module including the fully connected layer to obtain the time-frequency feature A' at the first feature scale; then B' and A' are respectively input into the first time-frequency feature extraction module to obtain the first time-frequency sub-feature B" and the output result feature Figure 1 (Output result features Figure 1 and B” are the same feature, where B” is used as the input of the next time-frequency feature extraction module and the output result feature Figure 1is used to calculate the loss); then, in order to obtain the time-frequency features at different feature scales, B' is input into the second feature extraction module including the fully connected layer + convolution layer + average pooling layer to obtain the time-frequency feature A' at the second feature scale, and A' and B' are respectively input into the second time-frequency feature extraction module to obtain the second time-frequency sub-feature B'' and the output result feature Figure 2 ; Then, B' is input into the third feature extraction module including the fully connected layer + convolution layer + average pooling layer, wherein the network parameters in the third feature extraction module are different from those in the second feature extraction module, and the time-frequency feature A'' is obtained at the third feature scale. A''' and B''' are respectively input into the third time-frequency feature extraction module to obtain the third time-frequency sub-feature and the output result feature Figure 3 ; Then, the third time-frequency sub-feature is used as the time-frequency feature of the speech signal to be processed; finally, the obtained time-frequency feature is dimensionalized, and the time-frequency feature map after dimensionalization is input into the average pooling layer for downsampling, and processed through the fully connected layer and the normalization layer to obtain the speech processing result.

[0114] It should be noted that the time-frequency feature map B' is the original time-frequency feature map. B' and A' after feature extraction are input into the time-frequency feature extraction module. A' can be used to capture the spatial features of B', which can enhance the expressiveness of the time-frequency features. In addition, the expression dimensions of the time-frequency features under multiple feature scales are richer and can better capture the characteristics of the speech signal to be processed.

[0115] See also Fig. 9 , Fig. 9It is a detailed architecture diagram of the time-frequency feature extraction module provided in the embodiment of the present application. First, a feature extraction submodule M1 (one-dimensional convolution operation (Conv1d) + two-dimensional convolution operation (Conv2d) + one-dimensional convolution operation (Conv1d)) is designed. The function of the submodule M1 is to extract the abstract representation of the input time-frequency feature. First, a one-dimensional convolution operation (Conv1d) is used to extract the feature dimension abstract vector, and then a two-dimensional convolution operation (Conv2d) is used to extract the local time-frequency abstract representation. Finally, a one-dimensional convolution operation (Conv1d) is used to further extract the feature dimension abstract vector; here, M1 only means that the representation method of each submodule is the same, but the network parameters of each submodule are different; then, the time-frequency feature A' at the first feature scale is input into the first feature extraction submodule to obtain the first feature vector, and the time The frequency feature map B' is respectively input into the second feature extraction submodule and the third feature extraction submodule to obtain the second feature vector and the third feature vector; then, the first feature vector is multiplied by the second feature vector to obtain the first operation result, and the first operation result is convolved to obtain the second operation result; then, the second operation result is normalized by the first normalization function to obtain the first normalized result, and the product of the first normalized result and the third feature vector is determined as the time domain feature; then, the time domain feature is transposed, and A' is transposed at the same time; then, the above calculation process is continued for the transposed time domain feature and the transposed A' to obtain the frequency domain feature; finally, after a fully connected layer, the extraction of time-frequency features is completed.

[0116] In the embodiment of the present application, by extracting and analyzing the time-frequency features of the speech signal to be processed, the accuracy of speech processing can be improved, so that the time-frequency multi-feature discriminator can better guide the generation effect of the generator and improve the fidelity and fluency of the speech generated by the generator.

[0117] It is understandable that in the embodiments of the present application, if data related to user information or corporate information is involved, when the embodiments of the present application are applied to specific products or technologies, it is necessary to obtain user permission or consent, or to blur this information to eliminate the correspondence between this information and the user; and the relevant data collection and processing should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, and the informed consent or separate consent of the subject of personal information should be obtained, and subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the subject of personal information.

[0118] The following is a description of an exemplary structure of the speech processing device 254 provided in the present application as a software module. In some embodiments, for example, Figure 2As shown, the speech processing device 254 includes: an acquisition module 2541, which is used to acquire the time-frequency feature diagram of the speech signal to be processed; a feature extraction module 2542, which is used to perform feature extraction at different feature scales on the time-frequency feature diagram to obtain time-frequency features at multiple feature scales; the feature extraction module 2542 is also used to perform time-frequency feature extraction on the time-frequency features at each feature scale based on the time-frequency feature diagram to obtain the time-frequency features of the speech signal to be processed; and a speech processing module 2543, which is used to perform speech processing on the speech signal to be processed according to the time-frequency features.

[0119] In the above scheme, the feature extraction module is also used to: extract features from the time-frequency feature graph through N feature extraction modules to obtain time-frequency features at N feature scales; wherein the network structures and network parameters of different feature extraction modules are different.

[0120] In the above scheme, one feature scale corresponds to one time-frequency feature extraction module, and the feature extraction module is further used to: extract time-frequency features from the time-frequency feature graph and the time-frequency features at the first feature scale through the time-frequency feature extraction module corresponding to the first feature scale to obtain a first time-frequency feature; extract time-frequency features from the i-1th time-frequency features and the time-frequency features at the i-th feature scale through the time-frequency feature extraction module corresponding to the i-th feature scale to obtain the i-th time-frequency feature; i is greater than 2 and less than or equal to N; when i is less than N, replace i with i+1 and trigger the step of extracting time-frequency features from the i-1th time-frequency features and the time-frequency features at the i-th feature scale through the time-frequency feature extraction module corresponding to the i-th feature scale; when i is equal to N, determine the i-th time-frequency feature as the time-frequency feature of the speech signal to be processed.

[0121] In the above scheme, the time-frequency feature extraction module corresponding to the first characteristic scale includes a first time-frequency module, a convolution layer and a pooling layer; the feature extraction module is also used to: perform time-frequency feature extraction on the time-frequency feature map and the time-frequency features at the first characteristic scale through the first time-frequency module to obtain a first time-frequency sub-feature; perform convolution processing on the first time-frequency sub-feature through the convolution layer to obtain the first time-frequency sub-feature after convolution processing; and perform average pooling processing on the first time-frequency sub-feature after convolution processing through the pooling layer to obtain the first time-frequency feature.

[0122] In the above scheme, the feature extraction module is also used to: perform time domain feature extraction on the time-frequency feature graph and the time-frequency features at the first feature scale through the feature extraction submodule in the first time-frequency module to obtain time domain features; perform a transposition operation on the time domain features to obtain transposed time domain features; perform a transposition operation on the time-frequency features at the first feature scale to obtain the transposed time-frequency features at the first feature scale; perform feature extraction on the transposed time domain features and the time-frequency features at the first feature scale through the feature processing submodule in the first time-frequency module to obtain the first time-frequency sub-features.

[0123] In the above scheme, the feature extraction submodule includes a first feature extraction submodule and a second feature extraction submodule; the feature extraction module is also used to: input the time-frequency features under the first feature scale into the first feature extraction submodule in the first time-frequency module to obtain a first feature vector; input the time-frequency feature map into the second feature extraction submodule to obtain a second feature vector; determine the time domain features based on the first feature vector and the second feature vector.

[0124] In the above scheme, the number of the second feature extraction sub-modules is M, the number of the second feature vectors is M, and the feature extraction module is also used to: divide the M second feature vectors into a first key vector and a first numerical vector; perform operations on the first feature vector and the first key vector to obtain a first operation result; perform convolution processing on the first operation result to obtain a second operation result; perform normalization processing on the second operation result to obtain a first normalized result; and determine the first normalized result and the first preset operation result of the first numerical vector as the time domain feature.

[0125] In the above scheme, the feature processing submodule includes a first feature processing submodule and a second feature processing submodule, and the number of the second feature processing submodules is K; the device also includes a determination module, which is used to input the time-frequency features at the first feature scale after transposition into the first feature processing submodule in the first time-frequency module to obtain a third feature vector; input the transposed time domain features into the second feature processing submodule to obtain K fourth feature vectors; divide the K fourth feature vectors into a second key vector and a second numerical vector; normalize the second preset operation results of the third feature vector and the second key vector to obtain a second normalized result; and determine the first time-frequency sub-feature according to the second normalized result.

[0126] In the above scheme, the determination module is also used to: perform convolution processing on the second normalized result to obtain a third operation result; perform normalization processing on the third operation result to obtain a third normalized result; determine the third normalized result and the third preset operation result of the second numerical vector as frequency domain features; perform transposition operation on the frequency domain features to obtain transposed frequency domain features; and determine the first time-frequency sub-feature based on the transposed frequency domain features.

[0127] It should be noted that the description of the device of the embodiment of the present application is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment, so it is not repeated. For technical details not disclosed in the embodiment of the device, please refer to the description of the method embodiment of the present application for understanding.

[0128] An embodiment of the present application provides an electronic device, comprising: a memory for storing executable instructions; and a processor for implementing the above-mentioned voice processing method when executing the executable instructions stored in the memory.

[0129] An embodiment of the present application provides a computer program product, which includes executable instructions stored in a computer-readable storage medium; wherein a processor of an electronic device reads the executable instructions from the computer-readable storage medium and implements the above-mentioned speech processing method when executing the executable instructions.

[0130] The present application embodiment provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will be caused to execute the method provided by the present application embodiment, for example, Figure 3 The method shown.

[0131] In some embodiments, the storage medium can be a computer-readable storage medium, for example, a ferroelectric random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPR OM), a flash memory, a magnetic surface memory, an optical disk, or a compact disk read-only memory (CD-ROM), etc.; it can also be various devices including one or any combination of the above memories.

[0132] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0133] As an example, executable instructions may, but need not correspond to, a file in a file system, may be stored as part of a file storing other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions). As an example, executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed at multiple locations and interconnected by a communication network.

[0134] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A speech processing method, characterized in that: The method comprises: Obtaining a time-frequency feature map of a speech signal to be processed; Extracting features at different feature scales on the time-frequency feature graph to obtain time-frequency features at N feature scales, where N is an integer greater than 1; Based on the time-frequency feature graph, extracting the time-frequency features at the N feature scales to obtain the time-frequency features of the speech signal to be processed; Perform speech processing on the speech signal to be processed according to the time-frequency characteristics.

2. The method according to claim 1, characterized in that The extracting features at different feature scales from the time-frequency feature graph to obtain time-frequency features at N feature scales includes: The time-frequency feature graph is subjected to feature extraction through N feature extraction modules to obtain time-frequency features at N feature scales; wherein the network structures and network parameters of different feature extraction modules are different.

3. The method according to claim 1, characterized in that One feature scale corresponds to one time-frequency feature extraction module, and the time-frequency feature extraction is performed on the time-frequency features under the N feature scales based on the time-frequency feature graph to obtain the time-frequency features of the speech signal to be processed, including: Performing time-frequency feature extraction on the time-frequency feature graph and the time-frequency features at the first feature scale through a time-frequency feature extraction module corresponding to the first feature scale to obtain a first time-frequency feature; The time-frequency feature extraction module corresponding to the i-th feature scale performs time-frequency feature extraction on the i-1th time-frequency feature and the time-frequency feature under the i-th feature scale to obtain the i-th time-frequency feature; i is greater than 2 and less than or equal to N; When i is less than N, i is replaced by i+1 and the step of extracting time-frequency features from the i-1th time-frequency features and the time-frequency features at the i-th feature scale by the time-frequency feature extraction module corresponding to the i-th feature scale is triggered; When i is equal to N, the i-th time-frequency feature is determined as the time-frequency feature of the speech signal to be processed.

4. The method according to claim 3, characterized in that The time-frequency feature extraction module corresponding to the first characteristic scale includes a first time-frequency module, a convolution layer, and a pooling layer; the time-frequency feature extraction module corresponding to the first characteristic scale extracts time-frequency features from the time-frequency feature graph and the time-frequency features at the first characteristic scale to obtain the first time-frequency features, including: Performing time-frequency feature extraction on the time-frequency feature graph and the time-frequency feature at the first feature scale through the first time-frequency module to obtain a first time-frequency sub-feature; The first time-frequency sub-feature is convolved by the convolution layer to obtain the first time-frequency sub-feature after the convolution process; and the first time-frequency sub-feature after the convolution process is averaged pooled by the pooling layer to obtain the first time-frequency feature.

5. The method according to claim 4, characterized in that The extracting time-frequency features from the time-frequency feature graph and the time-frequency features at the first feature scale by using the first time-frequency module to obtain a first time-frequency sub-feature includes: Performing time domain feature extraction on the time-frequency feature graph and the time-frequency feature at the first feature scale through the feature extraction submodule in the first time-frequency module to obtain a time domain feature; Performing a transposition operation on the time domain features to obtain transposed time domain features; Performing a transposition operation on the time-frequency feature at the first characteristic scale to obtain the transposed time-frequency feature at the first characteristic scale; The feature processing submodule in the first time-frequency module performs feature extraction on the transposed time domain feature and the transposed time-frequency feature at the first feature scale to obtain a first time-frequency sub-feature.

6. The method according to claim 5, characterized in that The feature extraction submodule includes a first feature extraction submodule and a second feature extraction submodule; the feature extraction submodule in the first time-frequency module extracts time-domain features from the time-frequency feature graph and the time-frequency features at the first feature scale to obtain time-domain features, including: Inputting the time-frequency features at the first feature scale into a first feature extraction submodule in a first time-frequency module to obtain a first feature vector; Inputting the time-frequency feature map into a second feature extraction submodule to obtain a second feature vector; A time domain feature is determined based on the first feature vector and the second feature vector.

7. The method according to claim 6, characterized in that The number of the second feature extraction submodules is M, the number of the second feature vectors is M, and the determining of the time domain features based on the first feature vectors and the second feature vectors includes: Dividing the M second eigenvectors into a first key vector and a first numerical vector; Performing an operation on the first feature vector and the first key vector to obtain a first operation result; Performing convolution processing on the first operation result to obtain a second operation result; Normalizing the second operation result to obtain a first normalized result; A first preset operation result of the first normalization result and the first numerical vector is determined as the time domain feature.

8. The method according to claim 5, characterized in that The feature processing submodule includes a first feature processing submodule and a second feature processing submodule, and the number of the second feature processing submodules is K; the feature processing submodule in the first time-frequency module extracts the transposed time domain features and the transposed time-frequency features at the first feature scale to obtain the first time-frequency subfeature, including: Inputting the transposed time-frequency features at the first characteristic scale into the first characteristic processing submodule in the first time-frequency module to obtain a third characteristic vector; Inputting the transposed time domain features into the second feature processing submodule to obtain K fourth feature vectors; Dividing the K fourth eigenvectors into second key vectors and second numerical vectors; Normalizing a second preset operation result of the third eigenvector and the second key vector to obtain a second normalized result; A first time-frequency sub-feature is determined according to the second normalization result.

9. The method according to claim 8, characterized in that The determining the first time-frequency sub-feature according to the second normalization result includes: Performing convolution processing on the second normalization result to obtain a third operation result; Normalizing the third operation result to obtain a third normalized result; Determining a third preset operation result of the third normalization result and the second numerical vector as a frequency domain feature; Performing a transposition operation on the frequency domain features to obtain transposed frequency domain features; The first time-frequency sub-feature is determined based on the transposed frequency domain feature.

10. A speech processing device, characterized in that: The device comprises: An acquisition module, used to acquire a time-frequency feature map of a speech signal to be processed; A feature extraction module, used to extract features at different feature scales from the time-frequency feature graph to obtain time-frequency features at N feature scales; N is an integer greater than 1; The feature extraction module is further used to extract the time-frequency features under the N feature scales based on the time-frequency feature graph to obtain the time-frequency features of the speech signal to be processed; The speech processing module is used to perform speech processing on the speech signal to be processed according to the time-frequency characteristics.

11. An electronic device, characterized in that: include: A memory for storing executable instructions; The processor is used to implement the speech processing method according to any one of claims 1 to 9 when executing the executable instructions stored in the memory.

12. A computer-readable storage medium, characterized in that: Executable instructions are stored, which are used to cause a processor to execute the executable instructions to implement the speech processing method according to any one of claims 1 to 9.

13. A computer program product comprising computer executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the speech processing method according to any one of claims 1 to 9 is implemented.

Images