Voice data processing method, electronic equipment and storage medium

Abstract

The invention discloses a voice data processing method, electronic equipment and a storage medium. The method comprises the following steps: in response to a voice input instruction acting on an operation interface, displaying a voice file containing original voice data on the operation interface; a voice processing instruction acting on the operation interface is responded, a target processing result corresponding to the original voice data is displayed on the operation interface, and the target processing result comprises one of target text data, a voice file containing the target voice data and a voice file containing the target voice data. The target processing result is obtained by processing the original voice data based on the original discretization characteristics of the original voice data by using the voice processing model, and the original discretization characteristics are obtained by discretizing the original voice data. According to the method and the device, the technical problems of low voice data processing efficiency, low accuracy, weak generalization ability and poor robustness caused by the fact that the prior art still depends on a conventional model for single-person modeling to process the voice data are solved.

Description

Technical Field

[0001] The present application relates to the fields of speech processing technology and computer technology, and in particular to a speech data processing method, electronic device and storage medium. Background Art

[0002] At present, large language models pre-trained based on massive texts have made technological breakthroughs in the field of semantic understanding. However, in the speech field, such as speech recognition, speech synthesis, speech enhancement and other scenarios, they still rely on individual characters to use small models to complete scene tasks, which leads to low speech data processing efficiency, low accuracy, weak generalization ability and poor robustness in the speech field.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a method for processing speech data, an electronic device, and a storage medium to at least solve the technical problems of low speech data processing efficiency, low accuracy, weak generalization ability, and poor robustness caused by the fact that the prior art still relies on a conventional model for single-person modeling to process speech data.

[0005] According to one aspect of an embodiment of the present application, a method for processing voice data is provided, comprising: responding to a voice input instruction acting on an operation interface, displaying a voice file containing original voice data on the operation interface, wherein the original voice data is voice data collected by driving an audio collection device; responding to a voice processing instruction acting on the operation interface, displaying a target processing result corresponding to the original voice data on the operation interface, wherein the target processing result comprises one of the following: target text data, a voice file containing target voice data, the target processing result is obtained by processing the original voice data based on the original discretized features of the original voice data using a voice processing model, and the original discretized features are obtained by discretizing the original voice data.

[0006] According to another aspect of an embodiment of the present application, a training method for a speech processing model is also provided, comprising: obtaining first training data and second training data, wherein the first training data comprises: first speech data and training text data, and the second training data comprises at least: second speech data and a preset processing result; discretizing the first speech data to obtain a first discretized feature of the first speech data; performing unsupervised training on an initial processing model using the first discretized feature and the training text data to obtain a target processing model; fine-tuning the target processing model using the second training data to obtain a speech processing model, wherein the speech processing model is used to process the original speech data based on the original discretized feature of the original speech data to obtain a target processing result corresponding to the original speech data, the original discretized feature is obtained by discretizing the original speech data, the original speech data is speech data collected by driving an audio acquisition device, and the target processing result comprises one of the following: target text data or a speech file containing target speech data.

[0007] According to another aspect of an embodiment of the present application, a method for processing speech data is also provided, including: driving an audio acquisition device to acquire original speech data; discretizing the original speech data to obtain original discretized features of the original speech data; processing the original speech data based on the original discretized features using a speech processing model to obtain a target processing result corresponding to the original speech data, wherein the target processing result includes one of the following: target text data, target speech data.

[0008] According to another aspect of an embodiment of the present application, a method for processing voice data is also provided, including: obtaining original voice data by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter is the original voice data, and the original voice data is the voice data collected by driving an audio acquisition device; discretizing the original voice data to obtain original discretized features of the original voice data; using a voice processing model to process the original voice data based on the original discretized features to obtain a target processing result corresponding to the original voice data, wherein the target processing result includes one of the following: target text data, target voice data; outputting the target processing result by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the target processing result.

[0009] According to another aspect of an embodiment of the present application, there is further provided an electronic device, comprising: a memory storing an executable program; and a processor for running the program, wherein any one of the above methods is executed when the program is running.

[0010] According to another aspect of an embodiment of the present application, a computer-readable storage medium is further provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the above methods.

[0011] In an embodiment of the present application, based on voice processing technology, in the field of voice, by responding to voice input instructions acting on an operation interface, a voice file containing original voice data is displayed on the operation interface, wherein the original voice data is voice data collected by driving an audio acquisition device, and further by responding to voice processing instructions acting on the operation interface, a target processing result corresponding to the original voice data is displayed on the operation interface, wherein the target processing result includes one of the following: target text data, a voice file containing target voice data, and the target processing result is obtained by processing the original voice data based on the original discretized features of the original voice data using a voice processing model, and the original discretized features are obtained by discretizing the original voice data.

[0012] It is easy to notice that, since the above-mentioned original discretized features can characterize the features of the original speech data with discretization advantages such as lower complexity and higher robustness, the target processing results obtained by processing the original discretized features using a speech processing model can extend the advantages of the discretized feature representation to the speech data processing scenario. Therefore, the present application achieves the purpose of processing speech data using a speech processing model based on the discretized feature representation, thereby achieving the technical effect of improving the efficiency, accuracy, generalization ability and robustness of speech data processing, and further solves the technical problem of low efficiency, low accuracy, weak generalization ability and poor robustness of speech data processing caused by the fact that the prior art still relies on conventional models for single-person modeling to process speech data.

[0013] It is easy to notice that the above general description and the following detailed description are only for the purpose of exemplifying and explaining the present application, and do not constitute a limitation of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0015] Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing a method for processing voice data is shown;

[0016] Figure 2 is a schematic diagram of a hardware environment of a virtual reality device according to a method for processing voice data in an embodiment of the present application;

[0017] Figure 3 is a structural block diagram of a computing environment of a method for processing voice data according to an embodiment of the present application;

[0018] Figure 4 is a flowchart of a method for processing voice data according to Embodiment 1 of the present application;

[0019] Figure 5 is a schematic diagram of an optional speech processing model pre-training process according to Example 1 of the present application;

[0020] Figure 6 is a schematic diagram of another optional speech processing model pre-training process according to Example 1 of the present application;

[0021] Figure 7 is a schematic diagram of another optional speech processing model reasoning process according to Embodiment 1 of the present application;

[0022] Figure 8 is a flowchart of a method for training a speech processing model according to Embodiment 2 of the present application;

[0023] Fig. 9 is a flowchart of a method for processing voice data according to Embodiment 3 of the present application;

[0024] Fig.10 is a flowchart of a method for processing voice data according to Embodiment 4 of the present application;

[0025] Fig.11 is a flowchart of a method for processing voice data according to Embodiment 5 of the present application;

[0026] Fig.12 is a schematic diagram of processing voice data on a cloud server according to Embodiment 5 of the present application;

[0027] Fig.13 is a structural schematic diagram of a voice data processing device according to Embodiment 6 of the present application;

[0028] Fig.14 is a schematic structural diagram of an optional voice data processing device according to Embodiment 6 of the present application;

[0029] Fig.15 is a schematic structural diagram of another optional voice data processing device according to Embodiment 6 of the present application;

[0030] Fig.16 is a structural schematic diagram of a training device for a speech processing model according to Example 6 of the present application;

[0031] Fig.17 is a structural schematic diagram of another voice data processing device according to Embodiment 6 of the present application;

[0032] Fig.18 is a structural schematic diagram of another voice data processing device according to Embodiment 6 of the present application;

[0033] Fig.19 is a structural schematic diagram of another voice data processing device according to Embodiment 6 of the present application;

[0034] Fig. 20 It is a structural block diagram of a computer terminal according to Example 7 of the present application. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0038] Speech Tokenizer: refers to dividing a continuous speech signal into discrete speech segments or words. In speech recognition or speech processing tasks, speech signals are usually in the form of continuous waveforms, and for subsequent analysis and processing, they need to be divided into discrete speech units. The discretized form of speech is more suitable for tasks such as speech signal processing, feature extraction, and speech recognition.

[0039] Speech codec (Encodec): refers to the algorithm used to encode and compress and decode and decompress speech signals.

[0040] Speech processing model: In this solution, it may refer to the Speech-Large Language Model (Speech-LLM), which is a deep learning language model trained based on a large-scale speech dataset for processing speech data.

[0041] SpeechGPT: refers to an unsupervised pre-training method based on a generative model, which aims to model and generate speech signals. It is an extension of the GPT (Generative Pre-trained Transformer) model in the field of speech. It uses unlabeled speech data for pre-training, based on a generative model, to learn the characteristics and structure of speech signals. SpeechGPT provides an effective transfer learning method for speech-related tasks.

[0042] Instruction Fine-tuning: refers to a technique used to optimize deep learning models. In deep learning, model training usually involves a large number of computing operations, including matrix multiplication, convolution, etc. These computing operations may have different performance on different hardware. The goal of instruction set fine-tuning is to optimize the performance of the model by fine-tuning the instruction set of the computing operation. Specifically, instruction set fine-tuning improves the execution efficiency of computing operations by utilizing specific instructions of the hardware.

[0043] Example 1

[0044] According to an embodiment of the present application, an embodiment of a method for processing voice data is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0045] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a method for processing voice data. Figure 1As shown, the computer terminal 10 (or mobile device 10) may include one or more (shown as 102a, 102b, ..., 102n in the figure) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor (Microcontroller Unit, MCU) or a programmable logic device (Field Programmable Gate Array, FPGA)), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, the computer terminal 10 may also include: a display, an input / output interface (I / O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a cursor control device (such as a mouse, a touchpad, etc.), a keyboard, a power supply and / or a camera.

[0046] It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0047] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0048] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the voice data processing method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the voice data processing method described above is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0049] The transmission device 106 is used to connect to a network via a network interface to receive or send data. The specific examples of the above network may include a wired and / or wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0050] like Figure 1 The display shown may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0051] It should be noted that, in some optional embodiments, the above Figure 1 The computer device (or mobile device) shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the above-described computer device (or mobile device).

[0052] It should be noted that, in some embodiments, the above Figure 1 The computer device (or mobile device) shown has a touch display (also called a "touch screen" or "touch display screen"). In some embodiments, the above Figure 1 The computer device (or mobile device) shown has a graphical user interface (GUI), and a user can perform human-computer interaction with the GUI through finger contacts and / or gestures on the touch-sensitive surface. The human-computer interaction functions here optionally include the following interactions: creating web pages, drawing, word processing, making electronic documents, games, video conferencing, instant messaging, sending and receiving emails, call interface, playing digital videos, playing digital music and / or web browsing, etc. The executable instructions for performing the above-mentioned human-computer interaction functions are configured / stored in a computer program product executable by one or more processors or a readable storage medium.

[0053] The method embodiment provided in the first embodiment of the present application can be executed in a virtual reality (VR) device or an augmented reality (AR) device. Taking the VR device as an example, Figure 2Schematic diagram of the hardware environment of a virtual reality device according to a method for processing voice data in an embodiment of the present application. Figure 2 As shown, the virtual reality device 204 is connected to the terminal 206, and the terminal 206 is connected to the server 202 through a network. The above-mentioned virtual reality device 204 is not limited to: virtual reality helmets, virtual reality glasses, virtual reality all-in-one machines, etc., and the above-mentioned terminal 204 is not limited to PCs, mobile phones, tablet computers, etc. The server 202 can be a server corresponding to a media file operator, and the above-mentioned network includes but is not limited to: a wide area network, a metropolitan area network or a local area network.

[0054] Optionally, the virtual reality device 204 of the embodiment of the present application includes: a memory 2041, a processor 2042 and a transmission device 2043 ( Figure 2 2041 is used to store an application program, which can be used to execute the method for processing voice data provided in the embodiment of the present application. The processor 2042 can call and execute the application program stored in the memory 2041 to obtain the target processing result of the voice data through the method for processing voice data provided in the embodiment of the present application.

[0055] Optionally, the terminal 206 of the embodiment of the present application can also be used to execute the voice data processing method, and display the visualization results corresponding to the voice data processing results through the display screen of the VR device or AR device, or play the processing results obtained by executing the voice data processing method through an audio output device associated with the VR device or AR device.

[0056] Figure 2 The hardware structure block diagram shown can not only serve as an exemplary block diagram of the hardware environment of the above-mentioned AR / VR device (or mobile device), but also as an exemplary block diagram of the above-mentioned server.

[0057] Figure 3 is a structural block diagram of a computing environment of a method for processing voice data according to an embodiment of the present application, Figure 3 The block diagram shows the use of the above Figure 2 The VR device (or mobile device) shown is an embodiment of a computing node in the computing environment 301.

[0058] Still like Figure 3 As shown, the computing environment 301 includes multiple ( Figure 3310-1, 310-2, ..., are used to illustrate computing nodes (such as servers). Each computing node includes local processing and memory resources, and end users 302 can remotely run applications or store data in computing environment 301. Applications can be provided as multiple services 320-1 (representing service A), 320-2 (representing service D), 320-3 (representing service E) and 320-4 (representing service H) in computing environment 301.

[0059] Still like Figure 3 As shown, the end user 302 can provide and access services through a web browser or other software application on the client, and in some embodiments, the provision and / or request of the end user 302 can be provided to the entry gateway 330. The entry gateway 330 may include a corresponding agent to handle the provision and / or request for the service 320 (one or more services provided in the computing environment 301).

[0060] Still like Figure 3 As shown, service 320 is provided or deployed according to various virtualization technologies supported by computing environment 301. In some embodiments, service 320 can be provided according to virtualization based on virtual machine (VM), virtualization based on container and / or similar methods. Virtualization based on virtual machine can be to simulate a real computer by initializing a virtual machine, and execute programs and applications without directly contacting any actual hardware resources. While the virtual machine virtualizes the machine, according to container-based virtualization, the container can be started to virtualize the entire operating system (OS) so that multiple workloads can run on a single operating system instance.

[0061] In an embodiment based on container virtualization, several containers of service 320 can be assembled into a POD (e.g., Kubernetes POD). Figure 3 As shown, service 320-2 can be equipped with one or more PODs 340-1, 340-2, ..., 340-N (collectively referred to as POD 340). Each POD 340 can include a proxy 345 and one or more containers 342-1, 342-2, ..., 342-M (collectively referred to as containers 342). One or more containers 342 in POD 340 process requests related to one or more corresponding functions of the service, and the proxy 345 generally controls network functions related to the service, such as routing, load balancing, etc. Other services 320 can also be PODs similar to POD 340.

[0062] During operation, executing a user request from an end user 302 may require invoking one or more services 320 in the computing environment 301, and executing one or more functions of a service 320 may require invoking one or more functions of another service 320. Figure 3 As shown, service 320-1 (representing service A) receives a user request from terminal user 302 from the entry gateway 330, service 320-1 (representing service A) can call service 320-2 (representing service D), and service 320-2 (representing service D) can request service 320-3 (representing service E) to perform one or more functions.

[0063] The computing environment described above can be a cloud computing environment, where the allocation of resources is managed by the cloud service provider, allowing the development of functions without considering the implementation, adjustment or expansion of servers. The computing environment allows developers to execute code in response to events without building or maintaining complex infrastructure. Services can be divided into a set of functions that can be automatically and independently scaled, rather than expanding a single hardware device to handle potential loads.

[0064] Under the above operating environment, this application provides Figure 4 The method for processing voice data shown. Figure 4 is a flowchart of a method for processing voice data according to Embodiment 1 of the present application, such as Figure 4 As shown, the method for processing the voice data includes:

[0065] Step S41, in response to a voice input instruction on the operation interface, a voice file containing original voice data is displayed on the operation interface, wherein the original voice data is the voice data collected by driving the audio collection device.

[0066] The above-mentioned operation interface may be a graphical user interface provided by the client. A file identifier is displayed on the graphical user interface, and the file identifier is used to display a voice file containing original voice data. The above-mentioned audio acquisition device may be an input device associated with a terminal device corresponding to the client, such as a microphone, a recorder, a sound card, an audio interface, etc. The above-mentioned voice data may be expressed in natural language. The above-mentioned voice input instruction may be triggered to be generated and executed by a first touch operation acting on the operation interface.

[0067] Step S42, in response to the voice processing instruction acting on the operation interface, the target processing result corresponding to the original voice data is displayed on the operation interface, wherein the target processing result includes one of the following: target text data, a voice file containing target voice data, the target processing result is obtained by processing the original voice data based on the original discretized features of the original voice data using a voice processing model, and the original discretized features are obtained by discretizing the original voice data.

[0068] Optionally, the above speech processing model is a large language model.

[0069] The above-mentioned voice processing instruction can be triggered to be generated and executed by a second touch operation on the operation interface. Displaying the target processing result corresponding to the original voice data on the operation interface includes one of the following: displaying the target text data on the operation interface, displaying the voice file identifier corresponding to the voice file on the operation interface, and controlling the audio output device corresponding to the operation interface to play the voice file containing the target voice data.

[0070] According to the above method provided by the embodiment of the present application, in the process of speech data processing, the advantages of using the original discretized features are: simplifying the complexity of the model, using discretized features can significantly reduce the complexity of the feature space, and reduce the computational complexity and storage requirements of the model; enhancing the robustness of the model, the influence of outliers or noise points in the speech data on the discretized features will be weakened, and the use of discretized features can reduce the sensitivity of the model to outliers and noise; improving the interpretability of the model, and the discretized features are easier to understand and explain; using discretized features can capture the nonlinear relationship between the original continuous features and the target variables, so that the model can better learn these nonlinear relationships; improving the generalization ability of the model, the discretized features can reduce the overfitting of the model to specific samples, and the use of discretized features can make the model better generalize to unseen samples.

[0071] Since the large language model has stronger expressive power, better generalization ability, higher accuracy, better transfer learning ability and stronger robustness than the conventional small model, in an application scenario of the above method provided in the embodiment of the present application, the large language model is used as the speech processing model, and the original speech data is processed based on the original discretized features of the original speech data to obtain the target processing result, which can extend the advantages of the large language model to the speech data processing scenario.

[0072] According to the above method provided in the embodiment of the present application, it is possible to integrate various functional requirements (for example, recognition, synthesis, translation, enhancement, monitoring, etc.) involved in speech processing into the speech processing model. Therefore, the above method can be applied to, but not limited to, scenes such as character speech recognition, speech synthesis, speech translation, speech enhancement, audio event monitoring, and audio emotion recognition. The implementation of the above method in the above scenario includes but is not limited to application programs (such as translation software, intelligent generation software, etc.), functional plug-ins, intelligent hardware (such as smart speakers, smart wearable devices, etc.), etc. Furthermore, the above speech processing model provided in the embodiment of the present application can also be combined with a semantic big model to expand the above method to a wider range of application scenarios. For example, the above method is used to provide speech data processing for enterprise big models, education big models, medical big models, etc.

[0073] Specifically, in the application scenario of the intelligent voice assistant, a graphical user interface is displayed on the terminal device running the client corresponding to the intelligent voice assistant, and the graphical user interface serves as an operation interface that can respond to touch operations. When a first touch operation acting on the graphical user interface is detected, a voice input instruction is generated and executed to display a voice file identifier on the graphical user interface. For example, the first touch operation can be: clicking the "speak" button, pressing and holding the "speak" button, long pressing the specified area, etc. The voice file identifier is an identifier corresponding to the original voice data input by the user. For example, the original voice data is 10s of voice data collected by the microphone of the terminal device after the user presses and holds the "speak" button, and the voice file identifier is an identifier with a data number, duration, and voice flag.

[0074] Further, in the application scenario of the intelligent voice assistant, when a second touch operation acting on the graphical user interface is detected, the voice data processing process is called, and the obtained target processing result is displayed at a specified position on the graphical user interface, or the target processing result is output to the user through an audio output device. For example, the second touch operation can be: clicking the "OK" button, long pressing the specified area, re-pressing the specified area, etc. The voice data processing process specifically includes: using the pre-trained voice assistant large model (i.e., the voice processing model) to process the original voice data based on the original discretization features of the original voice data. For example, in daily application scenarios, in the graphical user interface corresponding to the intelligent voice assistant, after a voice is input, the intelligent voice assistant will return a text and / or a voice. For example, through the voice input question "How do you say apple in English", the intelligent voice assistant first calls the voice data processing process to discretize the problem and uses the voice assistant large model to process the problem based on the discretization features of the problem. The processing results obtained include the text "Apple" and the audio "Apple". On this basis, the text message "Apple" is displayed on the graphical user interface of the intelligent voice assistant, and the speaker of the terminal device plays the audio corresponding to "Apple".

[0075] Similarly, in the application scenario of smart speakers, smart speakers may provide an operation interface (similar to smart voice assistants, which will not be described in detail), or may not provide an operation interface (only through button or voice wake-up or triggering the generation of instructions). When the smart speaker detects the wake-up word, it triggers the generation and execution of voice input instructions, and collects original voice data through the audio acquisition device of the smart speaker. For example, the wake-up word is "Hello". When the smart speaker detects that the user says "Hello", a recording event is triggered, that is, the next voice input by the user is obtained until the first preset time ends or no voice input is detected within the second preset time. Furthermore, the file identifier corresponding to the obtained original voice file is displayed on the operation interface (if it is only provided by the speaker).

[0076] Furthermore, in the application scenario of smart speakers, the voice processing instruction is set to be triggered and executed immediately after the original voice data collection is completed. After the smart speaker collects the question input by the user, it calls the voice data processing process and outputs the target processing result to the user in the form of audio. The voice data processing process is similar to that of the smart voice assistant.

[0077] It should be noted that in the application scenario of the smart speaker, the smart speaker can also be associated with a mobile terminal device or a smart wearable device, and the mobile terminal device or the smart wearable device provides an operation interface corresponding to the smart speaker, so as to realize the above-mentioned response function, display function, etc. through the operation interface. In an embodiment of the present application, based on voice processing technology, in the field of voice, by responding to the voice input instruction acting on the operation interface, a voice file containing original voice data is displayed on the operation interface, wherein the original voice data is the voice data collected by driving the audio acquisition device, and further by responding to the voice processing instruction acting on the operation interface, the target processing result corresponding to the original voice data is displayed on the operation interface, wherein the target processing result includes one of the following: target text data, a voice file containing target voice data, and the target processing result is obtained by processing the original voice data based on the original discretization features of the original voice data using the voice processing model, and the original discretization features are obtained by discretizing the original voice data.

[0078] It is easy to notice that, since the above-mentioned original discretized features can characterize the features of the original speech data with discretization advantages such as lower complexity and higher robustness, the target processing results obtained by processing the original discretized features using a speech processing model can extend the advantages of the discretized feature representation to the speech data processing scenario. Therefore, the present application achieves the purpose of processing speech data using a speech processing model based on the discretized feature representation, thereby achieving the technical effect of improving the efficiency, accuracy, generalization ability and robustness of speech data processing, and further solves the technical problem of low efficiency, low accuracy, weak generalization ability and poor robustness of speech data processing caused by the fact that the prior art still relies on conventional models for single-person modeling to process speech data.

[0079] In a specific implementation of the embodiment of the present application, the structure of the speech processing model (Speech-LLM) constructed in the application scenario is shown as follows: Figure 5 As shown, the training process of the speech processing model can include three stages: the first stage, the audio discretization (SpeechTokenizer) pre-training stage; the second stage, the Speech-LLM unsupervised pre-training stage based on massive speech data and text data; the third stage, the instruction fine-tuning stage.

[0080] In an optional embodiment, discretization processing is performed on the original speech data, including the following method steps:

[0081] Step S4211, encoding the original voice data using a voice encoder to obtain original voice features corresponding to the original voice data;

[0082] Step S4210, using the target quantizer to discretize the original speech features to obtain original discretized features.

[0083] In the above optional embodiment, the above original speech data includes original speech and original text. The above speech encoder is an encoder pre-trained based on massive speech data and text data. The above target quantizer is a quantizer pre-trained based on massive speech data and text data.

[0084] In an optional embodiment, the above-mentioned method for processing voice data further includes the following method steps:

[0085] Step S502, obtaining training speech data;

[0086] Step S504, encoding the training speech data using the initial encoder to obtain training speech features corresponding to the training speech data;

[0087] Step S506, using an initial quantizer to discretize the training speech features to obtain training discretized features;

[0088] Step S508, using the initial decoder to decode the training discretized features to obtain generated speech data;

[0089] Step S510: Based on the training speech data and the generated speech data, the initial encoder, the initial quantizer and the initial decoder are trained to obtain a speech encoder, a target quantizer and a speech decoder.

[0090] In a specific application scenario, the training speech data includes training speech and training text. In another specific application scenario, the training speech data includes training speech, and the training speech features corresponding to the training speech serve as training text.

[0091] like Figure 5 As shown, in the first stage of the training process of the speech processing model: the initial encoder (Encodec) is used to encode the training speech, and the obtained training speech features are used as training text; a discrete word segmenter (also known as an initial quantizer, GPT2Tokenizer can be used in the scenario) is used to discretize the training text to obtain training discretized features, and the training discretized features are used as input for the second stage.

[0092] like Figure 5 As shown, in the second stage of the training process of the speech processing model: Speech-LLM unsupervised training is performed based on the training discretization features to obtain the unsupervised training results, and the unsupervised training results are used as input for the third stage.

[0093] like Figure 5 As shown, in the third stage of the training process of the speech processing model: the initial decoder (Decodec) is used to decode the unsupervised training results to obtain the output speech (ie, speech data); the discrete word segmenter is used to de-discretize the unsupervised training results to obtain the output text.

[0094] Furthermore, based on the training speech, the output speech, the training text and the output text, the initial encoder, the initial quantizer and the initial decoder are trained to obtain a speech encoder, a target quantizer and a speech decoder.

[0095] In an optional embodiment, in step S510, the training speech data includes first speech data, and based on the first speech data and the generated speech data, an initial encoder, an initial quantizer, and an initial decoder are trained to obtain a speech encoder, a target quantizer, and a speech decoder, including the following method steps:

[0096] Step S511, using a discriminator to discriminate the first voice data and the generated voice data to obtain a discrimination result;

[0097] Step S512: adjusting parameters of the initial encoder, the initial quantizer, and the initial decoder based on the discrimination result to obtain a speech encoder, a target quantizer, and a speech decoder.

[0098] In a specific implementation, the specific process of pre-training the speech processing model is as follows: Figure 6 As shown, based on the training speech (i.e., the first speech data) and the output speech (i.e., the generated speech data), a discriminator is used to perform discrimination to obtain a discrimination result, and the discrimination result is used to adjust the parameters of the initial encoder, the initial quantizer, and the initial decoder to obtain a speech encoder, a target quantizer, and a speech decoder. Specifically, the above discriminator can be used for loss calculation, and the discrimination result includes the training loss.

[0099] Still like Figure 6 As shown, in the method provided by the present application, the audio codec model based on neural network and vector quantization (VectorQuantization) is used to discretize the speech data. The audio codec model includes four components: encoder (Encoder), residual vector quantizer (Residual Vector Quantizer, RVQ), decoder (Decoder) and discriminator (Discriminator). The encoder is used to encode the time domain speech waveform to obtain the hidden feature representation. RVQ (such as Figure 6 The components shown in the oblique line shadow are used to quantize the hidden layer feature representation layer by layer to obtain the layer by layer quantization result (such as Figure 6 In the process of layer-by-layer quantization, the next layer quantizes the residual information of the previous layer. The decoder is used to decode the hidden layer feature representation based on the layer-by-layer quantization results to obtain the speech waveform of the output speech. The discriminator and the encoder form a generative adversarial structure, and the parameters of the audio codec model are adjusted through adversarial training to improve the model performance.

[0100] In addition, if Figure 6 As shown, a denoising switch can also be set in the audio codec model to control the denoising process during the coding and decoding process.

[0101] Furthermore, after Figure 6 After the parameter adjustment process shown in the figure, the schematic diagram of the inference stage is as follows Figure 7 As shown, the encoder in the inference stage is a speech encoder obtained by adjusting the parameters of the initial encoder based on the discrimination result, and the target quantizer in the inference stage (such as Figure 7The components shown in the oblique line shadow are the target quantizers obtained by adjusting the parameters of the initial quantizer based on the discrimination results, and the decoder in the inference stage is the speech decoder obtained by adjusting the parameters of the initial decoder based on the discrimination results. The encoder and the target quantizer can be deployed at the transmitting end, and the decoder can be deployed at the receiving end. The interaction between the transmitting end and the receiving end is the quantization result obtained by the target quantizer.

[0102] It should be noted that in the above specific implementation of the present application, the bit rate required by the vector quantization module is reduced by compressing the frame rate and adopting different input features. For example, a speech with a duration of 1 second will be compressed into 100 discrete representations (i.e., 100 tokens / s). Compared with the discretization processing method based on Encodec (400 tokens / s) in the conventional method, the above specific implementation of the present application has fewer discrete representations, which is conducive to reducing the modeling complexity of Speech-LLM.

[0103] In addition, when the training speech data includes training speech and training text, in the second stage of the training process of the speech processing model: the speech encoder trained in the first stage is used to encode the training speech. For example, a speech A with a duration of 5 seconds is encoded to obtain 500 discrete token representations C; the target quantizer (such as GPT2Tokenizer) trained in the first stage is used to quantize the training text W to obtain a modeling unit sequence T.

[0104] Specifically, the discrete token representation C is expressed as: C = Encodec(A) = [c1, c2, ..., c 500 ].

[0105] Specifically, the modeling unit sequence T is expressed as: T = GPT2Tokenizer (W) = [w1, w2, ..., w n ].

[0106] For example, the training text W is "CompenyA Group's mission is to let the world have no difficult business." Correspondingly, the modeling unit sequence T is

[0107] Furthermore, the training samples S used to train the speech processing model can be represented as: S = {C, T}. That is, in the training stage of the speech processing model, the massive speech data is converted into discrete coding representations through an encoder, and these discrete coding representations are combined with massive text data, and then pre-trained through the language model pre-training criteria to obtain a speech processing model (LLM).

[0108] In an optional embodiment, when the target processing result includes: target text data, the original speech data is processed using a speech processing model based on the original discretized features of the original speech data, including the following method steps:

[0109] Step S4221, extracting based on the original discretized features using the speech processing model to obtain the original semantic features corresponding to the original speech data;

[0110] Step S4222, using a text decoder to decode the original semantic features to obtain target text data.

[0111] In the above optional embodiment, the above speech processing model may also be integrated with a semantic processing model. When the target processing result includes target text data, the semantic processing model is used to extract features based on the original discretized features to obtain original semantic features; the original semantic features are decoded using a text decoder to obtain target text data. The above text decoder may be pre-trained in a similar training method to the above speech decoder.

[0112] For example, in a speech recognition scenario, the above-mentioned original speech data is user input speech, and the user input speech is discretized to obtain discretized features corresponding to the user input speech. A speech recognition processing model (i.e., a speech processing model) is used to extract based on the discretized features to obtain audio semantic features corresponding to the user input speech. Further, a text decoder is used to decode the audio semantic features to obtain the recognized text corresponding to the user input speech (i.e., target text data). Similarly, in an optional embodiment, when the target processing result includes: a speech file containing target speech data, the original speech data is processed based on the original discretized features of the original speech data using a speech processing model, including the following method steps:

[0113] Step S4231, extracting based on the original discretized features using the speech processing model to obtain the original semantic features corresponding to the original speech data;

[0114] Step S4232, using a speech decoder to decode the original semantic features to obtain target speech data.

[0115] For another example, in a speech enhancement scenario, the above-mentioned original speech data is noisy speech data, and the noisy speech data is discretized to obtain the discretized features corresponding to the noisy speech data. The speech enhancement processing model (i.e., speech processing model) is used to extract based on the discretized features to obtain the audio semantic features corresponding to the noisy speech data. Further, the audio semantic features are decoded using a speech decoder to obtain the speech enhancement result corresponding to the noisy speech data (i.e., the target speech data, usually denoised speech data). Therefore, after the above-mentioned speech processing model provided in the embodiment of the present application is combined with the semantic processing model, the above-mentioned method can be extended to a wider range of application scenarios. For example, the above-mentioned method is used to provide speech data processing for enterprise processing models, education processing models, medical processing models, etc. In an optional embodiment, when the target processing result includes: in the case of a voice file containing target speech data, the original speech data is processed based on the original discretized features of the original speech data using the speech processing model, including the following method steps:

[0116] Step S4241, in response to a text input instruction acting on the operation interface, displaying original text data on the operation interface;

[0117] Step S4242, encoding the original text data using a text encoder to obtain original text features corresponding to the original text data;

[0118] Step S4243, using the speech processing model to extract based on the original discretized features and the original text features to obtain the target semantic features;

[0119] Step S4244, using a speech decoder to decode the target semantic features to obtain target speech data.

[0120] For another example, in a speech translation scenario, a user inputs a text to be translated in a source language by executing a text input command on an operation interface, and the text to be translated in the source language is displayed on the operation interface. The text to be translated in the source language is discretized to obtain discretized features. The text to be translated in the source language is encoded using a text encoder to obtain text features corresponding to the text to be translated in the source language. Further, a speech processing model is used to extract based on the discretized features and text features to obtain target semantic features, and the target semantic features correspond to the target language to be translated. Further, a speech decoder is used to decode the target semantic features to obtain target speech data in the target language (including text data in the target language and / or speech data in the target language).

[0121] In the above optional embodiment, the text input instruction is generated and executed by triggering a third touch operation on the operation interface. For example, the third touch operation is a touch typing operation, a long press voice-to-text operation, and the like.

[0122] When the speech processing model is integrated with the semantic processing model, the target semantic features can be obtained by combining the original text features and the original discretized features corresponding to the original speech data. Compared with the semantic features obtained based solely on text features or speech discretized features, the target semantic features can more accurately represent the semantics of the original input data. Furthermore, the target speech data obtained based on the target semantic features can also have higher accuracy, robustness and interpretability.

[0123] In an optional embodiment, the method for processing voice data further includes the following method steps:

[0124] Step S602, obtaining first training data and second training data, wherein the first training data includes: first voice data and training text data, and the second training data includes at least: second voice data and a preset processing result;

[0125] Step S604, performing discretization processing on the first speech data to obtain a first discretization feature of the first speech data;

[0126] Step S606, using the first discretized feature and the training text data, performing unsupervised training on the initial processing model to obtain a target processing model;

[0127] Step S608, fine-tuning the target processing model using the second training data to obtain a speech processing model.

[0128] In the above optional embodiment, the above target processing model is a processing model after pre-training. In order to adapt the target processing model to various downstream tasks related to audio, the first training data and the second training data are used to fine-tune the target processing model to obtain the final speech processing model. The above first training data and the second training data are obtained from task-specific data sets corresponding to various downstream tasks, and the sample data in the task-specific data sets are labeled sample data.

[0129] In a specific implementation, multiple audio-related downstream tasks include: Automatic Speech Recognition (ASR), Machine Translation (MT), Speech-to-Text Translation (S2TT), Personalized Text-to-Speech (PTTS), Speech Enhancement (SE), Audio Event Detection (AED), and Speech Emotion Recognition (SER). The input data types and output data types corresponding to the above multiple downstream tasks are shown in Table 1 below.

[0130] Table 1

[0131] Task Category Input data category Output data category Speech Recognition voice text Text Translation Source language text Target language text Speech to text translation Source language voice Target language text Personalized speech synthesis Text, Style, Voice Synthesized speech Speech Enhancement Noisy speech Denoising Speech Audio Event Detection voice text Audio emotion recognition voice Text {emotion label}

[0132] The above task-specific datasets can be pre-collected and created according to Table 1 above.

[0133] Taking the speech recognition task as an example, the input data of the speech processing model is the speech signal (Speech), and the model fine-tuning process specifically includes: using a trained and fixed-parameter speech encoder (Encodec) to encode the speech signal to obtain a discrete audio code sequence; inputting the discrete audio code sequence into the Speech-LLM of the second stage of pre-training, and the output data obtained is the predicted text annotation corresponding to the speech signal (that is, the preset processing result); based on the predicted text annotation, the cross-entropy criterion is used to optimize the Speech-LLM, and then the target processing model is obtained; the target processing model is fine-tuned by the instruction set to obtain the speech processing model, wherein the instructions corresponding to the instruction set fine-tuning are determined by the task-specific data set corresponding to the speech recognition task. Specifically, the designations corresponding to the above instruction set fine-tuning may include: Single Instruction-Multiple Data (SIMD) instructions, floating-point multiply-add instructions (Fused Multiply-Add, FMA) instructions, etc.

[0134] In an optional embodiment, in step S606, unsupervised training is performed on the initial processing model using the first discretized feature and the training text data to obtain a target processing model, including the following method steps:

[0135] Step S661, using the initial processing model to process the first discretized feature and the training text data to obtain predicted training data;

[0136] Step S662, constructing an objective function corresponding to the initial processing model based on the first training data and the predicted training data;

[0137] Step S663, adjusting the initial processing model based on the objective function to obtain a target processing model.

[0138] In an optional embodiment, in step S608, fine-tuning the target processing model using the second training data to obtain a speech processing model includes the following method steps:

[0139] Step S681, discretizing the second speech data to obtain a third discretized feature of the third speech data;

[0140] Step S682, using the target processing model to process the third speech data based on the third discretized feature to obtain a training processing result;

[0141] Step S683, based on the preset processing results and the training processing results, fine-tune the target processing model to obtain a speech processing model.

[0142] In summary, the embodiment of the present application proposes that the Speech-LLM framework uses SpeechTokenizer to discretize continuous audio signals, thereby using a GPT-based speech processing model for pre-training and data processing. Furthermore, the Speech-LLM obtained by pre-training based on massive voice data and text data will fine-tune instructions through tasks related to the audio modality, so that Speech-LLM can integrate audio modality-related tasks (e.g., recognition, synthesis, translation, enhancement, monitoring, etc.) into the speech processing model.

[0143] It is easy to notice that the above method provided in the embodiment of the present application not only optimizes the prior art in terms of SpeechTokenizer, but also performs LLM pre-training based on massive audio text data; the speech processing model in the above method is not only oriented to audio generation, but also unifies speech modality-related tasks under the Speech-LLM framework.

[0144] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0145] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0146] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), including several instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0147] Example 2

[0148] In the operating environment as in Example 1, the present application provides Figure 8 A training method for a speech processing model is shown. Figure 8 is a flow chart of a method for training a speech processing model according to Embodiment 2 of the present application, such as Figure 8 As shown, the training method of the above speech processing model includes:

[0149] Step S81, obtaining first training data and second training data, wherein the first training data includes: first voice data and training text data, and the second training data includes at least: second voice data and a preset processing result;

[0150] Step S82, discretizing the first speech data to obtain a first discretized feature of the first speech data;

[0151] Step S83, using the first discretized feature and the training text data, performing unsupervised training on the initial processing model to obtain a target processing model;

[0152] Step S84, using the second training data to fine-tune the target processing model to obtain a speech processing model, wherein the speech processing model is used to process the original speech data based on the original discretized features of the original speech data to obtain a target processing result corresponding to the original speech data, the original discretized features are obtained by discretizing the original speech data, the original speech data is the speech data collected by driving the audio acquisition device, and the target processing result includes one of the following: target text data or a speech file containing target speech data.

[0153] Optionally, the above-mentioned speech processing model is a large language model.

[0154] According to the training method of the speech processing model provided in the embodiment of the present application, various functional requirements (e.g., recognition, synthesis, translation, enhancement, monitoring, etc.) involved in speech processing can be integrated into the speech processing model through the training stage. Therefore, the training method of the speech processing model can be applied to, but not limited to, human speech recognition, speech synthesis, speech translation, speech enhancement, audio event monitoring, audio emotion recognition and other application scenarios. The implementation of the above method in the above scenario includes, but is not limited to, application programs, functional plug-ins, etc.

[0155] The training method of the speech processing model provided in the embodiment of the present application can be run on a client or a server. The server can be an independent server, a server cluster or a cloud server. When the server is a cloud server, real-time interaction with the client can be performed in the form of Software as a Service (SaaS), the cloud server obtains the first training data and the second training data provided by the client, obtains the speech processing model using the training method of the speech processing model, and returns the speech processing model to the client.

[0156] According to the training method of the above-mentioned speech processing model provided in the embodiment of the present application, the advantages of using the original discretized features in the training process are: simplifying the model complexity, the use of discretized features can significantly reduce the complexity of the feature space, and reduce the computational complexity and storage requirements of the model; enhancing the robustness of the model, the influence of outliers or noise points in the speech data on the discretized features will be weakened, and the use of discretized features can reduce the sensitivity of the model to outliers and noise; improving the interpretability of the model, the discretized features are easier to understand and explain; the use of discretized features can capture the nonlinear relationship between the original continuous features and the target variables, so that the model can better learn these nonlinear relationships; improving the generalization ability of the model, the discretized features can reduce the overfitting of the model to specific samples, and the use of discretized features can make the model better generalize to unseen samples.

[0157] Since the large model has stronger expressive power, better generalization ability, higher accuracy, better transfer learning ability and stronger robustness than the conventional small model, according to the training method of the above-mentioned speech processing model provided in the embodiment of the present application, in the process of speech data processing, the speech processing model is used to process the original speech data based on the original discretized features of the original speech data to obtain the target processing result, which can extend the advantages of the large model to the speech data processing scenario.

[0158] In an application scenario of an embodiment of the present application, through the training method of the above-mentioned speech processing model, a speech processing model for speech processing can be obtained based on the training method of a large language model. By using this speech processing model for speech processing, the technical effect of improving the efficiency, accuracy, generalization ability and robustness of speech data processing can be achieved, thereby solving the technical problem of low efficiency, low accuracy, weak generalization ability and poor robustness of speech data processing caused by the fact that the prior art still relies on conventional models for single-person modeling to process speech data.

[0159] In an optional embodiment, in step S82, discretization processing is performed on the first speech data to obtain a first discretization feature of the first speech data, including the following method steps:

[0160] Step S821, encoding the first voice data using a voice encoder to obtain a first voice feature corresponding to the first voice data;

[0161] Step S822: discretize the first speech feature using a target quantizer to obtain a first discretized feature.

[0162] In an optional embodiment, the training method of the speech processing model further includes the following method steps:

[0163] Step S851, obtaining training speech data;

[0164] Step S852, encoding the training speech data using the initial encoder to obtain training speech features corresponding to the training speech data;

[0165] Step S853, using an initial quantizer to discretize the training speech features to obtain training discretized features;

[0166] Step S854, using the initial decoder to decode the training discretized features to obtain generated speech data;

[0167] Step S855, based on the training speech data and the generated speech data, the initial encoder, the initial quantizer and the initial decoder are trained to obtain a speech encoder, a target quantizer and a speech decoder.

[0168] In summary, in the training method of the speech processing model proposed in the embodiment of the present application, the continuous audio signal can be discretized using SpeechTokenizer under the Speech-LLM framework, so as to use the GPT-based speech processing model for pre-training. In addition, the training method of the above-mentioned speech processing model provided in the embodiment of the present application not only optimizes the SpeechTokenizer of the model training in the prior art, but also performs LLM pre-training based on massive audio text data; the speech processing model trained by the above method is not only oriented to audio generation, but also unifies the tasks related to speech modality under the Speech-LLM framework.

[0169] It should be noted that the preferred implementation of this embodiment can refer to the relevant description in Example 1, which will not be repeated here.

[0170] Example 3

[0171] In the operating environment as in Example 1, the present application provides Fig. 9 A method for processing speech data is shown. Fig. 9 is a flowchart of a method for processing voice data according to Embodiment 3 of the present application. Fig. 9 As shown, the method for processing the above-mentioned voice data includes:

[0172] Step S91, driving the audio acquisition device to collect original voice data;

[0173] Step S92, discretizing the original speech data to obtain original discretized features of the original speech data;

[0174] Step S93, using the speech processing model to process the original speech data based on the original discretized features to obtain a target processing result corresponding to the original speech data, wherein the target processing result includes one of the following: target text data, target speech data.

[0175] Optionally, the above-mentioned speech processing model is a large language model.

[0176] The above-mentioned voice data processing method provided in the embodiment of the present application can be applied to, but not limited to, application scenarios involving voice data processing in the fields of e-commerce, education, medical treatment, conferences, social networks, financial products, logistics, and navigation. The above-mentioned audio acquisition device can be a hardware device associated with a terminal device running the above-mentioned voice data processing method, such as a microphone or a recorder; it can also be an audio interface, application software, etc. that interacts with the running process of the above-mentioned voice data processing method.

[0177] According to the above method provided by the embodiment of the present application, in the process of speech data processing, the advantages of using the original discretized features are: simplifying the complexity of the model, using discretized features can significantly reduce the complexity of the feature space, and reduce the computational complexity and storage requirements of the model; enhancing the robustness of the model, the influence of outliers or noise points in the speech data on the discretized features will be weakened, and the use of discretized features can reduce the sensitivity of the model to outliers and noise; improving the interpretability of the model, and the discretized features are easier to understand and explain; using discretized features can capture the nonlinear relationship between the original continuous features and the target variables, so that the model can better learn these nonlinear relationships; improving the generalization ability of the model, the discretized features can reduce the overfitting of the model to specific samples, and the use of discretized features can make the model better generalize to unseen samples.

[0178] Since the processing model has stronger expressive power, better generalization ability, higher accuracy, better transfer learning ability and stronger robustness than the conventional small model, according to the above method provided in the embodiment of the present application, in the process of speech data processing, the speech processing model is used to process the original speech data based on the original discretized features of the original speech data to obtain the target processing result, which can extend the advantages of the processing model to the speech data processing scenario.

[0179] In the embodiments of the present application, the above-mentioned method for processing speech data achieves the technical effect of improving the efficiency, accuracy, generalization ability and robustness of speech data processing, thereby solving the technical problem of low efficiency, low accuracy, weak generalization ability and poor robustness of speech data processing caused by the fact that the prior art still relies on conventional models for single-person modeling to process speech data.

[0180] It should be noted that the preferred implementation of this embodiment can refer to the relevant descriptions in Embodiments 1 and 2, which will not be repeated here.

[0181] Example 4

[0182] In the operating environment as in Example 1, the present application provides Fig.10 Another method for processing speech data is shown. Fig.10 is a flowchart of a method for processing voice data according to Embodiment 4 of the present application. Fig.10 As shown, the method for processing the above-mentioned voice data includes:

[0183] Step S1001, obtaining original voice data by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter is the original voice data, and the original voice data is the voice data collected by driving the audio collection device;

[0184] Step S1002, discretizing the original speech data to obtain original discretized features of the original speech data;

[0185] Step S1003, using the speech processing model to process the original speech data based on the original discretized features to obtain a target processing result corresponding to the original speech data, wherein the target processing result includes one of the following: target text data, target speech data;

[0186] Step S1004: output the target processing result by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the target processing result.

[0187] Optionally, the above-mentioned speech processing model is a large language model.

[0188] In the above-mentioned method for processing voice data provided in an embodiment of the present application, the above-mentioned first interface is a preset voice acquisition interface or an audio acquisition interface. The above-mentioned second interface is a preset voice output interface or an audio output interface. The terminal device running the above-mentioned method for processing voice data interacts with another device or a client through the above-mentioned first interface and the second interface, and the terminal device provides a voice data processing service for another device or a client. For example, the above-mentioned method for processing voice data runs on a cloud server, and the above-mentioned first interface and the second interface are a service uplink network interface and a service downlink network interface of Software as a Service (SaaS) between the cloud server and the client.

[0189] According to the above method provided by the embodiment of the present application, in the process of speech data processing, the advantages of using the original discretized features are: simplifying the complexity of the model, using discretized features can significantly reduce the complexity of the feature space, and reduce the computational complexity and storage requirements of the model; enhancing the robustness of the model, the influence of outliers or noise points in the speech data on the discretized features will be weakened, and the use of discretized features can reduce the sensitivity of the model to outliers and noise; improving the interpretability of the model, and the discretized features are easier to understand and explain; using discretized features can capture the nonlinear relationship between the original continuous features and the target variables, so that the model can better learn these nonlinear relationships; improving the generalization ability of the model, the discretized features can reduce the overfitting of the model to specific samples, and the use of discretized features can make the model better generalize to unseen samples.

[0190] Since the large model has stronger expressive power, better generalization ability, higher accuracy, better transfer learning ability and stronger robustness than the conventional small model, according to the above method provided in the embodiment of the present application, in the process of speech data processing, the speech processing model is used to process the original speech data based on the original discretized features of the original speech data to obtain the target processing result, which can extend the advantages of the large model to the speech data processing scenario.

[0191] In the embodiments of the present application, the above-mentioned method for processing speech data achieves the technical effect of improving the efficiency, accuracy, generalization ability and robustness of speech data processing, thereby solving the technical problem of low efficiency, low accuracy, weak generalization ability and poor robustness of speech data processing caused by the fact that the prior art still relies on conventional models for single-person modeling to process speech data.

[0192] It should be noted that the preferred implementation of this embodiment can refer to the relevant descriptions in Embodiments 1, 2, and 3, which will not be repeated here.

[0193] Example 5

[0194] In the operating environment as in Example 1, the present application provides Fig.11 Another method for processing speech data is shown. Fig.11 is a flowchart of a method for processing voice data according to Embodiment 5 of the present application. Fig.11 As shown, the method for processing the above-mentioned voice data includes:

[0195] Step S1101, the cloud server receives original voice data sent by the client, wherein the original voice data is the voice data collected by driving the audio collection device;

[0196] Step S1102, the cloud server discretizes the original voice data to obtain original discretized features of the original voice data;

[0197] Step S1103, the cloud server processes the original voice data using the voice processing model based on the original discretized features to obtain a target processing result corresponding to the original voice data, wherein the target processing result includes one of the following: target text data, target voice data;

[0198] Step S1104: the cloud server sends the target processing result to the client, wherein the target processing result is output by the client.

[0199] Optionally, the above-mentioned speech processing model is a large language model.

[0200] Optionally, Fig.12is a schematic diagram of processing voice data on a cloud server according to Embodiment 5 of the present application, such as Fig.12 As shown, the client uploads the original voice data to the cloud server, wherein the original voice data is the voice data collected by driving the audio acquisition device; then, the cloud server discretizes the original voice data to obtain the original discrete features of the original voice data; further, the cloud server uses the voice processing model to process the original voice data based on the original discrete features to obtain the target processing result corresponding to the original voice data, wherein the target processing result includes one of the following: target text data, target voice data; the cloud server will feedback the target processing result to the above-mentioned client, and the final target processing result will be provided to the user through the client's graphical user interface or audio output device.

[0201] It should be noted that the above-mentioned voice data processing method provided in the embodiment of the present invention can be applicable to, but not limited to, actual application scenarios involving voice data processing (such as voice recognition, voice synthesis, voice translation, voice enhancement, voice event monitoring, etc.) in the fields of e-commerce, education, medical care, conferences, social networks, financial products, logistics and navigation. By means of interaction between the SaaS server and the client, the SaaS server discretizes the original voice data to obtain the original discretized features of the original voice data, and uses the voice processing model to process the original voice data based on the original discretized features to obtain the target processing result corresponding to the original voice data, and the returned target processing result is provided to the user through the client.

[0202] In an embodiment of the present application, the system composed of a client device and a server can perform the following steps: the client device executes to obtain a voice input instruction through an operation interface and sends the voice input instruction to the server, and receives the target processing result returned by the server and provides the target processing result to the user; the server executes to receive the voice input instruction sent by the client, obtains the target processing result based on the voice data instruction using the voice data processing method provided in the present application, and returns the voice processing result to the client. It should be noted that, if the operating resources of the client device can meet the deployment and operating conditions of the processing model, the embodiment of the present application can be performed in the client device.

[0203] In the embodiments of the present application, the above-mentioned method for processing speech data achieves the technical effect of improving the efficiency, accuracy, generalization ability and robustness of speech data processing, thereby solving the technical problem of low efficiency, low accuracy, weak generalization ability and poor robustness of speech data processing caused by the fact that the prior art still relies on conventional models for single-person modeling to process speech data.

[0204] It should be noted that the preferred implementation of this embodiment can refer to the relevant descriptions in Embodiments 1, 2, and 3, which will not be repeated here.

[0205] Example 6

[0206] According to an embodiment of the present application, a device embodiment for implementing the method for processing voice data in the above-mentioned embodiment 1 is also provided. Fig.13 is a structural diagram of a voice data processing device according to Embodiment 6 of the present application, such as Fig.13 As shown, the device comprises:

[0207] The first response module 1301 is used to respond to the voice input command acting on the operation interface and display the voice file containing the original voice data on the operation interface, wherein the original voice data is the voice data collected by driving the audio collection device;

[0208] The second response module 1302 is used to respond to the voice processing instruction acting on the operation interface, and display the target processing result corresponding to the original voice data on the operation interface, wherein the target processing result includes one of the following: target text data, a voice file containing target voice data, and the target processing result is obtained by processing the original voice data based on the original discretized features of the original voice data using the voice processing model, and the original discretized features are obtained by discretizing the original voice data.

[0209] Optionally, Fig.14 is a schematic diagram of the structure of an optional voice data processing device according to Embodiment 6 of the present application, such as Fig.14 As shown, the device includes Fig.13 In addition to all the modules shown, it also includes: a processing module 1303, which is used to discretize the original speech data; the processing module 1303 is also used to: encode the original speech data using a speech encoder to obtain the original speech features corresponding to the original speech data; and discretize the original speech features using a target quantizer to obtain the original discretized features.

[0210] Optionally, Fig.15 is a schematic diagram of the structure of another optional voice data processing device according to Embodiment 6 of the present application, such as Fig.15 As shown, the device includes Fig.14In addition to all the modules shown, it also includes: a training module 1304, which is used to obtain training speech data; use an initial encoder to encode the training speech data to obtain training speech features corresponding to the training speech data; use an initial quantizer to discretize the training speech features to obtain training discretized features; use an initial decoder to decode the training discretized features to obtain generated speech data; based on the training speech data and the generated speech data, the initial encoder, initial quantizer and initial decoder are trained to obtain a speech encoder, a target quantizer and a speech decoder.

[0211] Optionally, the training module 1304 is further used to: use a discriminator to discriminate the first speech data and the generated speech data to obtain a discrimination result; and adjust parameters of an initial encoder, an initial quantizer and an initial decoder based on the discrimination result to obtain a speech encoder, a target quantizer and a speech decoder.

[0212] Optionally, when the target processing result includes: target text data, the above-mentioned training module 1304 is also used to: use the speech processing model to extract based on the original discretized features to obtain the original semantic features corresponding to the original speech data; use the text decoder to decode the original semantic features to obtain the target text data.

[0213] Optionally, when the target processing result includes: a voice file containing target voice data, the above-mentioned training module 1304 is also used to: use the voice processing model to extract based on the original discretized features to obtain the original semantic features corresponding to the original voice data; use the voice decoder to decode the original semantic features to obtain the target voice data.

[0214] Optionally, when the target processing result includes: a voice file containing target voice data, the above-mentioned training module 1304 is also used to: respond to text input instructions acting on the operation interface, and display the original text data on the operation interface; encode the original text data using a text encoder to obtain original text features corresponding to the original text data; extract based on the original discretized features and the original text features using a voice processing model to obtain target semantic features; and decode the target semantic features using a voice decoder to obtain target voice data.

[0215] Optionally, the above-mentioned training module 1304 is also used to: obtain first training data and second training data, wherein the first training data includes: first voice data and training text data, and the second training data at least includes: second voice data and preset processing results; discretize the first voice data to obtain the first discretized feature of the first voice data; use the first discretized feature and the training text data to perform unsupervised training on the initial processing model to obtain the target processing model; use the second training data to fine-tune the target processing model to obtain a voice processing model.

[0216] Optionally, the above-mentioned training module 1304 is also used to: use the initial processing model to process the first discretized feature and training text data to obtain predicted training data; construct an objective function corresponding to the initial processing model based on the first training data and the predicted training data; adjust the initial processing model based on the objective function to obtain a target processing model.

[0217] Optionally, the above-mentioned training module 1304 is also used to: discretize the second speech data to obtain a third discretized feature of the third speech data; use the target processing model to process the third speech data based on the third discretized feature to obtain a training processing result; based on the preset processing result and the training processing result, fine-tune the target processing model to obtain a speech processing model.

[0218] In an embodiment of the present application, based on speech processing technology, in the field of speech, through the first response module 1301, in response to the speech input instruction acting on the operation interface, a speech file containing original speech data is displayed on the operation interface, wherein the original speech data is the speech data collected by driving the audio acquisition device, and further through the second response module 1302, in response to the speech processing instruction acting on the operation interface, a target processing result corresponding to the original speech data is displayed on the operation interface, wherein the target processing result includes one of the following: target text data, a speech file containing target speech data, and the target processing result is obtained by processing the original speech data based on the original discretized features of the original speech data using the speech processing model, and the original discretized features are obtained by discretizing the original speech data.

[0219] It is easy to notice that, since the above-mentioned original discretized features can characterize the features of the original speech data with discretization advantages such as lower complexity and higher robustness, the target processing results obtained by processing the speech processing model based on the original discretized features can extend the advantages of the processing model to the speech data processing scenario. Therefore, the embodiment of the present application achieves the purpose of processing speech data using a speech processing model, thereby achieving the technical effect of improving the efficiency, accuracy, generalization ability and robustness of speech data processing, and further solves the technical problem of low efficiency, low accuracy, weak generalization ability and poor robustness of speech data processing caused by the fact that the prior art still relies on conventional models for single-person modeling to process speech data.

[0220] It should be noted that the first response module 1301 and the second response module 1302 correspond to steps S41 to S42 in Example 1, and the examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n), and the above-mentioned modules may also be part of the device and may be run in the computer terminal 10 provided in Example 1.

[0221] According to an embodiment of the present application, another device embodiment for implementing the training method of the speech processing model in the above-mentioned embodiment 2 is also provided. Fig.16 is a structural diagram of a training device for a speech processing model according to Example 6 of the present application, such as Fig.16 As shown, the device includes: an acquisition module 1601, which is used to acquire first training data and second training data, wherein the first training data includes: first voice data and training text data, and the second training data at least includes: second voice data and a preset processing result; a processing module 1602, which is used to discretize the first voice data to obtain a first discretized feature of the first voice data; a training module 1603, which is used to use the first discretized feature and the training text data to perform unsupervised training on the initial processing model to obtain a target processing model; an adjustment module 1604, which is used to fine-tune the target processing model using the second training data to obtain a voice processing model, wherein the voice processing model is used to process the original voice data based on the original discretized feature of the original voice data to obtain a target processing result corresponding to the original voice data, the original discretized feature is obtained by discretizing the original voice data, the original voice data is the voice data collected by driving the audio collection device, and the target processing result includes one of the following: target text data or a voice file containing target voice data.

[0222] Optionally, the processing module 1602 is further used to: encode the first speech data using a speech encoder to obtain a first speech feature corresponding to the first speech data; and discretize the first speech feature using a target quantizer to obtain a first discretized feature.

[0223] Optionally, the training device of the above-mentioned speech processing model also includes: a generation module (not shown in the figure), which is used to obtain training speech data; encode the training speech data using an initial encoder to obtain training speech features corresponding to the training speech data; discretize the training speech features using an initial quantizer to obtain training discretized features; decode the training discretized features using an initial decoder to obtain generated speech data; based on the training speech data and the generated speech data, train the initial encoder, initial quantizer and initial decoder to obtain a speech encoder, a target quantizer and a speech decoder.

[0224] It should be noted that the acquisition module 1601, the processing module 1602, the training module 1603 and the adjustment module 1604 correspond to steps S81 to S84 in Example 2, and the four modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 2. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n), and the above-mentioned modules may also be part of the device and may be run in the computer terminal 10 provided in Example 1.

[0225] According to an embodiment of the present application, another device embodiment for implementing the voice data processing method in the above-mentioned embodiment 3 is also provided. Fig.17 is a structural diagram of another voice data processing device according to Embodiment 6 of the present application, such as Fig.17 As shown, the device comprises:

[0226] The driver module 1701 is used to drive the audio acquisition device to collect original voice data;

[0227] The first processing module 1702 is used to discretize the original speech data to obtain the original discretized features of the original speech data;

[0228] The second processing module 1703 is used to process the original speech data based on the original discretized features using the speech processing model to obtain a target processing result corresponding to the original speech data, wherein the target processing result includes one of the following: target text data, target speech data.

[0229] It should be noted that the above-mentioned driving module 1701, the first processing module 1702, and the second processing module 1703 correspond to steps S91 to S93 in Embodiment 3, and the three modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Embodiment 3. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n), and the above-mentioned modules may also be part of the device and may be run in the computer terminal 10 provided in Embodiment 1.

[0230] According to an embodiment of the present application, another device embodiment for implementing the voice data processing method in the above-mentioned embodiment 4 is also provided. Fig.18 is a structural diagram of another voice data processing device according to Embodiment 6 of the present application, such as Fig.18 As shown, the device includes: an acquisition module 1801, which is used to acquire original voice data by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter is the original voice data, and the original voice data is the voice data collected by driving the audio collection device; a first processing module 1802, which is used to discretize the original voice data to obtain the original discretized features of the original voice data; a second processing module 1803, which is used to process the original voice data based on the original discretized features using a voice processing model to obtain a target processing result corresponding to the original voice data, wherein the target processing result includes one of the following: target text data, target voice data; an output module 1804, which is used to output the target processing result by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the target processing result.

[0231] It should be noted that the acquisition module 1801, the first processing module 1802, the second processing module 1803 and the output module 1804 correspond to steps S1001 to S1004 in Embodiment 4, and the four modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in Embodiment 4. It should be noted that the modules or units may be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n), and the modules may also be part of the device and run in the computer terminal 10 provided in Embodiment 1.

[0232] According to an embodiment of the present application, another device embodiment for implementing the voice data processing method in the above-mentioned embodiment 5 is also provided. Fig.19is a structural diagram of another voice data processing device according to Embodiment 6 of the present application, such as Fig.19 As shown, the device includes: a receiving module 1901, which is used for the cloud server to receive the original voice data sent by the client, wherein the original voice data is the voice data collected by driving the audio collection device; a first processing module 1902, which is used for the cloud server to discretize the original voice data to obtain the original discretized features of the original voice data; a second processing module 1903, which is used for the cloud server to process the original voice data based on the original discretized features using the voice processing model to obtain the target processing result corresponding to the original voice data, wherein the target processing result includes one of the following: target text data, target voice data; a sending module 1904, which is used for the cloud server to send the target processing result to the client, wherein the target processing result is output by the client.

[0233] It should be noted that the receiving module 1901, the first processing module 1902, the second processing module 1903 and the sending module 1904 correspond to steps S1101 to S1104 in Embodiment 5, and the four modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above Embodiment 5. It should be noted that the above modules or units may be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n), and the above modules may also be part of the device and may be run in the computer terminal 10 provided in Embodiment 1.

[0234] It should be noted that the preferred implementation of this embodiment can refer to the relevant descriptions in Embodiments 1 to 5, which will not be repeated here.

[0235] Example 7

[0236] According to an embodiment of the present application, a computer terminal is also provided, which can be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal can also be replaced by a terminal device such as a mobile terminal.

[0237] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of the computer network.

[0238] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the voice data processing method: responding to the voice input instruction acting on the operation interface, displaying the voice file containing the original voice data on the operation interface, wherein the original voice data is the voice data collected by driving the audio acquisition device; responding to the voice processing instruction acting on the operation interface, displaying the target processing result corresponding to the original voice data on the operation interface, wherein the target processing result includes one of the following: target text data, a voice file containing the target voice data, and the target processing result is obtained by processing the original voice data based on the original discretized features of the original voice data using the voice processing model, and the original discretized features are obtained by discretizing the original voice data.

[0239] Optionally, Fig. 20 is a structural block diagram of a computer terminal according to Embodiment 7 of the present application, such as Fig. 20 As shown, the computer terminal 200 may include: one or more (only one is shown in the figure) processors 2002, a memory 2004, a storage controller 2006, and a peripheral interface 2008, wherein the peripheral interface 2008 is connected to a radio frequency module, an audio module and a display.

[0240] Among them, the memory 2004 can be used to store software programs and modules, such as the program instructions / modules corresponding to the method and device for processing voice data in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned method for processing voice data. The memory 2004 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 2004 may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal 200 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0241] The processor 2002 can call the information and application programs stored in the memory through the transmission device to perform the following steps: responding to the voice input instruction on the operation interface, displaying the voice file containing the original voice data on the operation interface, wherein the original voice data is the voice data collected by driving the audio collection device; responding to the voice processing instruction on the operation interface, displaying the target processing result corresponding to the original voice data on the operation interface, wherein the target processing result includes one of the following: target text data, a voice file containing the target voice data, and the target processing result is obtained by processing the original voice data based on the original discretized features of the original voice data using the voice processing model, and the original discretized features are obtained by discretizing the original voice data.

[0242] Optionally, the processor 2002 may also execute the following program code: encoding the original speech data using a speech encoder to obtain original speech features corresponding to the original speech data; and discretizing the original speech features using a target quantizer to obtain original discretized features.

[0243] Optionally, the processor 2002 may also execute the following steps of program code: obtaining training speech data; encoding the training speech data using an initial encoder to obtain training speech features corresponding to the training speech data; discretizing the training speech features using an initial quantizer to obtain training discretized features; decoding the training discretized features using an initial decoder to obtain generated speech data; training the initial encoder, initial quantizer, and initial decoder based on the training speech data and the generated speech data to obtain a speech encoder, a target quantizer, and a speech decoder.

[0244] Optionally, the processor 2002 may also execute the following program code: using a discriminator to discriminate the first speech data and the generated speech data to obtain a discrimination result; adjusting parameters of an initial encoder, an initial quantizer and an initial decoder based on the discrimination result to obtain a speech encoder, a target quantizer and a speech decoder.

[0245] Optionally, the processor 2002 may also execute the program code of the following steps: extracting based on the original discretized features using a speech processing model to obtain original semantic features corresponding to the original speech data; and decoding the original semantic features using a text decoder to obtain target text data.

[0246] Optionally, the processor 2002 may also execute the program code of the following steps: extracting based on the original discretized features using a speech processing model to obtain original semantic features corresponding to the original speech data; and decoding the original semantic features using a speech decoder to obtain target speech data.

[0247] Optionally, the processor 2002 may also execute the following program code steps: respond to a text input instruction on the operation interface and display the original text data on the operation interface; encode the original text data using a text encoder to obtain original text features corresponding to the original text data; extract based on the original discretized features and the original text features using a speech processing model to obtain target semantic features; and decode the target semantic features using a speech decoder to obtain target speech data.

[0248] Optionally, the processor 2002 may also execute program codes of the following steps: obtaining first training data and second training data, wherein the first training data includes first voice data and training text data, and the second training data includes at least second voice data and a preset processing result; discretizing the first voice data to obtain first discretized features of the first voice data; performing unsupervised training on an initial processing model using the first discretized features and the training text data to obtain a target processing model; and fine-tuning the target processing model using the second training data to obtain a voice processing model.

[0249] Optionally, the processor 2002 may also execute the program code of the following steps: using the initial processing model to process the first discretized feature and the training text data to obtain predicted training data; constructing an objective function corresponding to the initial processing model based on the first training data and the predicted training data; and adjusting the initial processing model based on the objective function to obtain a target processing model.

[0250] Optionally, the processor 2002 may also execute the following program code: discretize the second speech data to obtain a third discretized feature of the third speech data; process the third speech data based on the third discretized feature using a target processing model to obtain a training processing result; and fine-tune the target processing model based on the preset processing result and the training processing result to obtain a speech processing model.

[0251] The processor 2002 can call the information and application stored in the memory through the transmission device to perform the following steps: obtain the first training data and the second training data, wherein the first training data includes: the first voice data and the training text data, and the second training data at least includes: the second voice data and the preset processing result; discretize the first voice data to obtain the first discretization feature of the first voice data; use the first discretization feature and the training text data to perform unsupervised training on the initial processing model to obtain the target processing model; use the second training data to fine-tune the target processing model to obtain the voice processing model, wherein the voice processing model is used to process the original voice data based on the original discretization feature of the original voice data to obtain the target processing result corresponding to the original voice data, the original discretization feature is obtained by discretizing the original voice data, the original voice data is the voice data collected by driving the audio collection device, and the target processing result includes one of the following: target text data or a voice file containing the target voice data.

[0252] Optionally, the processor 2002 may also execute the following program code: encoding the first speech data using a speech encoder to obtain a first speech feature corresponding to the first speech data; and discretizing the first speech feature using a target quantizer to obtain a first discretized feature.

[0253] Optionally, the processor 2002 may also execute the following steps of program code: obtaining training speech data; encoding the training speech data using an initial encoder to obtain training speech features corresponding to the training speech data; discretizing the training speech features using an initial quantizer to obtain training discretized features; decoding the training discretized features using an initial decoder to obtain generated speech data; training the initial encoder, initial quantizer, and initial decoder based on the training speech data and the generated speech data to obtain a speech encoder, a target quantizer, and a speech decoder.

[0254] The processor 2002 can call the information and application programs stored in the memory through the transmission device to execute the following steps: drive the audio acquisition device to collect original voice data; discretize the original voice data to obtain the original discretized features of the original voice data; use the voice processing model to process the original voice data based on the original discretized features to obtain the target processing result corresponding to the original voice data, wherein the target processing result includes one of the following: target text data, target voice data.

[0255] The processor 2002 can call the information and application programs stored in the memory through the transmission device to perform the following steps: obtain the original voice data by calling the first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter is the original voice data, and the original voice data is the voice data collected by driving the audio collection device; discretize the original voice data to obtain the original discretized features of the original voice data; use the voice processing model to process the original voice data based on the original discretized features to obtain the target processing result corresponding to the original voice data, wherein the target processing result includes one of the following: target text data, target voice data; output the target processing result by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the target processing result.

[0256] The processor 2002 can call the information and application programs stored in the memory through the transmission device to execute the following steps: the cloud server receives the original voice data sent by the client, wherein the original voice data is the voice data collected by driving the audio collection device; the cloud server discretizes the original voice data to obtain the original discretized features of the original voice data; the cloud server uses the voice processing model to process the original voice data based on the original discretized features to obtain the target processing result corresponding to the original voice data, wherein the target processing result includes one of the following: target text data, target voice data; the cloud server sends the target processing result to the client, wherein the target processing result is output by the client.

[0257] By adopting the embodiment of the present application, a solution for a computer terminal for processing voice data is provided. Based on voice processing technology, in the field of voice, by responding to the voice input command acting on the operation interface, a voice file containing original voice data is displayed on the operation interface, wherein the original voice data is the voice data collected by driving the audio acquisition device, and further by responding to the voice processing command acting on the operation interface, a target processing result corresponding to the original voice data is displayed on the operation interface, wherein the target processing result includes one of the following: target text data, a voice file containing target voice data, and the target processing result is obtained by processing the original voice data based on the original discretized features of the original voice data using the voice processing model, and the original discretized features are obtained by discretizing the original voice data.

[0258] It is easy to notice that, since the above-mentioned original discretized features can characterize the features of the original speech data with discretization advantages such as lower complexity and higher robustness, the target processing results obtained by processing the speech processing model based on the original discretized features can extend the advantages of the processing model to the speech data processing scenario. Therefore, the embodiment of the present application achieves the purpose of processing speech data using a speech processing model, thereby achieving the technical effect of improving the efficiency, accuracy, generalization ability and robustness of speech data processing, and further solves the technical problem of low efficiency, low accuracy, weak generalization ability and poor robustness of speech data processing caused by the fact that the prior art still relies on conventional models for single-person modeling to process speech data.

[0259] It can be understood by those skilled in the art that Fig. 20 The structure shown is for illustration only, and the computer terminal may also be a terminal device such as a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, and a mobile Internet device (Mobile Internet Devices, MID). Fig. 20 The structure of the computer terminal is not limited. For example, the computer terminal 200 may also include Fig. 20 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Fig. 20 Different configurations are shown.

[0260] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which can include: a flash drive, ROM, RAM, a magnetic disk or an optical disk, etc.

[0261] Example 8

[0262] According to an embodiment of the present application, a computer-readable storage medium is also provided. Optionally, in this embodiment, the storage medium can be used to store program codes executed by any of the above methods.

[0263] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0264] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: in response to a voice input instruction acting on an operation interface, a voice file containing original voice data is displayed on the operation interface, wherein the original voice data is voice data collected by driving an audio acquisition device; in response to a voice processing instruction acting on the operation interface, a target processing result corresponding to the original voice data is displayed on the operation interface, wherein the target processing result includes one of the following: target text data, a voice file containing target voice data, and the target processing result is obtained by processing the original voice data based on the original discretized features of the original voice data using a voice processing model, and the original discretized features are obtained by discretizing the original voice data.

[0265] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: encoding the original speech data using a speech encoder to obtain original speech features corresponding to the original speech data; and discretizing the original speech features using a target quantizer to obtain original discretized features.

[0266] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: obtaining training speech data; encoding the training speech data using an initial encoder to obtain training speech features corresponding to the training speech data; discretizing the training speech features using an initial quantizer to obtain training discretized features; decoding the training discretized features using an initial decoder to obtain generated speech data; training the initial encoder, initial quantizer and initial decoder based on the training speech data and the generated speech data to obtain a speech encoder, a target quantizer and a speech decoder.

[0267] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: using a discriminator to discriminate the first speech data and the generated speech data to obtain a discrimination result; based on the discrimination result, adjusting parameters of an initial encoder, an initial quantizer, and an initial decoder to obtain a speech encoder, a target quantizer, and a speech decoder.

[0268] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: extracting based on the original discretized features using a speech processing model to obtain original semantic features corresponding to the original speech data; and decoding the original semantic features using a text decoder to obtain target text data.

[0269] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: extracting based on the original discretized features using a speech processing model to obtain original semantic features corresponding to the original speech data; and decoding the original semantic features using a speech decoder to obtain target speech data.

[0270] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: displaying original text data on the operation interface in response to a text input instruction acting on the operation interface; encoding the original text data using a text encoder to obtain original text features corresponding to the original text data; extracting based on the original discretized features and the original text features using a speech processing model to obtain target semantic features; and decoding the target semantic features using a speech decoder to obtain target speech data.

[0271] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: obtaining first training data and second training data, wherein the first training data includes: first voice data and training text data, and the second training data includes at least: second voice data and a preset processing result; discretizing the first voice data to obtain a first discretized feature of the first voice data; performing unsupervised training on an initial processing model using the first discretized feature and the training text data to obtain a target processing model; and fine-tuning the target processing model using the second training data to obtain a voice processing model.

[0272] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: using the initial processing model to process the first discretized feature and the training text data to obtain predicted training data; based on the first training data and the predicted training data, constructing an objective function corresponding to the initial processing model; based on the objective function, adjusting the initial processing model to obtain a target processing model.

[0273] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: discretizing the second speech data to obtain third discretized features of the third speech data; processing the third speech data based on the third discretized features using a target processing model to obtain a training processing result; and fine-tuning the target processing model based on preset processing results and training processing results to obtain a speech processing model.

[0274] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: obtaining first training data and second training data, wherein the first training data includes: first voice data and training text data, and the second training data includes at least: second voice data and a preset processing result; discretizing the first voice data to obtain a first discretized feature of the first voice data; performing unsupervised training on an initial processing model using the first discretized feature and the training text data to obtain a target processing model; fine-tuning the target processing model using the second training data to obtain a voice processing model, wherein the voice processing model is used to process the original voice data based on the original discretized feature of the original voice data to obtain a target processing result corresponding to the original voice data, the original discretized feature is obtained by discretizing the original voice data, the original voice data is voice data collected by driving an audio acquisition device, and the target processing result includes one of the following: target text data or a voice file containing target voice data.

[0275] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: encoding the first speech data using a speech encoder to obtain a first speech feature corresponding to the first speech data; and discretizing the first speech feature using a target quantizer to obtain a first discretized feature.

[0276] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: obtaining training speech data; encoding the training speech data using an initial encoder to obtain training speech features corresponding to the training speech data; discretizing the training speech features using an initial quantizer to obtain training discretized features; decoding the training discretized features using an initial decoder to obtain generated speech data; training the initial encoder, initial quantizer and initial decoder based on the training speech data and the generated speech data to obtain a speech encoder, a target quantizer and a speech decoder.

[0277] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: driving an audio acquisition device to acquire original voice data; discretizing the original voice data to obtain original discretized features of the original voice data; processing the original voice data based on the original discretized features using a voice processing model to obtain a target processing result corresponding to the original voice data, wherein the target processing result includes one of the following: target text data, target voice data.

[0278] Optionally, in this embodiment, a computer-readable storage medium is configured to store program codes for executing the following steps: obtaining original voice data by calling a first interface, wherein the first interface includes a first parameter, and a parameter value of the first parameter is the original voice data, and the original voice data is the voice data collected by driving an audio collection device; discretizing the original voice data to obtain original discretized features of the original voice data; processing the original voice data based on the original discretized features using a voice processing model to obtain a target processing result corresponding to the original voice data, wherein the target processing result includes one of the following: target text data, target voice data; outputting the target processing result by calling a second interface, wherein the second interface includes a second parameter, and a parameter value of the second parameter is the target processing result.

[0279] Optionally, in this embodiment, a computer-readable storage medium is configured to store program codes for executing the following steps: a cloud server receives original voice data sent by a client, wherein the original voice data is voice data collected by driving an audio acquisition device; the cloud server discretizes the original voice data to obtain original discretized features of the original voice data; the cloud server processes the original voice data based on the original discretized features using a voice processing model to obtain a target processing result corresponding to the original voice data, wherein the target processing result includes one of the following: target text data, target voice data; the cloud server sends the target processing result to the client, wherein the target processing result is output by the client.

[0280] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0281] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0282] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0283] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0284] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0285] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, or all or part of the technical solution. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, ROM, RAM, mobile hard disks, magnetic disks or optical disks.

[0286] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for processing speech data, characterized in that: include: In response to a voice input command acting on the operation interface, a voice file containing original voice data is displayed on the operation interface, wherein the original voice data is voice data collected by driving an audio collection device; In response to the voice processing instruction acting on the operation interface, the target processing result corresponding to the original voice data is displayed on the operation interface, wherein the target processing result includes one of the following: target text data, a voice file containing target voice data, and the target processing result is obtained by processing the original voice data based on the original discretized features of the original voice data using a voice processing model, and the original discretized features are obtained by discretizing the original voice data.

2. The method according to claim 1, characterized in that Discretization processing is performed on the original voice data, including: Encoding the original speech data using a speech encoder to obtain original speech features corresponding to the original speech data; The original speech features are discretized using a target quantizer to obtain the original discretized features.

3. The method according to claim 2, characterized in that The method further comprises: Get training speech data; Encoding the training speech data using an initial encoder to obtain training speech features corresponding to the training speech data; Discretizing the training speech features using an initial quantizer to obtain training discretized features; Decoding the training discretized features using an initial decoder to generate speech data; Based on the training speech data and the generated speech data, the initial encoder, the initial quantizer and the initial decoder are trained to obtain the speech encoder, the target quantizer and the speech decoder.

4. The method according to claim 1, characterized in that: In the case where the target processing result includes: the target text data, the original speech data is processed based on the original discretized features of the original speech data using a speech processing model, including: Extracting the original discretized features using the speech processing model to obtain original semantic features corresponding to the original speech data; The original semantic features are decoded using a text decoder to obtain the target text data.

5. The method according to claim 1, characterized in that: In the case where the target processing result includes: a voice file including the target voice data, processing the original voice data based on the original discretized features of the original voice data using a voice processing model includes: Extracting the original discretized features using the speech processing model to obtain original semantic features corresponding to the original speech data; The original semantic features are decoded by using a speech decoder to obtain the target speech data.

6. The method according to claim 1, characterized in that In the case where the target processing result includes: a voice file including the target voice data, processing the original voice data based on the original discretized features of the original voice data using a voice processing model includes: In response to a text input instruction acting on the operation interface, displaying original text data on the operation interface; Encoding the original text data using a text encoder to obtain original text features corresponding to the original text data; Using the speech processing model to extract based on the original discretized features and the original text features, to obtain target semantic features; The target semantic feature is decoded by using a speech decoder to obtain the target speech data.

7. The method according to claim 1, characterized in that The method further comprises: Acquire first training data and second training data, wherein the first training data includes: first voice data and training text data, and the second training data includes at least: second voice data and a preset processing result; Performing discretization processing on the first speech data to obtain first discretization features of the first speech data; Using the first discretized features and the training text data, unsupervised training is performed on the initial processing model to obtain a target processing model; The target processing model is fine-tuned using the second training data to obtain the speech processing model.

8. A method for training a speech processing model, characterized in that: include: Acquire first training data and second training data, wherein the first training data includes: first voice data and training text data, and the second training data includes at least: second voice data and a preset processing result; Performing discretization processing on the first speech data to obtain first discretization features of the first speech data; Using the first discretized features and the training text data, unsupervised training is performed on the initial processing model to obtain a target processing model; The target processing model is fine-tuned using the second training data to obtain a speech processing model, wherein the speech processing model is used to process the original speech data based on the original discretized features of the original speech data to obtain a target processing result corresponding to the original speech data, the original discretized features are obtained by discretizing the original speech data, the original speech data is speech data collected by driving an audio acquisition device, and the target processing result includes one of the following: target text data or a speech file containing target speech data.

9. The method according to claim 8, characterized in that Discretizing the first speech data to obtain a first discretized feature of the first speech data includes: Encoding the first voice data using a voice encoder to obtain a first voice feature corresponding to the first voice data; The first speech feature is discretized using a target quantizer to obtain the first discretized feature.

10. A method for processing speech data, characterized in that: include: The cloud server receives the original voice data sent by the client, wherein the original voice data is the voice data collected by driving the audio collection device; The cloud server discretizes the original voice data to obtain original discretized features of the original voice data; The cloud server processes the original voice data based on the original discretized features using a voice processing model to obtain a target processing result corresponding to the original voice data, wherein the target processing result includes one of the following: target text data, target voice data; The cloud server sends the target processing result to the client, wherein the target processing result is output by the client.

11. A method for processing speech data, characterized in that: include: Drive the audio acquisition device to collect original voice data; Performing discretization processing on the original speech data to obtain original discretized features of the original speech data; The original speech data is processed based on the original discretized features using a speech processing model to obtain a target processing result corresponding to the original speech data, wherein the target processing result includes one of the following: target text data, target speech data.

12. A method for processing speech data, characterized in that: include: Acquire original voice data by calling a first interface, wherein the first interface includes a first parameter, a parameter value of the first parameter is the original voice data, and the original voice data is voice data collected by driving an audio collection device; Performing discretization processing on the original speech data to obtain original discretized features of the original speech data; Processing the original speech data based on the original discretized features using a speech processing model to obtain a target processing result corresponding to the original speech data, wherein the target processing result includes one of the following: target text data, target speech data; The target processing result is output by calling a second interface, wherein the second interface includes a second parameter, and a parameter value of the second parameter is the target processing result.

13. An electronic device, characterized in that: include: A memory storing an executable program; A processor, configured to run the program, wherein the program executes the method according to any one of claims 1 to 12 when running.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 12.

Images