Speech data processing method, and electronic device and storage medium

Abstract

A speech data processing method, and an electronic device and a storage medium. The method comprises: in response to a speech input instruction that acts on an operation interface, displaying, on the operation interface, a speech file including original speech data, wherein the original speech data is speech data that an audio collection device is driven to collect (S41); and in response to a speech processing instruction that acts on the operation interface, displaying, on the operation interface, a target processing result corresponding to the original speech data, wherein the target processing result comprises one of the following: target text data, and a speech file including target speech data, the target processing result is obtained by means of processing the original speech data by using a speech processing model and on the basis of an original discretization feature of the original speech data, and the original discretization feature is obtained by means of performing discretization processing on the original speech data (S42). The present application solves the technical problems of the efficiency of speech data processing being low, the accuracy being low, the generalization capability being weak and the robustness being poor due to the fact that in the prior art, speech data is still processed by means of a conventional model for performing modeling for a single person.

Description

Voice data processing method, electronic device and storage medium Technical Field

[0001] The present application relates to the fields of speech processing technology and computer technology, and in particular to a method for processing speech data, an electronic device, and a storage medium. Background Art

[0002] Currently, large language models pre-trained on massive amounts of text have achieved technological breakthroughs in semantic understanding. However, in the speech domain, scenarios such as speech recognition, speech synthesis, and speech enhancement still rely on individual characters and small models to complete specific tasks. This results in low speech data processing efficiency, low accuracy, weak generalization, and poor robustness.

[0003] To address the above-mentioned problems, no effective solutions have been proposed so far.

[0004] Summary of the Invention

[0005] The embodiments of the present application provide a method for processing speech data, an electronic device, and a storage medium to at least solve the technical problems of low speech data processing efficiency, low accuracy, weak generalization ability, and poor robustness caused by the fact that the existing technology still relies on conventional models for single-person modeling to process speech data.

[0006] According to one aspect of an embodiment of the present application, a method for processing voice data is provided, comprising: responding to a voice input instruction acting on an operation interface, displaying a voice file containing original voice data on the operation interface, wherein the original voice data is voice data collected by driving an audio acquisition device; responding to a voice processing instruction acting on the operation interface, displaying a target processing result corresponding to the original voice data on the operation interface, wherein the target processing result includes one of the following: target text data, a voice file containing target voice data, and the target processing result is obtained by processing the original voice data based on the original discretized features of the original voice data using a voice processing model, and the original discretized features are obtained by discretizing the original voice data.

[0007] According to another aspect of an embodiment of the present application, a training method for a speech processing model is also provided, including: obtaining first training data and second training data, wherein the first training data includes: first speech data and training text data, and the second training data includes at least: second speech data and a preset processing result; discretizing the first speech data to obtain a first discretized feature of the first speech data; using the first discretized feature and the training text data, performing unsupervised training on the initial processing model to obtain a target processing model; using the second training data to fine-tune the target processing model to obtain a speech processing model, wherein the speech processing model is used to process the original speech data based on the original discretized feature of the original speech data to obtain a target processing result corresponding to the original speech data, the original discretized feature is obtained by discretizing the original speech data, and the original speech data is the speech data collected by driving the audio acquisition device, and the target processing result includes one of the following: target text data or a speech file containing target speech data.

[0008] According to another aspect of an embodiment of the present application, a method for processing speech data is also provided, including: driving an audio acquisition device to acquire original speech data; discretizing the original speech data to obtain original discretized features of the original speech data; processing the original speech data based on the original discretized features using a speech processing model to obtain a target processing result corresponding to the original speech data, wherein the target processing result includes one of the following: target text data, target speech data.

[0009] According to another aspect of an embodiment of the present application, a method for processing voice data is also provided, including: obtaining original voice data by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter is the original voice data, and the original voice data is the voice data collected by driving an audio acquisition device; discretizing the original voice data to obtain the original discretized features of the original voice data; using a voice processing model to process the original voice data based on the original discretized features to obtain a target processing result corresponding to the original voice data, wherein the target processing result includes one of the following: target text data, target voice data; outputting the target processing result by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the target processing result.

[0010] According to another aspect of an embodiment of the present application, an electronic device is further provided, including: a memory storing an executable program; and a processor configured to run the program, wherein any one of the above methods is executed when the program is run.

[0011] According to another aspect of an embodiment of the present application, a computer-readable storage medium is further provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is run, the device where the computer-readable storage medium is located is controlled to execute any one of the above methods.

[0012] In an embodiment of the present application, based on voice processing technology, in the field of voice, by responding to voice input instructions acting on an operation interface, a voice file containing original voice data is displayed on the operation interface, wherein the original voice data is the voice data collected by driving the audio acquisition device, and further by responding to the voice processing instructions acting on the operation interface, the target processing result corresponding to the original voice data is displayed on the operation interface, wherein the target processing result includes one of the following: target text data, a voice file containing target voice data, and the target processing result is obtained by processing the original voice data based on the original discretized features of the original voice data using a voice processing model, and the original discretized features are obtained by discretizing the original voice data.

[0013] It is easy to notice that since the above-mentioned original discretized features can characterize the characteristics of the original speech data with discretization advantages such as lower complexity and higher robustness, the target processing results obtained by processing the original discretized features using a speech processing model can extend the advantages of discretized feature representation to the speech data processing scenario. Therefore, the present application achieves the purpose of processing speech data using a speech processing model based on discretized feature representation, thereby achieving the technical effect of improving the efficiency, accuracy, generalization ability and robustness of speech data processing, and thus solves the technical problem of low efficiency, low accuracy, weak generalization ability and poor robustness of speech data processing caused by the fact that the existing technology still relies on conventional models for single-person modeling to process speech data.

[0014] It is easy to notice that the above general description and the following detailed description are merely for the purpose of exemplifying and explaining the present application, and do not constitute a limitation of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0016] FIG1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a method for processing voice data;

[0017] FIG2 is a schematic diagram of a hardware environment of a virtual reality device according to a method for processing voice data in an embodiment of the present application;

[0018] FIG3 is a structural block diagram of a computing environment of a method for processing voice data according to an embodiment of the present application;

[0019] FIG4 is a flowchart of a method for processing voice data according to Example 1 of the present application;

[0020] FIG5 is a schematic diagram of an optional speech processing model pre-training process according to Example 1 of the present application;

[0021] FIG6 is a schematic diagram of another optional speech processing model pre-training process according to Example 1 of the present application;

[0022] FIG7 is a schematic diagram of another optional speech processing model reasoning process according to Example 1 of the present application;

[0023] FIG8 is a flow chart of a method for training a speech processing model according to Example 2 of the present application;

[0024] FIG9 is a flowchart of a method for processing voice data according to Example 3 of the present application;

[0025] FIG10 is a flowchart of a method for processing voice data according to Example 4 of the present application;

[0026] FIG11 is a flowchart of a method for processing voice data according to Example 5 of the present application;

[0027] FIG12 is a schematic diagram of processing voice data on a cloud server according to Embodiment 5 of the present application;

[0028] FIG13 is a schematic structural diagram of a voice data processing device according to Example 6 of the present application;

[0029] FIG14 is a schematic structural diagram of an optional apparatus for processing voice data according to Example 6 of the present application;

[0030] FIG15 is a schematic structural diagram of another optional apparatus for processing voice data according to Example 6 of the present application;

[0031] FIG16 is a schematic structural diagram of a training device for a speech processing model according to Example 6 of the present application;

[0032] FIG17 is a schematic structural diagram of another apparatus for processing voice data according to Example 6 of the present application;

[0033] FIG18 is a schematic structural diagram of another apparatus for processing voice data according to Embodiment 6 of the present application;

[0034] FIG19 is a schematic structural diagram of another apparatus for processing voice data according to Embodiment 6 of the present application;

[0035] Figure 20 is a structural block diagram of a computer terminal according to Example 7 of the present application. DETAILED DESCRIPTION

[0036] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0037] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0038] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following interpretations:

[0039] Speech Tokenization (SpeechTokenizer) is the process of dividing a continuous speech signal into discrete segments or words. In speech recognition and speech processing, speech signals are typically continuous waveforms, but for subsequent analysis and processing, they need to be divided into discrete speech units. This discretized form of speech is more suitable for tasks such as speech signal processing, feature extraction, and speech recognition.

[0040] Speech codec (Encodec): refers to the algorithm used to encode and compress speech signals and decode and decompress them.

[0041] Speech processing model: In this solution, it refers to the Speech-Large Language Model (Speech-LLM), which is a deep learning language model trained based on a large-scale speech dataset for processing speech data.

[0042] SpeechGPT (Generative Pre-training for Unsupervised Speech) is an unsupervised pre-training method based on generative models that aims to model and generate speech signals. It is an extension of the GPT (Generative Pre-trained Transformer) model for speech. Pre-training uses unlabeled speech data and, based on generative models, learns the characteristics and structure of speech signals. SpeechGPT provides an effective transfer learning method for speech-related tasks.

[0043] Instruction fine-tuning: A technique used to optimize deep learning models. In deep learning, model training typically involves a large number of computational operations, including matrix multiplication and convolution. These operations may exhibit different performance on different hardware. The goal of instruction fine-tuning is to optimize model performance by fine-tuning the instruction set used for these operations. Specifically, instruction fine-tuning improves the efficiency of these operations by leveraging hardware-specific instructions.

[0044] Example 1

[0045] According to an embodiment of the present application, an embodiment of a method for processing voice data is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0046] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a method for processing voice data. As shown in Figure 1, the computer terminal 10 (or mobile device 10) may include one or more (102a, 102b, ..., 102n are used in the figure to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor (Microcontroller Unit, MCU) or a programmable logic device (Field Programmable Gate Array, FPGA)), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, the computer terminal 10 may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a cursor control device (such as a mouse, a touchpad, etc.), a keyboard, a power supply and / or a camera.

[0047] Those skilled in the art will appreciate that the structure shown in FIG1 is merely illustrative and does not limit the structure of the electronic device. For example, the computer terminal 10 may include more or fewer components than shown in FIG1 , or may have a configuration different from that shown in FIG1 .

[0048] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0049] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the voice data processing method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizes the above-mentioned voice data processing method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0050] The transmission device 106 is configured to connect to a network via a network interface to receive or transmit data. Specific examples of the aforementioned network may include a wired and / or wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module configured to communicate with the Internet wirelessly.

[0051] The display shown in FIG1 may be, for example, a touch screen liquid crystal display (LCD), which enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0052] It should be noted that, in some optional embodiments, the computer device (or mobile device) shown in FIG. 1 may include hardware components (including circuits), software components (including computer code stored on a computer-readable medium), or a combination of hardware and software components. It should be noted that FIG. 1 is merely an example of a specific embodiment and is intended to illustrate the types of components that may be present in the computer device (or mobile device).

[0053] It should be noted that, in some embodiments, the computer device (or mobile device) shown in FIG. 1 has a touch display (also referred to as a "touch screen" or "touch display screen"). In some embodiments, the computer device (or mobile device) shown in FIG. 1 has a graphical user interface (GUI), and a user can interact with the GUI through finger contacts and / or gestures on the touch-sensitive surface. The human-computer interaction functions herein may optionally include the following interactions: creating web pages, drawing, word processing, creating electronic documents, playing games, video conferencing, instant messaging, sending and receiving emails, call interfaces, playing digital videos, playing digital music, and / or browsing the Internet, etc. Executable instructions for executing the above human-computer interaction functions are configured / stored in a computer program product or readable storage medium executable by one or more processors.

[0054] The method embodiment provided in Example 1 of the present application can be executed in a virtual reality (VR) device or an augmented reality (AR) device. Taking a VR device as an example, FIG2 is a schematic diagram of the hardware environment of a virtual reality device according to a method for processing voice data in an embodiment of the present application. As shown in FIG2 , the virtual reality device 204 is connected to the terminal 206, and the terminal 206 is connected to the server 202 via a network. The above-mentioned virtual reality device 204 is not limited to: a virtual reality helmet, virtual reality glasses, a virtual reality all-in-one machine, etc. The above-mentioned terminal 204 is not limited to a PC, a mobile phone, a tablet computer, etc. The server 202 can be a server corresponding to a media file operator, and the above-mentioned network includes but is not limited to: a wide area network, a metropolitan area network or a local area network.

[0055] Optionally, the virtual reality device 204 of the embodiment of the present application includes: a memory 2041, a processor 2042, and a transmission device 2043 (not shown in Figure 2). The memory 2041 is used to store an application program, which can be used to execute the voice data processing method provided in the embodiment of the present application. The processor 2042 can call and execute the application program stored in the memory 2041 to obtain the target processing result of the voice data using the voice data processing method provided in the embodiment of the present application.

[0056] Optionally, the terminal 206 of the embodiment of the present application can also be used to execute the voice data processing method, and display the visualization results corresponding to the voice data processing results through the display screen of the VR device or AR device, or play the processing results obtained by executing the voice data processing method through the audio output device associated with the VR device or AR device.

[0057] The hardware structure block diagram shown in Figure 2 can not only serve as an exemplary block diagram of the hardware environment of the above-mentioned AR / VR device (or mobile device), but also as an exemplary block diagram of the above-mentioned server.

[0058] Figure 3 is a structural block diagram of a computing environment for a method for processing voice data according to an embodiment of the present application. Figure 3 shows in a block diagram an embodiment of using the VR device (or mobile device) shown in Figure 2 above as a computing node in the computing environment 301.

[0059] Still as shown in FIG3 , computing environment 301 includes multiple computing nodes (e.g., servers) (illustrated in FIG3 as 310 - 1 , 310 - 2 , ...) running on a distributed network. Each computing node includes local processing and memory resources, and end user 302 can remotely run applications or store data in computing environment 301. Applications can be provided as multiple services 320 - 1 (representing service A), 320 - 2 (representing service D), 320 - 3 (representing service E), and 320 - 4 (representing service H) in computing environment 301.

[0060] Still as shown in FIG3 , end user 302 can provide and access services through a web browser or other software application on a client. In some embodiments, the provisioning and / or request of end user 302 can be provided to ingress gateway 330. Ingress gateway 330 can include a corresponding agent to handle the provisioning and / or request for service 320 (one or more services provided in computing environment 301).

[0061] Still as shown in Figure 3, service 320 is provided or deployed based on various virtualization technologies supported by computing environment 301. In some embodiments, service 320 can be provided based on virtual machine (VM)-based virtualization, container-based virtualization, and / or similar methods. Virtual machine-based virtualization can be to simulate a real computer by initializing a virtual machine, executing programs and applications without directly contacting any actual hardware resources. While virtual machines virtualize machines, according to container-based virtualization, containers can be started to virtualize the entire operating system (OS) so that multiple workloads can run on a single operating system instance.

[0062] In one embodiment based on container virtualization, several containers of a service 320 can be assembled into a POD (e.g., a Kubernetes POD). For example, as shown in Figure 3, service 320-2 can be equipped with one or more PODs 340-1, 340-2, ..., 340-N (collectively referred to as POD 340). Each POD 340 can include a proxy 345 and one or more containers 342-1, 342-2, ..., 342-M (collectively referred to as containers 342). One or more containers 342 in POD 340 process requests related to one or more corresponding functions of the service, and the proxy 345 generally controls network functions related to the service, such as routing, load balancing, etc. Other services 320 can also be PODs similar to POD 340.

[0063] During operation, executing a user request from an end user 302 may require invoking one or more services 320 in the computing environment 301. Executing one or more functions of one service 320 may require invoking one or more functions of another service 320. As shown in Figure 3, service 320-1 (representing service A) receives a user request from an end user 302 from an ingress gateway 330. Service 320-1 (representing service A) may invoke service 320-2 (representing service D), and service 320-2 (representing service D) may request service 320-3 (representing service E) to execute one or more functions.

[0064] This computing environment can be a cloud computing environment, where resource allocation is managed by the cloud service provider, allowing for feature development without having to worry about implementing, adjusting, or scaling servers. This computing environment allows developers to execute code in response to events without building or maintaining complex infrastructure. Services can be partitioned to perform a set of functions that can scale independently and automatically, rather than scaling a single hardware device to handle the potential load.

[0065] In the above operating environment, the present application provides a method for processing voice data as shown in Figure 4. Figure 4 is a flow chart of a method for processing voice data according to Example 1 of the present application. As shown in Figure 4, the method for processing voice data includes:

[0066] Step S41 , in response to a voice input instruction applied to the operation interface, a voice file containing original voice data is displayed on the operation interface, where the original voice data is voice data collected by driving the audio collection device.

[0067] The operation interface may be a graphical user interface provided by the client. A file identifier may be displayed on the graphical user interface, which is used to display a voice file containing the original voice data. The audio acquisition device may be an input device associated with the terminal device corresponding to the client, such as a microphone, recorder, sound card, audio interface, etc. The voice data may be expressed in natural language. The voice input command may be triggered and executed by a first touch operation on the operation interface.

[0068] Step S42, in response to the voice processing instruction acting on the operation interface, the target processing result corresponding to the original voice data is displayed on the operation interface, the target processing result includes one of the following: target text data, a voice file containing target voice data, the target processing result is obtained by processing the original voice data based on the original discretization features of the original voice data using the voice processing model, and the original discretization features are obtained by discretizing the original voice data.

[0069] Optionally, the above-mentioned speech processing model is a large language model.

[0070] The voice processing instruction can be generated and executed by triggering a second touch operation on the operation interface. Displaying the target processing result corresponding to the original voice data on the operation interface includes one of the following: displaying the target text data on the operation interface, displaying the voice file identifier corresponding to the voice file on the operation interface, and controlling the audio output device corresponding to the operation interface to play the voice file containing the target voice data.

[0071] According to the above method provided in the embodiment of the present application, in the process of speech data processing, the advantages of using the original discretized features are: simplifying the model complexity, using discretized features can significantly reduce the complexity of the feature space, reducing the computational complexity and storage requirements of the model; enhancing the robustness of the model, the influence of outliers or noise points in the speech data on the discretized features will be weakened, and using discretized features can reduce the sensitivity of the model to outliers and noise; improving the interpretability of the model, the discretized features are easier to understand and explain; using discretized features can capture the nonlinear relationship between the original continuous features and the target variables, so that the model can better learn these nonlinear relationships; improving the generalization ability of the model, discretized features can reduce the overfitting of the model to specific samples, and using discretized features can make the model better generalize to unseen samples.

[0072] Because large language models have stronger expressive power, better generalization ability, higher accuracy, better transfer learning ability, and stronger robustness than conventional small models, in one application scenario of the above method provided in an embodiment of the present application, a large language model is used as a speech processing model, and the original speech data is processed based on the original discretized features of the original speech data to obtain the target processing result, which can extend the advantages of the large language model to the speech data processing scenario.

[0073] According to the above method provided in the embodiment of the present application, it is possible to integrate various functional requirements involved in speech processing (for example, recognition, synthesis, translation, enhancement, monitoring, etc.) into the speech processing model. Therefore, the above method can be applied to, but is not limited to, scenes such as character speech recognition, speech synthesis, speech translation, speech enhancement, audio event monitoring, and audio emotion recognition. The implementation of the above method in the above scenario includes but is not limited to application programs (such as translation software, intelligent generation software, etc.), functional plug-ins, intelligent hardware (such as smart speakers, smart wearable devices, etc.), etc. Furthermore, the above speech processing model provided in the embodiment of the present application can also be combined with a semantic big model to extend the above method to a wider range of application scenarios. For example, the above method is used to provide speech data processing for enterprise big models, education big models, medical big models, etc.

[0074] Specifically, in the application scenario of the intelligent voice assistant, a graphical user interface is displayed on the terminal device where the client corresponding to the intelligent voice assistant is running. The graphical user interface serves as an operation interface that can respond to touch operations. When a first touch operation acting on the graphical user interface is detected, a voice input instruction is generated and executed to display a voice file identifier on the graphical user interface. For example, the first touch operation can be: clicking the "speak" button, pressing and holding the "speak" button, long pressing the designated area, etc. The voice file identifier is an identifier corresponding to the original voice data input by the user. For example, the original voice data is 10 seconds of voice data collected by the microphone of the terminal device after the user presses and holds the "speak" button, and the voice file identifier is an identifier that identifies the data number, duration, and voice flag.

[0075] Furthermore, in the application scenario of the intelligent voice assistant, when a second touch operation acting on the graphical user interface is detected, the voice data processing process is called, and the target processing result obtained is displayed at a specified location on the graphical user interface, or the target processing result is output to the user through an audio output device. For example, the second touch operation can be: clicking the "OK" button, long pressing the specified area, pressing the specified area again, etc. The voice data processing process specifically includes: using the pre-trained voice assistant large model (i.e., the voice processing model) to process the original voice data based on the original discretized features of the original voice data. For example, in daily application scenarios, after entering a voice in the graphical user interface corresponding to the intelligent voice assistant, the intelligent voice assistant will return a text and / or voice. For example, by voice inputting the question "How do you say apple in English", the intelligent voice assistant first calls the voice data processing process to discretize the question and uses the voice assistant large model to process the question based on the discretized features of the question. The processing results obtained include the text "Apple" and the audio "Apple". On this basis, the text message "Apple" is displayed on the graphical user interface of the intelligent voice assistant, and the speaker of the terminal device plays the audio corresponding to "Apple".

[0076] Similarly, in the application scenario of smart speakers, smart speakers may provide an operation interface (similar to smart voice assistants, which will not be described in detail), or may not provide an operation interface (only through button or voice wake-up or triggering to generate instructions). When the smart speaker detects the wake-up word, it triggers the generation and execution of voice input instructions, and collects original voice data through the audio acquisition device of the smart speaker. For example, the wake-up word is "Hello". When the smart speaker detects that the user says "Hello", a recording event is triggered, that is, the next voice input by the user is obtained until the end of the first preset time or no voice input is detected within the second preset time. Furthermore, the file identifier corresponding to the obtained original voice file is displayed on the operation interface (if it can only be provided by the speaker).

[0077] Furthermore, in the application scenario of smart speakers, the voice processing instructions are set to be triggered and executed immediately after the original voice data is collected. After the smart speaker collects the question input by the user, it calls the voice data processing process and outputs the target processing result to the user in audio form. The voice data processing process is similar to that of smart voice assistants.

[0078] It should be noted that in the application scenario of the smart speaker, the smart speaker can also be associated with a mobile terminal device or a smart wearable device, and the mobile terminal device or the smart wearable device provides an operation interface corresponding to the smart speaker, so as to realize the above-mentioned response function, display function, etc. through the operation interface. In an embodiment of the present application, based on voice processing technology, in the field of voice, by responding to the voice input instruction acting on the operation interface, a voice file containing original voice data is displayed on the operation interface, wherein the original voice data is the voice data collected by driving the audio acquisition device, and further by responding to the voice processing instruction acting on the operation interface, the target processing result corresponding to the original voice data is displayed on the operation interface, wherein the target processing result includes one of the following: target text data, a voice file containing target voice data, and the target processing result is obtained by processing the original voice data based on the original discretized features of the original voice data using the voice processing model, and the original discretized features are obtained by discretizing the original voice data.

[0079] It is easy to notice that since the above-mentioned original discretized features can characterize the characteristics of the original speech data with discretization advantages such as lower complexity and higher robustness, the target processing results obtained by processing the original discretized features using a speech processing model can extend the advantages of discretized feature representation to the speech data processing scenario. Therefore, the present application achieves the purpose of processing speech data using a speech processing model based on discretized feature representation, thereby achieving the technical effect of improving the efficiency, accuracy, generalization ability and robustness of speech data processing, and thus solves the technical problem of low efficiency, low accuracy, weak generalization ability and poor robustness of speech data processing caused by the fact that the existing technology still relies on conventional models for single-person modeling to process speech data.

[0080] In a specific implementation of an embodiment of the present application, the structure of the speech processing model (Speech-LLM) constructed in the application scenario is shown in Figure 5. The training process of the speech processing model may include three stages: the first stage is the audio discretization (SpeechTokenizer) pre-training stage; the second stage is the Speech-LLM unsupervised pre-training stage based on massive speech data and text data; the third stage is the instruction fine-tuning stage.

[0081] In an optional embodiment, discretization processing is performed on the original speech data, including the following method steps:

[0082] Step S4211, encoding the original speech data using a speech encoder to obtain original speech features corresponding to the original speech data;

[0083] Step S4210: discretize the original speech features using a target quantizer to obtain original discretized features.

[0084] In the above optional embodiment, the original speech data includes original speech and original text. The speech encoder is an encoder pre-trained based on massive speech data and text data. The target quantizer is a quantizer pre-trained based on massive speech data and text data.

[0085] In an optional embodiment, the voice data processing method further includes the following steps:

[0086] Step S502, obtaining training speech data;

[0087] Step S504: Encode the training speech data using the initial encoder to obtain training speech features corresponding to the training speech data;

[0088] Step S506, using an initial quantizer to discretize the training speech features to obtain training discretized features;

[0089] Step S508, using the initial decoder to decode the training discretized features to obtain generated speech data;

[0090] Step S510 : Based on the training speech data and the generated speech data, the initial encoder, the initial quantizer, and the initial decoder are trained to obtain a speech encoder, a target quantizer, and a speech decoder.

[0091] In one specific application scenario, the training speech data includes a training speech and a training text. In another specific application scenario, the training speech data includes a training speech, and a training speech feature corresponding to the training speech serves as the training text.

[0092] As shown in Figure 5, in the first stage of the training process of the speech processing model: the initial encoder (Encodec) is used to encode the training speech, and the obtained training speech features are used as training text; the discrete word segmenter (also known as the initial quantizer, GPT2Tokenizer can be used in the scenario) is used to discretize the training text to obtain training discretized features, and the training discretized features are used as input for the second stage.

[0093] As shown in FIG5 , in the second stage of the training process of the speech processing model: Speech-LLM unsupervised training is performed based on the training discretized features to obtain unsupervised training results, which are used as input for the third stage.

[0094] As shown in Figure 5, in the third stage of the training process of the speech processing model: the initial decoder (Decodec) is used to decode the unsupervised training results to obtain the output speech (i.e., speech data); the discrete word segmenter is used to de-discretize the unsupervised training results to obtain the output text.

[0095] Furthermore, based on the training speech, the output speech, the training text and the output text, the initial encoder, the initial quantizer and the initial decoder are trained to obtain a speech encoder, a target quantizer and a speech decoder.

[0096] In an optional embodiment, in step S510, the training speech data includes first speech data, and based on the first speech data and the generated speech data, an initial encoder, an initial quantizer, and an initial decoder are trained to obtain a speech encoder, a target quantizer, and a speech decoder, including the following method steps:

[0097] Step S511, using a discriminator to discriminate the first voice data and the generated voice data to obtain a discrimination result;

[0098] Step S512: Adjust the parameters of the initial encoder, the initial quantizer, and the initial decoder based on the discrimination result to obtain a speech encoder, a target quantizer, and a speech decoder.

[0099] In one specific embodiment, the specific process of pre-training the speech processing model is shown in Figure 6. Based on the training speech (i.e., the first speech data) and the output speech (i.e., the generated speech data), a discriminator performs discrimination to obtain a discrimination result. This discrimination result is used to adjust the parameters of the initial encoder, initial quantizer, and initial decoder to obtain the speech encoder, target quantizer, and speech decoder. Specifically, the discriminator can be used for loss calculation, and the discrimination result includes the training loss.

[0100] Still as shown in Figure 6, in the method provided by the present application, the audio codec model based on neural network and vector quantization (Vector Quantization) is used to discretize the speech data. The audio codec model includes four components: an encoder (Encoder), a residual vector quantizer (RVQ), a decoder (Decoder) and a discriminator (Discriminator). The encoder is used to encode the time domain speech waveform to obtain a hidden feature representation. The RVQ (the component shown by the oblique shadow as shown in Figure 6) is used to quantize the hidden feature representation layer by layer to obtain a layer-by-layer quantization result (Q1 to Q6 as shown in Figure 6). During the layer-by-layer quantization process, the latter layer quantizes the residual information of the previous layer. The decoder is used to decode the hidden feature representation based on the layer-by-layer quantization result to obtain the speech waveform of the output speech. The discriminator and the encoder constitute a generative adversarial structure, and the parameters of the audio codec model are adjusted through adversarial training to improve the model performance.

[0101] In addition, as shown in FIG6 , a denoising switch can be set in the audio codec model to control the denoising process during the codec process.

[0102] Furthermore, after the parameter adjustment process shown in Figure 6, the inference stage is shown in Figure 7. The encoder in the inference stage is a speech encoder obtained by adjusting the parameters of the initial encoder based on the discrimination results. The target quantizer in the inference stage (the component indicated by the diagonal shading in Figure 7) is a target quantizer obtained by adjusting the parameters of the initial quantizer based on the discrimination results. The decoder in the inference stage is a speech decoder obtained by adjusting the parameters of the initial decoder based on the discrimination results. The encoder and target quantizer can be deployed at the transmitter, and the decoder can be deployed at the receiver. The interaction between the transmitter and receiver is the quantization result obtained by the target quantizer.

[0103] It should be noted that in the above specific implementation of the present application, the bit rate required by the vector quantization module is reduced by compressing the frame rate and adopting different input features. For example, a speech with a duration of 1 second will be compressed into 100 discrete representations (i.e., 100 tokens / s). Compared with the discretization processing method based on Encodec in the conventional method (400 tokens / s), the above specific implementation of the present application has fewer discrete representations, which is conducive to reducing the modeling complexity of Speech-LLM.

[0104] In addition, when the training speech data includes training speech and training text, in the second stage of the training process of the speech processing model: the speech encoder obtained by training in the first stage is used to encode the training speech. For example, for a speech A with a duration of 5 seconds, 500 discrete token representations C are encoded; the target quantizer (such as GPT2Tokenizer) obtained by training in the first stage is used to quantize the training text W to obtain a modeling unit sequence T.

[0105] Specifically, the discrete token representation C is expressed as: C = Encodec(A) = [c1, c2, ..., c 500 ].

[0106] Specifically, the modeling unit sequence T is expressed as: T = GPT2Tokenizer(W) = [w1,w2,…,w n ].

[0107] For example, the training text W is “CompenyA Group's mission is to let the world have no difficult business.”, and the corresponding modeling unit sequence T is “CompenyA ”.

[0108] Furthermore, the training samples S used to train the speech processing model can be represented as: S = {C, T}. That is, during the training phase of the speech processing model, the massive amount of speech data is converted into discrete encoded representations through an encoder. These discrete encoded representations are then combined with the massive amount of text data and pre-trained using the language model pre-training criteria to produce the speech processing model (LLM).

[0109] In an optional embodiment, when the target processing result includes target text data, processing the original speech data using the speech processing model based on the original discretized features of the original speech data includes the following method steps:

[0110] Step S4221, extracting original semantic features corresponding to the original speech data using a speech processing model based on the original discretized features;

[0111] Step S4222: Decode the original semantic features using a text decoder to obtain target text data.

[0112] In this optional embodiment, the speech processing model may also be integrated with a semantic processing model. When the target processing result includes target text data, the semantic processing model is used to extract features based on the original discretized features to obtain original semantic features; the original semantic features are then decoded using a text decoder to obtain the target text data. The text decoder may be pre-trained using a similar training method as the speech decoder.

[0113] For example, in a speech recognition scenario, the above-mentioned original speech data is user input speech, and the user input speech is discretized to obtain the discretized features corresponding to the user input speech. A speech recognition processing model (i.e., a speech processing model) is used to extract based on the discretized features to obtain the audio semantic features corresponding to the user input speech. Furthermore, a text decoder is used to decode the audio semantic features to obtain the recognized text corresponding to the user input speech (i.e., target text data). Similarly, in an optional embodiment, when the target processing result includes: a speech file containing target speech data, the original speech data is processed based on the original discretized features of the original speech data using the speech processing model, including the following method steps:

[0114] Step S4231, extracting original semantic features corresponding to the original speech data using a speech processing model based on the original discretized features;

[0115] Step S4232: Use a speech decoder to decode the original semantic features to obtain target speech data.

[0116] For another example, in the scenario of speech enhancement, the above-mentioned original speech data is noisy speech data, and the noisy speech data is discretized to obtain the discretized features corresponding to the noisy speech data. The speech enhancement processing model (i.e., the speech processing model) is used to extract based on the discretized features to obtain the audio semantic features corresponding to the noisy speech data. Furthermore, the audio semantic features are decoded using a speech decoder to obtain the speech enhancement result corresponding to the noisy speech data (i.e., the target speech data, usually denoised speech data). Therefore, after the above-mentioned speech processing model provided in the embodiment of the present application is combined with the semantic processing model, the above-mentioned method can be extended to a wider range of application scenarios. For example, the above-mentioned method is used to provide speech data processing for enterprise processing models, education processing models, medical processing models, etc. In an optional embodiment, when the target processing result includes: in the case of a voice file containing target speech data, the original speech data is processed based on the original discretized features of the original speech data using the speech processing model, including the following method steps:

[0117] Step S4241, in response to a text input instruction applied to the operation interface, displaying original text data on the operation interface;

[0118] Step S4242: Encode the original text data using a text encoder to obtain original text features corresponding to the original text data;

[0119] Step S4243, using the speech processing model to extract based on the original discretized features and the original text features to obtain target semantic features;

[0120] Step S4244: Use a speech decoder to decode the target semantic features to obtain target speech data.

[0121] For another example, in the scenario of speech translation, the user inputs the text to be translated in the source language by executing a text input instruction on the operation interface, and the text to be translated in the source language is displayed on the operation interface. The text to be translated in the source language is discretized to obtain discretized features. The text to be translated in the source language is encoded using a text encoder to obtain text features corresponding to the text to be translated in the source language. Furthermore, a speech processing model is used to extract based on the discretized features and text features to obtain target semantic features, and the target semantic features correspond to the target language to be translated. Furthermore, a speech decoder is used to decode the target semantic features to obtain target speech data in the target language (including text data in the target language and / or speech data in the target language).

[0122] In the above optional embodiment, the text input instruction is generated and executed by triggering a third touch operation on the operation interface. For example, the third touch operation is a touch typing operation, a long press voice-to-text operation, etc.

[0123] When a speech processing model is integrated with a semantic processing model, target semantic features can be derived by combining the original text features with the original discretized features corresponding to the original speech data. Compared to semantic features derived solely from text features or discretized speech features, these target semantic features can more accurately represent the semantics of the original input data. Furthermore, the target speech data derived from these target semantic features can also have higher accuracy, robustness, and interpretability.

[0124] In an optional embodiment, the method for processing voice data further includes the following steps:

[0125] Step S602: Acquire first training data and second training data, wherein the first training data includes: first voice data and training text data, and the second training data includes at least: second voice data and a preset processing result;

[0126] Step S604: discretize the first speech data to obtain a first discretized feature of the first speech data;

[0127] Step S606, using the first discretized features and the training text data, performing unsupervised training on the initial processing model to obtain a target processing model;

[0128] Step S608: fine-tune the target processing model using the second training data to obtain a speech processing model.

[0129] In the aforementioned optional embodiment, the target processing model is a pre-trained processing model. To adapt the target processing model to various audio-related downstream tasks, the target processing model is fine-tuned using the first training data and the second training data to obtain a final speech processing model. The first training data and the second training data are obtained from task-specific datasets corresponding to the various downstream tasks, and the sample data in the task-specific datasets are labeled sample data.

[0130] In a specific embodiment, multiple audio-related downstream tasks include: Automatic Speech Recognition (ASR), Machine Translation (MT), Speech-to-Text Translation (S2TT), Personalized Text-to-Speech (PTTS), Speech Enhancement (SE), Audio Event Detection (AED), and Speech Emotion Recognition (SER). The input and output data types corresponding to the above-mentioned downstream tasks are shown in Table 1 below.

[0131] Table 1

[0132] The task-specific datasets can be pre-collected and created according to Table 1.

[0133] Taking the speech recognition task as an example, the input data of the speech processing model is the speech signal (Speech), and the model fine-tuning process specifically includes: using a trained and fixed-parameter speech encoder (Encodec) to encode the speech signal to obtain a discrete audio code sequence; inputting the discrete audio code sequence into the Speech-LLM of the second stage of pre-training, and the output data obtained is the predicted text annotation corresponding to the speech signal (that is, the preset processing result); based on the predicted text annotation, the cross-entropy criterion is used to optimize the Speech-LLM to obtain the target processing model; the target processing model is fine-tuned by the instruction set to obtain the speech processing model, wherein the instructions corresponding to the instruction set fine-tuning are determined by the task-specific data set corresponding to the speech recognition task. Specifically, the designations corresponding to the above-mentioned instruction set fine-tuning may include: Single Instruction-Multiple Data (SIMD) instructions, floating-point multiply-add instructions (FMA) instructions, etc.

[0134] In an optional embodiment, in step S606, unsupervised training is performed on the initial processing model using the first discretized features and the training text data to obtain a target processing model, including the following method steps:

[0135] Step S661, using the initial processing model to process the first discretized feature and the training text data to obtain predicted training data;

[0136] Step S662: constructing an objective function corresponding to the initial processing model based on the first training data and the predicted training data;

[0137] Step S663: Adjust the initial processing model based on the objective function to obtain a target processing model.

[0138] In an optional embodiment, in step S608, fine-tuning the target processing model using the second training data to obtain a speech processing model includes the following method steps:

[0139] Step S681, discretizing the second speech data to obtain third discretized features of the third speech data;

[0140] Step S682: Processing the third speech data based on the third discretized feature using the target processing model to obtain a training processing result;

[0141] Step S683: Based on the preset processing results and the training processing results, the target processing model is fine-tuned to obtain a speech processing model.

[0142] In summary, the embodiment of the present application proposes that the Speech-LLM framework uses SpeechTokenizer to discretize continuous audio signals, thereby adopting a GPT-based speech processing model for pre-training and data processing. Furthermore, the Speech-LLM obtained by pre-training based on massive voice data and text data will be fine-tuned through instructions related to audio modality, so that Speech-LLM can integrate audio modality-related tasks (for example, recognition, synthesis, translation, enhancement, monitoring, etc.) into the speech processing model.

[0143] It is easy to notice that the above method provided in the embodiment of the present application not only optimizes the existing technology in terms of SpeechTokenizer, but also performs LLM pre-training based on massive audio text data; the speech processing model in the above method is not only oriented to audio generation, but also unifies speech modality-related tasks under the Speech-LLM framework.

[0144] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0145] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0146] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0147] Example 2

[0148] In the operating environment as in Example 1, the present application provides a training method for a speech processing model as shown in Figure 8. Figure 8 is a flow chart of a training method for a speech processing model according to Example 2 of the present application. As shown in Figure 8, the training method for the speech processing model includes:

[0149] Step S81, obtaining first training data and second training data, wherein the first training data includes: first voice data and training text data, and the second training data includes at least: second voice data and a preset processing result;

[0150] Step S82, performing discretization processing on the first speech data to obtain a first discretization feature of the first speech data;

[0151] Step S83, using the first discretized features and the training text data, performing unsupervised training on the initial processing model to obtain a target processing model;

[0152] Step S84: Use the second training data to fine-tune the target processing model to obtain a speech processing model, wherein the speech processing model is used to process the original speech data based on the original discretized features of the original speech data to obtain a target processing result corresponding to the original speech data, the original discretized features are obtained by discretizing the original speech data, the original speech data is the speech data collected by driving the audio acquisition device, and the target processing result includes one of the following: target text data or a speech file containing target speech data.

[0153] Optionally, the above-mentioned speech processing model is a large language model.

[0154] According to the training method of the speech processing model provided in the embodiments of the present application, multiple functional requirements involved in speech processing (for example, recognition, synthesis, translation, enhancement, monitoring, etc.) can be integrated into the speech processing model through the training phase. Therefore, the training method of the speech processing model can be applied to, but not limited to, application scenarios such as human speech recognition, speech synthesis, speech translation, speech enhancement, audio event monitoring, and audio emotion recognition. The implementation of the above method in the above scenarios includes, but is not limited to, applications, functional plug-ins, etc.

[0155] The training method of the speech processing model provided in the embodiment of the present application can be run on a client or a server. The server can be an independent server, a server cluster, or a cloud server. When the server is a cloud server, it can interact with the client in real time through Software as a Service (SaaS). The cloud server obtains the first training data and the second training data provided by the client, obtains the speech processing model using the training method of the speech processing model, and returns the speech processing model to the client.

[0156] According to the training method of the above-mentioned speech processing model provided in the embodiment of the present application, the advantages of using the original discretized features during the training process are: simplifying the model complexity, the use of discretized features can significantly reduce the complexity of the feature space, and reduce the computational complexity and storage requirements of the model; enhancing the robustness of the model, the influence of outliers or noise points in the speech data on the discretized features will be weakened, and the use of discretized features can reduce the sensitivity of the model to outliers and noise; improving the interpretability of the model, the discretized features are easier to understand and explain; the use of discretized features can capture the nonlinear relationship between the original continuous features and the target variables, so that the model can better learn these nonlinear relationships; improving the generalization ability of the model, the discretized features can reduce the overfitting of the model to specific samples, and the use of discretized features can enable the model to better generalize to unseen samples.

[0157] Because large models have stronger expressive power, better generalization ability, higher accuracy, better transfer learning ability and stronger robustness than conventional small models, according to the training method of the above-mentioned speech processing model provided in the embodiment of the present application, during the process of speech data processing, the speech processing model is used to process the original speech data based on the original discretized features of the original speech data to obtain the target processing result, which can extend the advantages of large models to speech data processing scenarios.

[0158] In an application scenario of an embodiment of the present application, through the training method of the above-mentioned speech processing model, a speech processing model for speech processing can be obtained based on the training method of a large language model. Using this speech processing model for speech processing can achieve the technical effect of improving the efficiency, accuracy, generalization ability and robustness of speech data processing, thereby solving the technical problem of low efficiency, low accuracy, weak generalization ability and poor robustness of speech data processing caused by the fact that the existing technology still relies on conventional models for single-person modeling to process speech data.

[0159] In an optional embodiment, in step S82, discretization processing is performed on the first speech data to obtain a first discretized feature of the first speech data, including the following method steps:

[0160] Step S821, encoding the first voice data using a voice encoder to obtain a first voice feature corresponding to the first voice data;

[0161] Step S822: Discretize the first speech feature using a target quantizer to obtain a first discretized feature.

[0162] In an optional embodiment, the training method of the speech processing model further includes the following method steps:

[0163] Step S851, obtaining training speech data;

[0164] Step S852: Encode the training speech data using the initial encoder to obtain training speech features corresponding to the training speech data;

[0165] Step S853, using an initial quantizer to discretize the training speech features to obtain training discretized features;

[0166] Step S854: using the initial decoder to decode the training discretized features to obtain generated speech data;

[0167] Step S855 : Based on the training speech data and the generated speech data, the initial encoder, the initial quantizer, and the initial decoder are trained to obtain a speech encoder, a target quantizer, and a speech decoder.

[0168] In summary, the speech processing model training method proposed in the embodiment of the present application can discretize continuous audio signals using SpeechTokenizer under the Speech-LLM framework, thereby using a GPT-based speech processing model for pre-training. In addition, the speech processing model training method provided in the embodiment of the present application not only optimizes the SpeechTokenizer of the prior art in model training, but also performs LLM pre-training based on massive audio text data; the speech processing model trained by the above method is not only oriented to audio generation, but also unifies tasks related to speech modality under the Speech-LLM framework.

[0169] It should be noted that the preferred implementation of this embodiment can be found in the relevant description in Example 1 and will not be repeated here.

[0170] Example 3

[0171] In the operating environment as in Example 1, the present application provides a method for processing voice data as shown in Figure 9. Figure 9 is a flow chart of a method for processing voice data according to Example 3 of the present application. As shown in Figure 9, the method for processing voice data includes:

[0172] Step S91: driving the audio acquisition device to collect original voice data;

[0173] Step S92, discretizing the original speech data to obtain original discretized features of the original speech data;

[0174] Step S93: Process the original speech data based on the original discretized features using the speech processing model to obtain a target processing result corresponding to the original speech data, wherein the target processing result includes one of the following: target text data, target speech data.

[0175] Optionally, the above-mentioned speech processing model is a large language model.

[0176] The voice data processing method provided in the embodiments of the present application can be applied to, but is not limited to, application scenarios involving voice data processing in e-commerce, education, healthcare, conferencing, social networking, financial products, logistics, and navigation. The audio acquisition device can be a hardware device associated with a terminal device running the voice data processing method, such as a microphone or recorder; it can also be an audio interface or application software that interacts with the running process of the voice data processing method.

[0177] According to the above method provided in the embodiment of the present application, in the process of speech data processing, the advantages of using the original discretized features are: simplifying the model complexity, using discretized features can significantly reduce the complexity of the feature space, reducing the computational complexity and storage requirements of the model; enhancing the robustness of the model, the influence of outliers or noise points in the speech data on the discretized features will be weakened, and using discretized features can reduce the sensitivity of the model to outliers and noise; improving the interpretability of the model, the discretized features are easier to understand and explain; using discretized features can capture the nonlinear relationship between the original continuous features and the target variables, so that the model can better learn these nonlinear relationships; improving the generalization ability of the model, discretized features can reduce the overfitting of the model to specific samples, and using discretized features can make the model better generalize to unseen samples.

[0178] Since the processing model has stronger expressive power, better generalization ability, higher accuracy, better transfer learning ability and stronger robustness than conventional small models, according to the above method provided in the embodiment of the present application, in the process of speech data processing, the speech processing model is used to process the original speech data based on the original discretized features of the original speech data to obtain the target processing result, which can extend the advantages of the processing model to the speech data processing scenario.

[0179] In the embodiment of the present application, the above-mentioned method for processing speech data achieves the technical effect of improving the efficiency, accuracy, generalization ability and robustness of speech data processing, thereby solving the technical problem of low efficiency, low accuracy, weak generalization ability and poor robustness of speech data processing caused by the fact that the existing technology still relies on conventional models for single-person modeling to process speech data.

[0180] It should be noted that the preferred implementation of this embodiment can be found in the relevant descriptions in Examples 1 and 2, and will not be repeated here.

[0181] Example 4

[0182] In the operating environment as in Example 1, the present application provides another method for processing voice data as shown in Figure 10. Figure 10 is a flow chart of a method for processing voice data according to Example 4 of the present application. As shown in Figure 10, the method for processing voice data includes:

[0183] Step S1001: acquiring original voice data by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter is the original voice data, and the original voice data is voice data collected by driving an audio collection device;

[0184] Step S1002, discretizing the original speech data to obtain original discretized features of the original speech data;

[0185] Step S1003: Processing the original speech data based on the original discretized features using the speech processing model to obtain a target processing result corresponding to the original speech data, wherein the target processing result includes one of the following: target text data, target speech data;

[0186] Step S1004: output the target processing result by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the target processing result.

[0187] Optionally, the above-mentioned speech processing model is a large language model.

[0188] In the above-mentioned method for processing voice data provided in an embodiment of the present application, the above-mentioned first interface is a preset voice acquisition interface or an audio acquisition interface. The above-mentioned second interface is a preset voice output interface or an audio output interface. The terminal device running the above-mentioned method for processing voice data interacts with another device or client through the above-mentioned first interface and second interface, and the terminal device provides voice data processing services for the other device or client. For example, the above-mentioned method for processing voice data runs on a cloud server, and the above-mentioned first interface and second interface are the service uplink network interface and service downlink network interface of Software as a Service (SaaS) between the cloud server and the client.

[0189] According to the above method provided in the embodiment of the present application, in the process of speech data processing, the advantages of using the original discretized features are: simplifying the model complexity, using discretized features can significantly reduce the complexity of the feature space, reducing the computational complexity and storage requirements of the model; enhancing the robustness of the model, the influence of outliers or noise points in the speech data on the discretized features will be weakened, and using discretized features can reduce the sensitivity of the model to outliers and noise; improving the interpretability of the model, the discretized features are easier to understand and explain; using discretized features can capture the nonlinear relationship between the original continuous features and the target variables, so that the model can better learn these nonlinear relationships; improving the generalization ability of the model, discretized features can reduce the overfitting of the model to specific samples, and using discretized features can make the model better generalize to unseen samples.

[0190] Because large models have stronger expressive power, better generalization ability, higher accuracy, better transfer learning ability, and stronger robustness compared to conventional small models, according to the above method provided in the embodiment of the present application, during the process of speech data processing, the speech processing model is used to process the original speech data based on the original discretized features of the original speech data to obtain the target processing result, which can extend the advantages of large models to speech data processing scenarios.

[0191] In the embodiment of the present application, the above-mentioned method for processing speech data achieves the technical effect of improving the efficiency, accuracy, generalization ability and robustness of speech data processing, thereby solving the technical problem of low efficiency, low accuracy, weak generalization ability and poor robustness of speech data processing caused by the fact that the existing technology still relies on conventional models for single-person modeling to process speech data.

[0192] It should be noted that the preferred implementation of this embodiment can be found in the relevant descriptions in Examples 1, 2, and 3, and will not be repeated here.

[0193] Example 5

[0194] In the operating environment as in Example 1, the present application provides another method for processing voice data as shown in Figure 11. Figure 11 is a flow chart of a method for processing voice data according to Example 5 of the present application. As shown in Figure 11, the method for processing voice data includes:

[0195] Step S1101: The cloud server receives original voice data sent by the client. The original voice data is the voice data collected by the audio collection device.

[0196] Step S1102: The cloud server discretizes the original speech data to obtain original discretized features of the original speech data.

[0197] Step S1103: The cloud server processes the original speech data using the speech processing model based on the original discretized features to obtain a target processing result corresponding to the original speech data. The target processing result includes one of the following: target text data, target speech data;

[0198] In step S1104 , the cloud server sends the target processing result to the client, and the target processing result is output by the client.

[0199] Optionally, the above-mentioned speech processing model is a large language model.

[0200] Optionally, Figure 12 is a schematic diagram of voice data processing on a cloud server according to Example 5 of the present application. As shown in Figure 12, the client uploads the original voice data to the cloud server, and the original voice data is the voice data collected by driving the audio acquisition device; then, the cloud server discretizes the original voice data to obtain the original discrete features of the original voice data; further, the cloud server uses the voice processing model to process the original voice data based on the original discrete features to obtain the target processing result corresponding to the original voice data, and the target processing result includes one of the following: target text data, target voice data; the cloud server will feedback the target processing result to the above-mentioned client, and the final target processing result will be provided to the user through the client's graphical user interface or audio output device.

[0201] It should be noted that the above-mentioned voice data processing method provided in the embodiment of the present invention can be applied to, but is not limited to, actual application scenarios involving voice data processing (such as voice recognition, voice synthesis, voice translation, voice enhancement, voice event monitoring, etc.) in the fields of e-commerce, education, medical care, conferences, social networks, financial products, logistics and navigation. Through the interaction between the SaaS server and the client, the SaaS server discretizes the original voice data to obtain the original discretized features of the original voice data, and uses the voice processing model to process the original voice data based on the original discretized features to obtain the target processing result corresponding to the original voice data, and the returned target processing result is provided to the user through the client.

[0202] In an embodiment of the present application, a system consisting of a client device and a server can perform the following steps: the client device executes to obtain a voice input instruction through an operation interface and sends the voice input instruction to the server, and receives a target processing result returned by the server and provides the target processing result to the user; the server executes to receive the voice input instruction sent by the client, obtains a target processing result based on the voice data instruction using the voice data processing method provided in this application, and returns the voice processing result to the client. It should be noted that if the operating resources of the client device can meet the deployment and operation conditions of the processing model, the embodiment of the present application can be performed in the client device.

[0203] In the embodiment of the present application, the above-mentioned method for processing speech data achieves the technical effect of improving the efficiency, accuracy, generalization ability and robustness of speech data processing, thereby solving the technical problem of low efficiency, low accuracy, weak generalization ability and poor robustness of speech data processing caused by the fact that the existing technology still relies on conventional models for single-person modeling to process speech data.

[0204] It should be noted that the preferred implementation of this embodiment can be found in the relevant descriptions in Examples 1, 2, and 3, and will not be repeated here.

[0205] Example 6

[0206] According to an embodiment of the present application, a device embodiment for implementing the method for processing voice data in the above-mentioned embodiment 1 is also provided. FIG13 is a schematic structural diagram of a voice data processing device according to embodiment 6 of the present application. As shown in FIG13 , the device includes:

[0207] The first response module 1301 is configured to respond to a voice input command applied to the operation interface and display a voice file containing original voice data on the operation interface, wherein the original voice data is voice data collected by driving the audio collection device;

[0208] The second response module 1302 is used to respond to the voice processing instruction acting on the operation interface and display the target processing result corresponding to the original voice data on the operation interface, wherein the target processing result includes one of the following: target text data, a voice file containing target voice data, and the target processing result is obtained by processing the original voice data based on the original discretized features of the original voice data using the voice processing model, and the original discretized features are obtained by discretizing the original voice data.

[0209] Optionally, Figure 14 is a structural diagram of an optional speech data processing device according to Example 6 of the present application. As shown in Figure 14, in addition to all the modules shown in Figure 13, the device also includes: a processing module 1303, which is used to discretize the original speech data; the processing module 1303 is also used to: encode the original speech data using a speech encoder to obtain the original speech features corresponding to the original speech data; and discretize the original speech features using a target quantizer to obtain original discretized features.

[0210] Optionally, Figure 15 is a structural diagram of another optional speech data processing device according to Example 6 of the present application. As shown in Figure 15, in addition to all the modules shown in Figure 14, the device also includes: a training module 1304, used to obtain training speech data; use the initial encoder to encode the training speech data to obtain training speech features corresponding to the training speech data; use the initial quantizer to discretize the training speech features to obtain training discretized features; use the initial decoder to decode the training discretized features to obtain generated speech data; based on the training speech data and the generated speech data, the initial encoder, initial quantizer and initial decoder are trained to obtain a speech encoder, a target quantizer and a speech decoder.

[0211] Optionally, the training module 1304 is further configured to: discriminate the first speech data and the generated speech data using a discriminator to obtain a discrimination result; and adjust parameters of the initial encoder, initial quantizer, and initial decoder based on the discrimination result to obtain a speech encoder, a target quantizer, and a speech decoder.

[0212] Optionally, when the target processing result includes: target text data, the above-mentioned training module 1304 is also used to: use the speech processing model to extract based on the original discretized features to obtain the original semantic features corresponding to the original speech data; use the text decoder to decode the original semantic features to obtain the target text data.

[0213] Optionally, when the target processing result includes: a voice file containing target voice data, the above-mentioned training module 1304 is also used to: use the voice processing model to extract based on the original discretized features to obtain the original semantic features corresponding to the original voice data; use the voice decoder to decode the original semantic features to obtain the target voice data.

[0214] Optionally, when the target processing result includes: a voice file containing target voice data, the above-mentioned training module 1304 is also used to: respond to text input instructions on the operation interface, and display the original text data on the operation interface; use a text encoder to encode the original text data to obtain the original text features corresponding to the original text data; use a voice processing model to extract based on the original discretized features and the original text features to obtain the target semantic features; use a voice decoder to decode the target semantic features to obtain the target voice data.

[0215] Optionally, the above-mentioned training module 1304 is also used to: obtain first training data and second training data, wherein the first training data includes: first voice data and training text data, and the second training data at least includes: second voice data and preset processing results; discretize the first voice data to obtain the first discretized feature of the first voice data; use the first discretized feature and training text data to perform unsupervised training on the initial processing model to obtain the target processing model; use the second training data to fine-tune the target processing model to obtain a voice processing model.

[0216] Optionally, the above-mentioned training module 1304 is also used to: use the initial processing model to process the first discretized feature and training text data to obtain predicted training data; construct an objective function corresponding to the initial processing model based on the first training data and the predicted training data; adjust the initial processing model based on the objective function to obtain a target processing model.

[0217] Optionally, the above-mentioned training module 1304 is also used to: discretize the second speech data to obtain a third discretized feature of the third speech data; use the target processing model to process the third speech data based on the third discretized feature to obtain a training processing result; based on the preset processing result and the training processing result, fine-tune the target processing model to obtain a speech processing model.

[0218] In an embodiment of the present application, based on voice processing technology, in the field of voice, through the first response module 1301, in response to the voice input instruction acting on the operation interface, a voice file containing original voice data is displayed on the operation interface, wherein the original voice data is the voice data collected by driving the audio acquisition device, and further through the second response module 1302, in response to the voice processing instruction acting on the operation interface, the target processing result corresponding to the original voice data is displayed on the operation interface, wherein the target processing result includes one of the following: target text data, a voice file containing target voice data, and the target processing result is obtained by processing the original voice data based on the original discretized features of the original voice data using the voice processing model, and the original discretized features are obtained by discretizing the original voice data.

[0219] It is easy to notice that, since the above-mentioned original discretized features can characterize the features of the original speech data with discretization advantages such as lower complexity and higher robustness, the target processing results obtained by processing the speech processing model based on the original discretized features can extend the advantages of the processing model to the speech data processing scenario. Therefore, the embodiment of the present application achieves the purpose of processing speech data using the speech processing model, thereby achieving the technical effect of improving the efficiency, accuracy, generalization ability and robustness of speech data processing, and thus solves the technical problem of low efficiency, low accuracy, weak generalization ability and poor robustness of speech data processing caused by the fact that the existing technology still relies on conventional models for single-person modeling to process speech data.

[0220] It should be noted that the first response module 1301 and the second response module 1302 correspond to steps S41 to S42 in Example 1. The examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the above modules or units can be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules can also be run as part of the device in the computer terminal 10 provided in Example 1.

[0221] According to an embodiment of the present application, another device embodiment for implementing the training method of the speech processing model in the above-mentioned embodiment 2 is also provided. Figure 16 is a structural schematic diagram of a training device for a speech processing model according to Example 6 of the present application. As shown in Figure 16, the device includes: an acquisition module 1601, used to acquire first training data and second training data, wherein the first training data includes: first speech data and training text data, and the second training data includes at least: second speech data and a preset processing result; a processing module 1602, used to discretize the first speech data to obtain a first discretized feature of the first speech data; a training module 1603, used to perform unsupervised training on the initial processing model using the first discretized feature and the training text data to obtain a target processing model; an adjustment module 1604, used to fine-tune the target processing model using the second training data to obtain a speech processing model, wherein the speech processing model is used to process the original speech data based on the original discretized features of the original speech data to obtain a target processing result corresponding to the original speech data, the original discretized features are obtained by discretizing the original speech data, the original speech data is speech data collected by driving the audio acquisition device, and the target processing result includes one of the following: target text data or a speech file containing target speech data.

[0222] Optionally, the processing module 1602 is further configured to: encode the first speech data using a speech encoder to obtain a first speech feature corresponding to the first speech data; and discretize the first speech feature using a target quantizer to obtain a first discretized feature.

[0223] Optionally, the training device of the above-mentioned speech processing model also includes: a generation module (not shown in the figure) for obtaining training speech data; using an initial encoder to encode the training speech data to obtain training speech features corresponding to the training speech data; using an initial quantizer to discretize the training speech features to obtain training discretized features; using an initial decoder to decode the training discretized features to obtain generated speech data; based on the training speech data and the generated speech data, the initial encoder, initial quantizer and initial decoder are trained to obtain a speech encoder, a target quantizer and a speech decoder.

[0224] It should be noted that the acquisition module 1601, processing module 1602, training module 1603, and adjustment module 1604 correspond to steps S81 to S84 in Example 2. The examples and application scenarios implemented by the four modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 2. It should be noted that the modules or units can be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The modules can also be part of the device and can be run in the computer terminal 10 provided in Example 1.

[0225] According to an embodiment of the present application, another embodiment of a device for implementing the method for processing voice data in the above-mentioned embodiment 3 is also provided. FIG17 is a schematic structural diagram of another device for processing voice data according to embodiment 6 of the present application. As shown in FIG17 , the device includes:

[0226] Driver module 1701 is used to drive the audio acquisition device to collect original voice data;

[0227] The first processing module 1702 is used to discretize the original speech data to obtain original discretized features of the original speech data;

[0228] The second processing module 1703 is used to process the original speech data based on the original discretized features using the speech processing model to obtain a target processing result corresponding to the original speech data, wherein the target processing result includes one of the following: target text data, target speech data.

[0229] It should be noted that the above-mentioned driving module 1701, first processing module 1702, and second processing module 1703 correspond to steps S91 to S93 in Example 3. The examples and application scenarios implemented by the three modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Example 3. It should be noted that the above-mentioned modules or units can be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above-mentioned modules can also be run as part of the device in the computer terminal 10 provided in Example 1.

[0230] According to an embodiment of the present application, another device embodiment for implementing the method for processing voice data in the above-mentioned embodiment 4 is also provided. Figure 18 is a structural schematic diagram of another voice data processing device according to embodiment 6 of the present application. As shown in Figure 18, the device includes: an acquisition module 1801, which is used to obtain original voice data by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter is the original voice data, and the original voice data is the voice data collected by driving the audio acquisition device; a first processing module 1802, which is used to discretize the original voice data to obtain the original discretized features of the original voice data; a second processing module 1803, which is used to process the original voice data based on the original discretized features using a voice processing model to obtain a target processing result corresponding to the original voice data, wherein the target processing result includes one of the following: target text data, target voice data; an output module 1804, which is used to output the target processing result by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the target processing result.

[0231] It should be noted that the acquisition module 1801, the first processing module 1802, the second processing module 1803, and the output module 1804 correspond to steps S1001 to S1004 in Example 4. The examples and application scenarios implemented by the four modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Example 4. It should be noted that the above-mentioned modules or units can be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above-mentioned modules can also be part of the device and can be run in the computer terminal 10 provided in Example 1.

[0232] According to an embodiment of the present application, another device embodiment for implementing the method for processing voice data in the above-mentioned embodiment 5 is also provided. Figure 19 is a structural schematic diagram of another voice data processing device according to embodiment 6 of the present application. As shown in Figure 19, the device includes: a receiving module 1901, which is used for the cloud server to receive the original voice data sent by the client, wherein the original voice data is the voice data collected by driving the audio acquisition device; a first processing module 1902, which is used for the cloud server to discretize the original voice data to obtain the original discretized features of the original voice data; a second processing module 1903, which is used for the cloud server to process the original voice data based on the original discretized features using a voice processing model to obtain a target processing result corresponding to the original voice data, wherein the target processing result includes one of the following: target text data, target voice data; a sending module 1904, which is used for the cloud server to send the target processing result to the client, wherein the target processing result is output by the client.

[0233] It should be noted that the receiving module 1901, the first processing module 1902, the second processing module 1903, and the sending module 1904 correspond to steps S1101 to S1104 in Example 5. The examples and application scenarios implemented by the four modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 5. It should be noted that the modules or units can be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The modules can also be part of the device and can be run in the computer terminal 10 provided in Example 1.

[0234] It should be noted that the preferred implementation of this embodiment can be found in the relevant descriptions in Examples 1 to 5, and will not be repeated here.

[0235] Example 7

[0236] According to an embodiment of the present application, a computer terminal is further provided, which can be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal can also be replaced by a terminal device such as a mobile terminal.

[0237] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of a computer network.

[0238] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the voice data processing method: responding to the voice input instruction on the operation interface, displaying the voice file containing the original voice data on the operation interface, wherein the original voice data is the voice data collected by driving the audio acquisition device; responding to the voice processing instruction on the operation interface, displaying the target processing result corresponding to the original voice data on the operation interface, wherein the target processing result includes one of the following: target text data, a voice file containing the target voice data, and the target processing result is obtained by processing the original voice data based on the original discretized features of the original voice data using the voice processing model, and the original discretized features are obtained by discretizing the original voice data.

[0239] Optionally, Figure 20 is a structural block diagram of a computer terminal according to Example 7 of the present application. As shown in Figure 20, the computer terminal 200 may include: one or more (only one is shown in the figure) processors 2002, a memory 2004, a storage controller 2006, and a peripheral interface 2008, wherein the peripheral interface 2008 is connected to the radio frequency module, the audio module and the display.

[0240] Among them, the memory 2004 can be used to store software programs and modules, such as the program instructions / modules corresponding to the voice data processing method and device in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned voice data processing method. The memory 2004 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 2004 may further include a memory remotely located relative to the processor, and these remote memories can be connected to the computer terminal 200 via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0241] The processor 2002 can call the information and application stored in the memory through the transmission device to perform the following steps: responding to the voice input instruction on the operation interface, displaying the voice file containing the original voice data on the operation interface, wherein the original voice data is the voice data collected by driving the audio acquisition device; responding to the voice processing instruction on the operation interface, displaying the target processing result corresponding to the original voice data on the operation interface, wherein the target processing result includes one of the following: target text data, a voice file containing the target voice data, and the target processing result is obtained by processing the original voice data based on the original discretized features of the original voice data using the voice processing model, and the original discretized features are obtained by discretizing the original voice data.

[0242] Optionally, the processor 2002 may also execute the program code of the following steps: encoding the original speech data using a speech encoder to obtain original speech features corresponding to the original speech data; and discretizing the original speech features using a target quantizer to obtain original discretized features.

[0243] Optionally, the processor 2002 may also execute the program code of the following steps: obtaining training speech data; encoding the training speech data using an initial encoder to obtain training speech features corresponding to the training speech data; discretizing the training speech features using an initial quantizer to obtain training discretized features; decoding the training discretized features using an initial decoder to obtain generated speech data; and training the initial encoder, initial quantizer, and initial decoder based on the training speech data and the generated speech data to obtain a speech encoder, a target quantizer, and a speech decoder.

[0244] Optionally, the processor 2002 may also execute the program code of the following steps: using a discriminator to discriminate the first speech data and the generated speech data to obtain a discrimination result; and adjusting the parameters of the initial encoder, the initial quantizer, and the initial decoder based on the discrimination result to obtain a speech encoder, a target quantizer, and a speech decoder.

[0245] Optionally, the processor 2002 may also execute the program code of the following steps: extracting based on the original discretized features using a speech processing model to obtain original semantic features corresponding to the original speech data; and decoding the original semantic features using a text decoder to obtain target text data.

[0246] Optionally, the processor 2002 may also execute the program code of the following steps: extracting based on the original discretized features using a speech processing model to obtain original semantic features corresponding to the original speech data; and decoding the original semantic features using a speech decoder to obtain target speech data.

[0247] Optionally, the processor 2002 may also execute the program code of the following steps: responding to a text input instruction on the operation interface, displaying the original text data on the operation interface; encoding the original text data using a text encoder to obtain original text features corresponding to the original text data; extracting target semantic features based on the original discretized features and the original text features using a speech processing model; and decoding the target semantic features using a speech decoder to obtain target speech data.

[0248] Optionally, the processor 2002 may also execute the program code of the following steps: obtaining first training data and second training data, wherein the first training data includes: first voice data and training text data, and the second training data includes at least: second voice data and a preset processing result; discretizing the first voice data to obtain a first discretized feature of the first voice data; performing unsupervised training on the initial processing model using the first discretized feature and the training text data to obtain a target processing model; and fine-tuning the target processing model using the second training data to obtain a voice processing model.

[0249] Optionally, the processor 2002 may also execute the program code of the following steps: using the initial processing model to process the first discretized feature and the training text data to obtain predicted training data; constructing an objective function corresponding to the initial processing model based on the first training data and the predicted training data; and adjusting the initial processing model based on the objective function to obtain a target processing model.

[0250] Optionally, the processor 2002 may also execute the program code of the following steps: discretizing the second voice data to obtain a third discretized feature of the third voice data; processing the third voice data based on the third discretized feature using the target processing model to obtain a training processing result; and fine-tuning the target processing model based on the preset processing result and the training processing result to obtain a voice processing model.

[0251] The processor 2002 can call the information and application stored in the memory through the transmission device to perform the following steps: obtaining first training data and second training data, wherein the first training data includes: first voice data and training text data, and the second training data at least includes: second voice data and a preset processing result; discretizing the first voice data to obtain a first discretized feature of the first voice data; using the first discretized feature and the training text data, performing unsupervised training on the initial processing model to obtain a target processing model; using the second training data to fine-tune the target processing model to obtain a voice processing model, wherein the voice processing model is used to process the original voice data based on the original discretized feature of the original voice data to obtain a target processing result corresponding to the original voice data, the original discretized feature is obtained by discretizing the original voice data, and the original voice data is the voice data collected by driving the audio collection device, and the target processing result includes one of the following: target text data or a voice file containing target voice data.

[0252] Optionally, the processor 2002 may also execute the program code of the following steps: encoding the first speech data using a speech encoder to obtain a first speech feature corresponding to the first speech data; and discretizing the first speech feature using a target quantizer to obtain a first discretized feature.

[0253] Optionally, the processor 2002 may also execute the program code of the following steps: obtaining training speech data; encoding the training speech data using an initial encoder to obtain training speech features corresponding to the training speech data; discretizing the training speech features using an initial quantizer to obtain training discretized features; decoding the training discretized features using an initial decoder to obtain generated speech data; and training the initial encoder, initial quantizer, and initial decoder based on the training speech data and the generated speech data to obtain a speech encoder, a target quantizer, and a speech decoder.

[0254] The processor 2002 can call the information and application stored in the memory through the transmission device to execute the following steps: drive the audio acquisition device to collect original voice data; discretize the original voice data to obtain the original discretized features of the original voice data; use the voice processing model to process the original voice data based on the original discretized features to obtain the target processing result corresponding to the original voice data, wherein the target processing result includes one of the following: target text data, target voice data.

[0255] The processor 2002 can call the information and application stored in the memory through the transmission device to perform the following steps: obtain the original voice data by calling the first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter is the original voice data, and the original voice data is the voice data collected by driving the audio collection device; discretize the original voice data to obtain the original discretized features of the original voice data; use the voice processing model to process the original voice data based on the original discretized features to obtain the target processing result corresponding to the original voice data, wherein the target processing result includes one of the following: target text data, target voice data; output the target processing result by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the target processing result.

[0256] The processor 2002 can call the information and application stored in the memory through the transmission device to perform the following steps: the cloud server receives the original voice data sent by the client, wherein the original voice data is the voice data collected by driving the audio collection device; the cloud server discretizes the original voice data to obtain the original discrete features of the original voice data; the cloud server uses the voice processing model to process the original voice data based on the original discrete features to obtain the target processing result corresponding to the original voice data, wherein the target processing result includes one of the following: target text data, target voice data; the cloud server sends the target processing result to the client, wherein the target processing result is output by the client.

[0257] According to an embodiment of the present application, a solution for a computer terminal for processing voice data is provided. Based on voice processing technology, in the field of voice, by responding to voice input instructions on an operation interface, a voice file containing original voice data is displayed on the operation interface, wherein the original voice data is voice data collected by driving an audio acquisition device. Further, by responding to the voice processing instructions on the operation interface, a target processing result corresponding to the original voice data is displayed on the operation interface, wherein the target processing result includes one of the following: target text data, a voice file containing target voice data, and the target processing result is obtained by processing the original voice data using a voice processing model based on the original discretized features of the original voice data, wherein the original discretized features are obtained by discretizing the original voice data.

[0258] It is easy to notice that, since the above-mentioned original discretized features can characterize the features of the original speech data with discretization advantages such as lower complexity and higher robustness, the target processing results obtained by processing the speech processing model based on the original discretized features can extend the advantages of the processing model to the speech data processing scenario. Therefore, the embodiment of the present application achieves the purpose of processing speech data using the speech processing model, thereby achieving the technical effect of improving the efficiency, accuracy, generalization ability and robustness of speech data processing, and thus solves the technical problem of low efficiency, low accuracy, weak generalization ability and poor robustness of speech data processing caused by the fact that the existing technology still relies on conventional models for single-person modeling to process speech data.

[0259] Those skilled in the art will appreciate that the structure shown in FIG20 is merely illustrative, and the computer terminal may also be a terminal device such as a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, or a mobile internet device (MID). FIG20 does not limit the structure of the computer terminal. For example, the computer terminal 200 may include more or fewer components (such as a network interface, a display device, etc.) than those shown in FIG20 , or may have a configuration different from that shown in FIG20 .

[0260] A person skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which may include: a flash drive, ROM, RAM, a magnetic disk or an optical disk, etc.

[0261] Example 8

[0262] According to an embodiment of the present application, a computer-readable storage medium is further provided. Optionally, in this embodiment, the storage medium can be used to store program codes executed by any of the above methods.

[0263] Optionally, in this embodiment, the above-mentioned storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0264] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: in response to a voice input instruction acting on an operation interface, displaying a voice file containing original voice data on the operation interface, wherein the original voice data is voice data collected by driving an audio acquisition device; in response to a voice processing instruction acting on the operation interface, displaying a target processing result corresponding to the original voice data on the operation interface, wherein the target processing result includes one of the following: target text data, a voice file containing target voice data, and the target processing result is obtained by processing the original voice data based on the original discretized features of the original voice data using a voice processing model, and the original discretized features are obtained by discretizing the original voice data.

[0265] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: encoding the original speech data using a speech encoder to obtain original speech features corresponding to the original speech data; and discretizing the original speech features using a target quantizer to obtain original discretized features.

[0266] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: obtaining training speech data; encoding the training speech data using an initial encoder to obtain training speech features corresponding to the training speech data; discretizing the training speech features using an initial quantizer to obtain training discretized features; decoding the training discretized features using an initial decoder to obtain generated speech data; training the initial encoder, initial quantizer and initial decoder based on the training speech data and the generated speech data to obtain a speech encoder, a target quantizer and a speech decoder.

[0267] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: using a discriminator to discriminate the first speech data and the generated speech data to obtain a discrimination result; and adjusting the parameters of the initial encoder, the initial quantizer, and the initial decoder based on the discrimination result to obtain a speech encoder, a target quantizer, and a speech decoder.

[0268] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: extracting based on the original discretized features using a speech processing model to obtain original semantic features corresponding to the original speech data; and decoding the original semantic features using a text decoder to obtain target text data.

[0269] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: extracting based on the original discretized features using a speech processing model to obtain original semantic features corresponding to the original speech data; and decoding the original semantic features using a speech decoder to obtain target speech data.

[0270] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: displaying original text data on the operation interface in response to a text input instruction acting on the operation interface; encoding the original text data using a text encoder to obtain original text features corresponding to the original text data; extracting target semantic features based on the original discretized features and the original text features using a speech processing model; and decoding the target semantic features using a speech decoder to obtain target speech data.

[0271] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: obtaining first training data and second training data, wherein the first training data includes: first voice data and training text data, and the second training data includes at least: second voice data and a preset processing result; discretizing the first voice data to obtain a first discretized feature of the first voice data; performing unsupervised training on the initial processing model using the first discretized feature and the training text data to obtain a target processing model; and fine-tuning the target processing model using the second training data to obtain a voice processing model.

[0272] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: using the initial processing model to process the first discretized feature and the training text data to obtain predicted training data; constructing an objective function corresponding to the initial processing model based on the first training data and the predicted training data; and adjusting the initial processing model based on the objective function to obtain a target processing model.

[0273] Optionally, the processor 1502 may also execute the following program code: discretize the second voice data to obtain a third discretized feature of the third voice data; process the third voice data based on the third discretized feature using the target processing model to obtain a training processing result; and fine-tune the target processing model based on the preset processing result and the training processing result to obtain a voice processing model.

[0274] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: obtaining first training data and second training data, wherein the first training data includes: first voice data and training text data, and the second training data includes at least: second voice data and a preset processing result; discretizing the first voice data to obtain a first discretized feature of the first voice data; performing unsupervised training on the initial processing model using the first discretized feature and the training text data to obtain a target processing model; fine-tuning the target processing model using the second training data to obtain a voice processing model, wherein the voice processing model is used to process the original voice data based on the original discretized feature of the original voice data to obtain a target processing result corresponding to the original voice data, the original discretized feature is obtained by discretizing the original voice data, and the original voice data is the voice data collected by driving the audio acquisition device, and the target processing result includes one of the following: target text data or a voice file containing target voice data.

[0275] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: encoding the first speech data using a speech encoder to obtain a first speech feature corresponding to the first speech data; and discretizing the first speech feature using a target quantizer to obtain a first discretized feature.

[0276] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: obtaining training speech data; encoding the training speech data using an initial encoder to obtain training speech features corresponding to the training speech data; discretizing the training speech features using an initial quantizer to obtain training discretized features; decoding the training discretized features using an initial decoder to obtain generated speech data; training the initial encoder, initial quantizer and initial decoder based on the training speech data and the generated speech data to obtain a speech encoder, a target quantizer and a speech decoder.

[0277] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: driving the audio acquisition device to acquire original voice data; discretizing the original voice data to obtain original discretized features of the original voice data; processing the original voice data based on the original discretized features using a voice processing model to obtain a target processing result corresponding to the original voice data, wherein the target processing result includes one of the following: target text data, target voice data.

[0278] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: obtaining original voice data by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter is the original voice data, and the original voice data is the voice data collected by driving the audio acquisition device; discretizing the original voice data to obtain the original discretized features of the original voice data; processing the original voice data based on the original discretized features using a voice processing model to obtain a target processing result corresponding to the original voice data, wherein the target processing result includes one of the following: target text data, target voice data; outputting the target processing result by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the target processing result.

[0279] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: the cloud server receives original voice data sent by the client, wherein the original voice data is voice data collected by driving the audio acquisition device; the cloud server discretizes the original voice data to obtain original discrete features of the original voice data; the cloud server uses a voice processing model to process the original voice data based on the original discrete features to obtain a target processing result corresponding to the original voice data, wherein the target processing result includes one of the following: target text data, target voice data; the cloud server sends the target processing result to the client, wherein the target processing result is output by the client.

[0280] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0281] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0282] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0283] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0284] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0285] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, ROM, RAM, mobile hard drives, magnetic disks or optical disks.

[0286] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for processing voice data, comprising: In response to a voice input command acting on the operation interface, a voice file containing original voice data is displayed on the operation interface, wherein the original voice data is voice data collected by driving an audio collection device; In response to the voice processing instruction acting on the operation interface, the target processing result corresponding to the original voice data is displayed on the operation interface, and the target processing result includes one of the following: target text data, a voice file containing target voice data, and the target processing result is obtained by processing the original voice data based on the original discretized features of the original voice data using a voice processing model, and the original discretized features are obtained by discretizing the original voice data.

2. The method according to claim 1, wherein: Discretization processing is performed on the original voice data, including: Encoding the original speech data using a speech encoder to obtain original speech features corresponding to the original speech data; The original speech features are discretized using a target quantizer to obtain the original discretized features.

3. The method according to claim 2, wherein: The method further comprises: Get training speech data; Encoding the training speech data using an initial encoder to obtain training speech features corresponding to the training speech data; Discretizing the training speech features using an initial quantizer to obtain training discretized features; Decoding the training discretized features using an initial decoder to generate speech data; Based on the training speech data and the generated speech data, the initial encoder, the initial quantizer and the initial decoder are trained to obtain the speech encoder, the target quantizer and the speech decoder.

4. The method according to claim 1, wherein: In the case where the target processing result includes: the target text data, the original speech data is processed based on the original discretized features of the original speech data using a speech processing model, including: Extracting the original discretized features using the speech processing model to obtain original semantic features corresponding to the original speech data; The original semantic features are decoded using a text decoder to obtain the target text data.

5. The method according to claim 1, wherein: In the case where the target processing result includes: a voice file including the target voice data, processing the original voice data based on the original discretized features of the original voice data using a voice processing model includes: Extracting the original discretized features using the speech processing model to obtain original semantic features corresponding to the original speech data; The original semantic features are decoded by using a speech decoder to obtain the target speech data.

6. The method according to claim 1, wherein: In the case where the target processing result includes: a voice file including the target voice data, processing the original voice data based on the original discretized features of the original voice data using a voice processing model includes: In response to a text input instruction acting on the operation interface, displaying original text data on the operation interface; Encoding the original text data using a text encoder to obtain original text features corresponding to the original text data; Using the speech processing model to extract based on the original discretized features and the original text features, to obtain target semantic features; The target semantic feature is decoded by using a speech decoder to obtain the target speech data.

7. The method according to claim 1, wherein: The method further comprises: Acquire first training data and second training data, wherein the first training data includes: first voice data and training text data, and the second training data includes at least: second voice data and a preset processing result; Performing discretization processing on the first speech data to obtain first discretization features of the first speech data; Using the first discretized features and the training text data, unsupervised training is performed on the initial processing model to obtain a target processing model; The target processing model is fine-tuned using the second training data to obtain the speech processing model.

8. A method for training a speech processing model, comprising: Acquire first training data and second training data, wherein the first training data includes: first voice data and training text data, and the second training data includes at least: second voice data and a preset processing result; Performing discretization processing on the first speech data to obtain first discretization features of the first speech data; Using the first discretized features and the training text data, unsupervised training is performed on the initial processing model to obtain a target processing model; The target processing model is fine-tuned using the second training data to obtain a speech processing model, wherein the speech processing model is configured to process the original speech data based on the original discretized features of the original speech data to obtain a target processing result corresponding to the original speech data, wherein the original discretized features are obtained by discretizing the original speech data, and the original speech data is speech data collected by driving an audio collection device, and the target processing result includes one of the following: target text data or The audio file containing the target audio data.

9. The method according to claim 8, wherein: Discretizing the first speech data to obtain a first discretized feature of the first speech data includes: Encoding the first voice data using a voice encoder to obtain a first voice feature corresponding to the first voice data; The first speech feature is discretized using a target quantizer to obtain the first discretized feature.

10. A method for processing voice data, comprising: The cloud server receives the original voice data sent by the client, wherein the original voice data is the voice data collected by the driving audio collection device; The cloud server discretizes the original voice data to obtain original discretized features of the original voice data; The cloud server processes the original voice data based on the original discretized features using a voice processing model to obtain a target processing result corresponding to the original voice data, wherein the target processing result includes one of the following: target text data, target voice data; The cloud server sends the target processing result to the client, and the target processing result is output by the client.

11. A method for processing voice data, comprising: Drive the audio acquisition device to collect original voice data; Performing discretization processing on the original speech data to obtain original discretized features of the original speech data; The original speech data is processed based on the original discretized features using a speech processing model to obtain a target processing result corresponding to the original speech data, wherein the target processing result includes one of the following: target text data and target speech data.

12. A method for processing voice data, comprising: Acquire original voice data by calling a first interface, wherein the first interface includes a first parameter, and a parameter value of the first parameter is the original voice data, and the original voice data is voice data collected by driving an audio collection device; Performing discretization processing on the original speech data to obtain original discretized features of the original speech data; Processing the original speech data based on the original discretized features using a speech processing model to obtain a target processing result corresponding to the original speech data, wherein the target processing result includes one of the following: target text data, target speech data; The target processing result is output by calling the second interface, wherein the second interface includes a second parameter. The parameter value of the second parameter is the target processing result.

13. An electronic device, comprising: A memory storing an executable program; A processor is configured to run the program, and the program, when running, performs the method according to any one of claims 1 to 12.

14. A computer-readable storage medium, comprising a stored executable program, which controls a device where the computer-readable storage medium is located to execute the method according to any one of claims 1 to 12 when the executable program is executed.

Images