Audio processing method and device, equipment and storage medium
By preprocessing the prompt text and using the key-value (KV) cache mechanism for incremental audio processing, the problems of audio processing delay and resource waste are solved, and more efficient audio processing is achieved.
Patent Information
- Application Number
- CN202410257541.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-06
- Publication Date
- 2025-09-09
AI Technical Summary
In the prior art, there is a long delay and waste of computing resources when processing audio, especially when the prompt text is repeatedly processed.
By preprocessing the prompt text to obtain intermediate features, and then incrementally processing them with the target audio input language model, the key-value KV cache mechanism is used to reduce repeated calculations.
It reduces the audio processing delay, improves processing efficiency, and saves computing resources.
Smart Images
Figure CN120612928A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present disclosure relate to the field of audio processing technology, and in particular to an audio processing method, apparatus, device, and storage medium. Background Art
[0002] With the rapid development of computer technology, large language models (LLMs) have recently gained widespread attention for their ability to more intelligently understand and respond to user questions. Large language models are often used to understand audio, but related audio processing techniques often suffer from long latency. Summary of the Invention
[0003] The embodiments of the present disclosure provide an audio processing method, which can reduce the delay of audio processing and thus improve the efficiency of audio processing.
[0004] In a first aspect, an embodiment of the present disclosure provides an audio processing method, including:
[0005] Obtaining an intermediate feature of the prompt text as a first intermediate feature; wherein the first intermediate feature is obtained by preprocessing the prompt text based on a language model;
[0006] Inputting the first intermediate feature and the collected target audio into the language model, and outputting the intermediate feature corresponding to the target audio as the second intermediate feature; wherein the target audio corresponds to the prompt text, and the first intermediate feature and the second intermediate feature are both cached in a key-value (KV) format;
[0007] The first intermediate feature and the second intermediate feature are input into the language model to generate a processing result corresponding to the target audio.
[0008] In a second aspect, an embodiment of the present disclosure further provides an audio processing device, comprising:
[0009] A first intermediate feature acquisition module is configured to acquire an intermediate feature of the prompt text as a first intermediate feature; wherein the first intermediate feature is obtained by preprocessing the prompt text based on a language model;
[0010] A second intermediate feature acquisition module is configured to input the first intermediate feature and the collected target audio into the language model, and output the intermediate feature corresponding to the target audio as the second intermediate feature; wherein the target audio corresponds to the prompt text, and the first intermediate feature and the second intermediate feature are both cached in a key-value (KV) format;
[0011] An audio processing result generation module is used to input the first intermediate feature and the second intermediate feature into the language model to generate a processing result corresponding to the target audio.
[0012] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:
[0013] one or more processors;
[0014] a storage device for storing one or more programs,
[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement the audio processing method as described in the embodiment of the present disclosure.
[0016] In a fourth aspect, an embodiment of the present disclosure further provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to execute the audio processing method as described in the embodiment of the present disclosure.
[0017] The embodiments of the present disclosure disclose an audio processing method, apparatus, device and storage medium, which obtain the intermediate features of the prompt text as the first intermediate features; wherein the first intermediate features are obtained by pre-processing the prompt text based on a language model; the first intermediate features and the collected target audio are input into the language model, and the intermediate features corresponding to the target audio are output as the second intermediate features; wherein the target audio corresponds to the prompt text, and the first intermediate features and the second intermediate features are cached in the form of key-value KV; the first intermediate features and the second intermediate features are input into the language model to generate the processing results corresponding to the target audio. The audio processing method provided by the embodiments of the present disclosure processes the first intermediate features of the prompt text and the target audio obtained by pre-processing through a language model to obtain the processing results of the target audio. Pre-processing the prompt text can save the time spent in processing the prompt text, reduce the delay of audio processing, and thus improve the efficiency of audio processing. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale.
[0019] Figure 1 is a flowchart of an audio processing method provided by an embodiment of the present disclosure;
[0020] Figure 2 is a structural diagram of an audio processing device provided by an embodiment of the present disclosure;
[0021] Figure 3 It is a structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION
[0022] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.
[0023] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.
[0024] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.
[0025] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0026] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".
[0027] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.
[0028] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.
[0029] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.
[0030] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.
[0031] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.
[0032] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) must comply with the requirements of relevant laws, regulations and relevant provisions.
[0033] In the related art, when processing audio based on a large language model, the prompt text and the audio to be processed are first processed simultaneously. This process often takes tens to hundreds of milliseconds, and this time overhead will significantly increase the delay of audio processing. Moreover, if the prompt texts corresponding to different audios are the same, the prompt texts need to be processed repeatedly when processing each audio, wasting computing resources. In this embodiment, the delay of the audio processing process is reduced by pre-processing the prompt text before the audio collection is completed, and then performing incremental processing on the collected target audio; at the same time, for new audio, if the prompt text corresponding to the audio has been processed in advance, the processing results of the prompt text can be directly used for further audio processing to save computing resources.
[0034] Figure 1 This is a flow chart of an audio processing method provided in an embodiment of the present disclosure. The embodiment of the present disclosure is applicable to situations where audio is processed based on a language model. The method can be performed by an audio processing device, which can be implemented in the form of software and / or hardware. Optionally, it can be implemented by an electronic device, which can be a mobile terminal, PC or server, etc.
[0035] like Figure 1 As shown, the method includes:
[0036] S110: Obtain an intermediate feature of the prompt text as a first intermediate feature.
[0037] The first intermediate feature is obtained by preprocessing the prompt text based on a language model. The language model can be a deep learning model trained on massive amounts of text data, known as a large language model. The machine can understand the meaning of text and handle various natural language tasks. The prompt text can be a prompt entered by the user based on the task requirements. For example, if a user wants to translate an audio clip described in Chinese into English, the prompt input might be "Please translate the following audio clip into English." The prompt text is used to instruct the language model on which task to perform.
[0038] The first intermediate feature is cached in a key-value (KV) format, namely, a KV cache.
[0039] In this embodiment, the language model includes a first attention module, a second attention module, and a third attention module. The first intermediate feature is obtained by processing the first attention module of the language model. The second attention module can be understood as an incremental processing attention module.
[0040] Specifically, the method for obtaining the intermediate features of the prompt text can be: at the starting time of acquisition of the target audio, the intermediate features of the prompt text corresponding to the target audio are obtained from the cache; if the intermediate features of the prompt text corresponding to the target audio are not obtained from the cache, the prompt text corresponding to the target audio is input into the first attention module, and the intermediate features of the prompt text are output.
[0041] In this application scenario, when the user needs to process the audio, he first inputs the prompt text for processing the audio through the terminal device, and then collects the target audio through the terminal device. When it is detected that the terminal device starts to collect the target audio, the intermediate features of the prompt text corresponding to the target audio are obtained from the cache, which can be a cache in the video memory. If the intermediate features of the prompt text corresponding to the target audio are obtained from the cache, it means that the processing task corresponding to the target audio is the same as that of the historical target audio, that is, the prompt text is the same, and the prompt text has been pre-processed by the language model to obtain the intermediate features. If the intermediate features of the prompt text corresponding to the target audio are not obtained from the cache, the prompt text corresponding to the target audio has not been processed or the intermediate features of the corresponding prompt text have been removed from the cache. At this time, the prompt text corresponding to the target audio is input into the first attention module, thereby outputting the intermediate features of the prompt text, that is, the KV cache of the prompt text.
[0042] Among them, the process of inputting the prompt text corresponding to the target audio into the first attention module can be: scheduling the program code corresponding to the first attention module to process the prompt text.
[0043] S120: Input the first intermediate feature and the collected target audio into a language model, and output the intermediate feature corresponding to the target audio as the second intermediate feature.
[0044] The target audio corresponds to the prompt text, and both the first and second intermediate features are cached as key-value pairs. The target audio can be the user's voice or other sound. Once audio is captured, if no further sound is captured after a certain period (e.g., 5 seconds) or if the user actively stops capturing, the target audio is considered complete.
[0045] Specifically, the process of inputting the first intermediate feature and the collected target audio into the language model and outputting the intermediate feature corresponding to the target audio can be: inputting the first intermediate feature and the collected target audio into the second attention module and outputting the intermediate feature corresponding to the target audio.
[0046] The second attention module can be understood as an incremental processing attention module, whose function can be understood as performing incremental processing on the target audio based on the first intermediate features, thereby obtaining the intermediate features of the target audio, i.e., the KV cache of the target audio. The process of inputting the first intermediate features and the collected target audio into the second attention module can be to schedule the program code corresponding to the first attention model to process the first intermediate features and the collected target audio.
[0047] S130: Input the first intermediate feature and the second intermediate feature into a language model to generate a processing result corresponding to the target audio.
[0048] Specifically, the first intermediate feature and the second intermediate feature are input into the language model to generate a processing result corresponding to the target audio. The method can be: the first intermediate feature and the second intermediate feature are input into the third attention module to generate a processing result corresponding to the target audio.
[0049] In this embodiment, the KV cache corresponding to the prompt text and the KV cache corresponding to the target audio are input into the third attention module of the language model to obtain a processing result for the target audio. This processing result can be the text corresponding to the target audio, or a response text to the target audio. The process of inputting the first intermediate feature and the second intermediate feature into the third attention module can include scheduling the program code corresponding to the third attention module to process the first and second intermediate features.
[0050] Optionally, both the first intermediate feature and the second intermediate feature are cached in a video memory in a key-value KV format, where the video memory can be understood as a storage space in a graphics processing unit (GPU).
[0051] The audio processing method provided in this embodiment also includes the following steps: if no new target audio is collected within a set time or the occupancy rate of the video memory exceeds a set threshold, the first intermediate feature is removed from the video memory and the prompt text is cached in the internal memory; or, the first intermediate feature is transferred from the video memory to the internal memory for caching.
[0052] Among them, the memory can be understood as the central processing unit (CPU). The set threshold can be any value between 90% and 100%, for example: 95%. If the new target audio is not collected within the set time, it means that the task of processing the audio required by the user is completed. At this time, the first intermediate feature can be removed from the video memory, and the prompt text can be cached in the memory. Alternatively, the first intermediate feature can be transferred from the video memory to the memory for caching to free up the video memory space. This can not only save video memory space and improve computing power, but also facilitate the subsequent same audio processing tasks without the user having to repeatedly enter the prompt text. If the occupancy rate of the video memory exceeds the set threshold, in order to prevent downtime, the first intermediate feature is removed from the video memory and the prompt text is cached in the memory; alternatively, the first intermediate feature is transferred from the video memory to the memory for caching.
[0053] Optionally, if the first intermediate feature is removed from the video memory and the prompt text is cached in the memory, when a new target audio is collected, the prompt text and the target audio are input into the first attention module, and the first intermediate feature of the prompt text and the second intermediate feature of the new target audio are output.
[0054] In this embodiment, since the first intermediate feature corresponding to the prompt text has been moved out of the video memory, when new target audio is collected, the prompt text and the target audio can be directly input into the first attention module, and the first intermediate feature of the prompt text and the second intermediate feature of the new target audio can be output. This does not require the user to re-enter the prompt text, which can improve the audio processing rate to a certain extent.
[0055] Optionally, after generating the processing result corresponding to the target audio, the method further includes: displaying the processing result.
[0056] In this embodiment, the processing results are displayed on the terminal device for the user to view.
[0057] The technical solution of this embodiment obtains the intermediate features of the prompt text as the first intermediate features; wherein, the first intermediate features are obtained by pre-processing the prompt text based on the language model; the first intermediate features and the collected target audio are input into the language model, and the intermediate features corresponding to the target audio are output as the second intermediate features; wherein, the target audio corresponds to the prompt text, and the first intermediate features and the second intermediate features are cached in the form of key-value KV; the first intermediate features and the second intermediate features are input into the language model to generate the processing results corresponding to the target audio. The audio processing method provided by the embodiment of the present disclosure processes the first intermediate features of the prompt text and the target audio obtained by pre-processing through a language model to obtain the processing results of the target audio. Pre-processing the prompt text can save the time spent on processing the prompt text, reduce the delay of audio processing, and thus improve the efficiency of audio processing.
[0058] Figure 2 This is a structural diagram of an audio processing device provided by an embodiment of the present disclosure, such as Figure 2 As shown, the device includes:
[0059] A first intermediate feature acquisition module 210 is configured to acquire an intermediate feature of the prompt text as a first intermediate feature; wherein the first intermediate feature is obtained by pre-processing the prompt text based on a language model;
[0060] A second intermediate feature acquisition module 220 is configured to input the first intermediate feature and the collected target audio into the language model and output the intermediate feature corresponding to the target audio as the second intermediate feature; wherein the target audio corresponds to the prompt text, and the first intermediate feature and the second intermediate feature are both cached in a key-value (KV) format;
[0061] The audio processing result generating module 230 is configured to input the first intermediate feature and the second intermediate feature into the language model to generate a processing result corresponding to the target audio.
[0062] Optionally, the language model includes a first attention module, a second attention module and a third attention module.
[0063] Optionally, the first intermediate feature acquisition module 210 is further configured to:
[0064] At the start of target audio acquisition, obtaining intermediate features of the prompt text corresponding to the target audio from the cache;
[0065] If the intermediate features of the prompt text corresponding to the target audio are not obtained from the cache, the prompt text corresponding to the target audio is input into the first attention module, and the intermediate features of the prompt text are output.
[0066] Optionally, the second intermediate feature acquisition module 220 is further configured to:
[0067] Inputting the first intermediate features and the collected target audio into the second attention module, and outputting the intermediate features corresponding to the target audio;
[0068] Optionally, the third intermediate feature acquisition module 230 is further configured to:
[0069] The first intermediate feature and the second intermediate feature are input into the third attention module to generate a processing result corresponding to the target audio.
[0070] Optionally, the first intermediate feature and the second intermediate feature are both cached in a video memory in a key-value (KV) manner; and further comprising: a removal module, configured to:
[0071] If no new target audio is collected within a set time or the occupancy rate of the video memory exceeds a set threshold, the first intermediate feature is removed from the video memory and the prompt text is cached in the internal memory; alternatively, the first intermediate feature is transferred from the video memory to the internal memory for caching.
[0072] Optionally, it also includes: an audio processing module, which is used to: if the first intermediate feature is removed from the video memory and the prompt text is cached in the memory, then when a new target audio is collected, the prompt text and the target audio are input into the first attention module, and the first intermediate feature of the prompt text and the second intermediate feature of the new target audio are output.
[0073] Optionally, it further includes: a display module for:
[0074] The processing result is displayed.
[0075] The audio processing device provided by the embodiments of the present disclosure can execute the audio processing method provided by any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.
[0076] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present disclosure.
[0077] Figure 3 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Figure 3 , which shows an electronic device (eg Figure 3The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 3 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.
[0078] like Figure 3 As shown, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. Various programs and data required for the operation of the electronic device 500 are also stored in the RAM 503. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An edit / output (I / O) interface 505 is also connected to the bus 504.
[0079] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 3 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.
[0080] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.
[0081] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.
[0082] The electronic device provided by the embodiment of the present disclosure and the audio processing method provided by the above embodiment belong to the same inventive concept. For technical details not fully described in this embodiment, please refer to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.
[0083] An embodiment of the present disclosure provides a computer storage medium on which a computer program is stored. When the program is executed by a processor, the audio processing method provided by the above embodiment is implemented.
[0084] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.
[0085] In some embodiments, the client and server can communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.
[0086] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.
[0087] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:
[0088] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: obtains the intermediate features of the prompt text as the first intermediate features; wherein the first intermediate features are obtained by preprocessing the prompt text based on the language model; inputs the first intermediate features and the collected target audio into the language model, and outputs the intermediate features corresponding to the target audio as the second intermediate features; wherein the target audio corresponds to the prompt text, and the first intermediate features and the second intermediate features are both cached in the form of key-value KV; inputs the first intermediate features and the second intermediate features into the language model to generate the processing results corresponding to the target audio.
[0089] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).
[0090] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0091] The units involved in the embodiments described in this disclosure may be implemented in software or hardware. In some cases, the name of a unit does not limit the unit itself. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses."
[0092] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.
[0093] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0094] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.
[0095] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.
[0096] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.
Claims
1. An audio processing method, characterized in that: include: Obtaining an intermediate feature of the prompt text as a first intermediate feature; wherein the first intermediate feature is obtained by preprocessing the prompt text based on a language model; Inputting the first intermediate feature and the collected target audio into the language model, and outputting the intermediate feature corresponding to the target audio as the second intermediate feature; wherein the target audio corresponds to the prompt text, and the first intermediate feature and the second intermediate feature are both cached in a key-value (KV) format; The first intermediate feature and the second intermediate feature are input into the language model to generate a processing result corresponding to the target audio.
2. The method according to claim 1, characterized in that The language model includes a first attention module, a second attention module and a third attention module.
3. The method according to claim 2, characterized in that Get the intermediate features of the prompt text, including: At the start of target audio acquisition, obtaining intermediate features of the prompt text corresponding to the target audio from the cache; If the intermediate features of the prompt text corresponding to the target audio are not obtained from the cache, the prompt text corresponding to the target audio is input into the first attention module, and the intermediate features of the prompt text are output.
4. The method according to claim 2, characterized in that Inputting the first intermediate feature and the collected target audio into the language model, and outputting the intermediate feature corresponding to the target audio, including: Inputting the first intermediate features and the collected target audio into the second attention module, and outputting the intermediate features corresponding to the target audio; Inputting the first intermediate feature and the second intermediate feature into the language model to generate a processing result corresponding to the target audio includes: The first intermediate feature and the second intermediate feature are input into the third attention module to generate a processing result corresponding to the target audio.
5. The method according to claim 2, characterized in that The first intermediate feature and the second intermediate feature are both cached in a video memory in a key-value (KV) manner; the method further includes: If no new target audio is collected within a set time or the occupancy rate of the video memory exceeds a set threshold, the first intermediate feature is removed from the video memory and the prompt text is cached in the internal memory; alternatively, the first intermediate feature is transferred from the video memory to the internal memory for caching.
6. The method according to claim 5, characterized in that If the first intermediate feature is removed from the video memory and the prompt text is cached in the internal memory, when a new target audio is collected, the prompt text and the target audio are input into the first attention module, and the first intermediate feature of the prompt text and the second intermediate feature of the new target audio are output.
7. The method according to claim 1, characterized in that After generating the processing result corresponding to the target audio, the method further includes: The processing result is displayed.
8. An audio processing device, characterized in that: include: A first intermediate feature acquisition module is configured to acquire an intermediate feature of the prompt text as a first intermediate feature; wherein the first intermediate feature is obtained by preprocessing the prompt text based on a language model; A second intermediate feature acquisition module is configured to input the first intermediate feature and the collected target audio into the language model, and output the intermediate feature corresponding to the target audio as the second intermediate feature; wherein the target audio corresponds to the prompt text, and the first intermediate feature and the second intermediate feature are both cached in a key-value (KV) format; An audio processing result generation module is used to input the first intermediate feature and the second intermediate feature into the language model to generate a processing result corresponding to the target audio.
9. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the audio processing method according to any one of claims 1 to 7.
10. A storage medium comprising computer-executable instructions, wherein the computer-executable instructions are used to perform the audio processing method according to any one of claims 1 to 7 when executed by a computer processor.