Audio processing method and apparatus

Abstract

The embodiments of the present specification provide an audio processing method and device, wherein the audio processing method comprises: receiving audio to be processed of a target object in an interaction process; detecting the audio to be processed according to at least one detection dimension to obtain audio detection information corresponding to the audio to be processed, wherein the audio detection information is used to represent audio quality and reliability of the audio to be processed; determining an audio processing result corresponding to the audio to be processed according to the audio detection information and reference audio information corresponding to the target object, wherein the audio processing result is used to determine an interaction strategy corresponding to the interaction process, the audio processing result comprises target audio corresponding to the target object and / or an audio detection signal corresponding to the audio to be processed, and the target audio is obtained by extracting the audio to be processed.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of audio processing technology, and particularly to audio processing methods and apparatus. Background Technology

[0002] With the development of computer and internet technologies, voice interaction has become one of the important methods of human-computer interaction, widely used in various application scenarios such as smart homes, in-vehicle systems, customer service robots, and smart conference terminals. In these scenarios, users can converse with the interactive terminal, which can recognize and execute user commands and engage in continuous dialogue. However, during the dialogue between the user and the interactive terminal, if multiple people speak simultaneously, or if there is background noise in the user's environment, the interactive terminal needs to determine who spoke and whether it is an attempt to wake up the terminal, which can easily lead to response errors. Furthermore, background noise may interrupt the interactive terminal's voice broadcast during the dialogue, affecting the user experience. Therefore, an effective technical solution is urgently needed to address these issues. Summary of the Invention

[0003] In view of the above, embodiments of this specification provide an audio processing method. One or more embodiments of this specification also relate to an audio processing apparatus, an electronic device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0004] According to a first aspect of the embodiments of this specification, an audio processing method is provided, comprising: Receive the audio to be processed from the target object during the interaction process; The audio to be processed is detected according to at least one detection dimension to obtain audio detection information corresponding to the audio to be processed, wherein the audio detection information is used to characterize the audio quality and reliability of the audio to be processed; Based on the audio detection information and the reference audio information corresponding to the target object, the audio processing result corresponding to the audio to be processed is determined. The audio processing result is used to determine the interaction strategy corresponding to the interaction process. The audio processing result includes the target audio corresponding to the target object and / or the audio detection signal corresponding to the audio to be processed. The target audio is obtained by extracting the audio to be processed.

[0005] According to a second aspect of the embodiments of this specification, an audio processing apparatus is provided, comprising: The receiving module is configured to receive the audio to be processed by the target object during the interaction process; The detection module is configured to detect the audio to be processed according to at least one detection dimension to obtain audio detection information corresponding to the audio to be processed, wherein the audio detection information is used to characterize the audio quality and reliability of the audio to be processed; The determining module is configured to determine the audio processing result corresponding to the audio to be processed based on the audio detection information and the reference audio information corresponding to the target object. The audio processing result is used to determine the interaction strategy corresponding to the interaction process. The audio processing result includes the target audio corresponding to the target object and / or the audio detection signal corresponding to the audio to be processed. The target audio is obtained by extracting the audio to be processed.

[0006] According to a third aspect of the embodiments of this specification, an audio processing system is provided, comprising: The acquisition unit is configured to receive the audio to be processed by the target object during the interaction process; An object recognition unit is configured to detect the audio to be processed according to at least one detection dimension to obtain audio detection information corresponding to the audio to be processed, wherein the audio detection information is used to characterize the audio quality and reliability of the audio to be processed; and to determine the audio processing result corresponding to the audio to be processed based on the audio detection information and reference audio information corresponding to the target object, wherein the audio processing result is used to determine the interaction strategy corresponding to the interaction process, and the audio processing result includes target audio corresponding to the target object and / or audio detection signal corresponding to the audio to be processed, wherein the target audio is obtained by extracting the audio to be processed; A speech recognition unit is configured to convert the audio to be processed and the target audio into converted text; The semantic understanding unit is configured to generate response text based on the converted text; The voice broadcasting unit is configured to synthesize and broadcast the reply text. The decision unit is configured to determine an interaction strategy based on the transcribed text, the response text, and the audio detection signal.

[0007] According to a fourth aspect of the embodiments of this specification, an electronic device is provided, comprising: A memory and a processor, the memory and the processor being connected via a bus; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the above method.

[0008] According to a fifth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0009] According to a sixth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0010] In one embodiment of the audio processing method provided in this specification, the audio to be processed from the target object is received during the interaction process, thereby acquiring the real-time voice signal during the interaction with the target object. This provides basic data for subsequent audio detection and analysis, ensuring the real-time nature and continuity of the audio processing process. Subsequently, the audio to be processed is detected according to at least one detection dimension to obtain audio detection information that characterizes the audio quality and reliability. Through a multi-dimensional detection mechanism, the information of the audio to be processed can be comprehensively evaluated from different angles, thereby obtaining more comprehensive and accurate audio detection information. Then, the audio processing result can be determined based on the audio detection information and the reference audio information of the target object. In determining the audio processing result, not only the audio detection information of the audio to be processed itself is considered, but also the reference audio information of the target object is introduced as a reference basis, thus providing a reliable decision-making basis for determining the interaction strategy. The audio processing result may include target audio and audio detection signals. The target audio can be extracted from the audio to be processed, providing the target object with filtered and effective audio segments. The audio detection signal is used to characterize the relationship between the audio to be processed and the target object, providing a basis for judging the object identity dimension for determining the interaction strategy.

[0011] In the aforementioned audio processing method, by introducing reference audio information of the target object, the interruption judgment target corresponding to the interaction strategy is determined as the existence of valid speech of the target object. This effectively distinguishes the speech of the target object from that of other objects, accurately judging the speech state of the target object in multi-person speaking scenarios, significantly reducing the false interruption rate and improving the accuracy of interruption interaction detection. Simultaneously, the multi-dimensional detection mechanism enhances anti-interference capabilities, maintaining stable detection performance under different noise environments. Furthermore, the introduction of reference audio information allows for targeted audio extraction based on the acoustic characteristics of the target object, further improving the accuracy and reliability of audio detection and enhancing the user's interactive experience. Attached Figure Description

[0012] Figure 1 This is an architecture diagram of an audio processing system provided in one embodiment of this specification; Figure 2 This is a flowchart illustrating an audio processing method provided in one embodiment of this specification; Figure 3 This is a schematic diagram of a full-duplex voice interaction link in an audio processing method provided in one embodiment of this specification; Figure 4 This is a flowchart illustrating the audio processing method provided in one embodiment of this specification within the speaker recognition module. Figure 5 This is a schematic diagram of a timing scenario in an audio processing method provided in one embodiment of this specification; Figure 6 This is a schematic diagram of the structure of an audio processing device provided in one embodiment of this specification; Figure 7 This is a structural block diagram of an electronic device provided in one embodiment of this specification. Detailed Implementation

[0013] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0014] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0015] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0016] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in one or more embodiments of this specification are obtained through open-source datasets or public datasets that comply with their license agreements, or are obtained with full authorization from the relevant parties. Moreover, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0017] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundation model. It is pre-trained using large-scale unlabeled corpora to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0018] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as natural language processing tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0019] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0020] Voice Activity Detection (VAD) is a technique used to determine whether an audio signal contains speech components. It distinguishes between speech segments and non-speech segments by analyzing the energy, spectrum, and other characteristics of the audio signal. It is widely used in speech endpoint detection in fields such as voice communication and speech recognition.

[0021] Full-duplex voice interaction refers to an interaction mode in which both parties can send and receive voice messages simultaneously. It allows users to interrupt the system at any time during the playback. Compared to the half-duplex mode, which requires waiting for the system to finish playing before inputting, the full-duplex mode provides a more natural and efficient interactive experience.

[0022] Automatic Speech Recognition (ASR) is a technology that automatically converts speech signals into text. It decodes speech signals through modules such as acoustic models and language models and outputs the corresponding text sequence. It is one of the core technologies in human-computer voice interaction.

[0023] Text-to-speech (TTS) is a technology that converts text information into speech signals. It is used in voice interaction systems to output voice feedback to users, enabling the system to convey information to users in a natural speech manner.

[0024] Speaker Embedding is a fixed-dimensional vector representation extracted from a speech signal to characterize the speaker's acoustic features. The embedding vectors of different speakers are far apart in high-dimensional space, while the embedding vectors of the same speaker are relatively close. It can be used to compare and confirm the speaker's identity.

[0025] Interruption detection is a technique used in full-duplex voice interaction systems to detect whether a user intentionally interrupts the system's current broadcast. Accurate interruption detection can improve the naturalness of the interaction and the user experience.

[0026] Environmental noise estimation, also known as environmental noise detection, is a technique for analyzing and estimating background noise in the current acoustic environment. The results can be used to dynamically adjust the operating parameters of the speech detection system to adapt to different acoustic environments.

[0027] Density Mask: A binary label generated based on the density of speech frames within a time window. It is used to indicate whether the density of speech activity within the window meets preset conditions. Different density thresholds can be set to obtain masks with different levels of strictness.

[0028] Gain normalization is a processing technique that standardizes the amplitude of an audio signal to eliminate gain differences caused by different acquisition devices or transmission links, so that subsequent audio processing algorithms can work on a unified signal reference.

[0029] Pre-buffer: This is a mechanism used in real-time audio processing to cache recent audio data. When a specific event is detected, historical audio data can be retrieved from the buffer to ensure the integrity of the speech segment and avoid the loss of sentence beginning information.

[0030] A state machine is a computational model that performs state transitions based on the current state and input conditions. In audio processing, it can be used to maintain and manage different states during speech detection, such as silent state, active speech state, and exit hold state.

[0031] A ring buffer is a fixed-size circular data structure used to cache recent data in real-time processing systems. When the buffer is full, new data overwrites the oldest data. It is suitable for managing the caching of continuous data such as audio streams.

[0032] To address the aforementioned technical problems, this specification provides an audio processing method. This specification also relates to an audio processing apparatus, an audio processing system, an electronic device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0033] See Figure 1 , Figure 1 This specification illustrates an architecture diagram of an audio processing system according to one embodiment of the present specification. The audio processing system may include a client 100 and a server 200. Client 100 is used to send audio to be processed to server 200; Server 200 is used to receive audio to be processed from the target object during the interaction process; detect the audio to be processed according to at least one detection dimension to obtain audio detection information corresponding to the audio to be processed, wherein the audio detection information is used to characterize the audio quality and reliability of the audio to be processed; determine the audio processing result corresponding to the audio to be processed according to the audio detection information and the reference audio information corresponding to the target object, wherein the audio processing result is used to determine the interaction strategy corresponding to the interaction process, the audio processing result includes the target audio corresponding to the target object and the audio detection signal corresponding to the audio to be processed, wherein the target audio is obtained by extracting the audio to be processed, and the audio detection signal is used to characterize the relationship between the audio to be processed and the target object.

[0034] An audio processing system may include multiple clients 100 and a server 200. Clients 100 can be referred to as edge devices, and server 200 can be referred to as cloud devices. Multiple clients 100 can establish communication connections through server 200. In an audio processing scenario, server 200 is used to provide audio processing services between multiple clients 100. Each client 100 can act as a sender or receiver, communicating through server 200.

[0035] In this system, client 100 and server 200 establish a connection via a network. The network provides the medium for communication between client 100 and server 200. The network can include various connection types, such as wired or wireless communication links or fiber optic cables. Data transmitted by client 100 may need to undergo encoding, transcoding, compression, or other processing before being published to server 200.

[0036] Client 100 can be a browser, an app (application), a web application such as an H5 (HyperText Markup Language 5) application, a lightweight application (also known as a mini-program), or a cloud application. Client 100 can be developed based on the software development kit (SDK) of the corresponding service provided by server 200, such as a real-time communication (RTC) SDK. Client 100 can be deployed on a computing device and depends on the device or certain apps on the device to run. The computing device may have a display screen and support information browsing, such as a personal mobile terminal like a mobile phone, tablet, or personal computer. Various other types of applications can also be configured on the computing device, such as human-computer interaction applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, and social media platform software.

[0037] Server 200 may include servers providing various services, such as servers providing communication services to multiple clients, servers supporting backend training of models used on clients, and servers processing data sent by clients. It should be noted that server 200 can be implemented as a distributed server cluster composed of multiple servers, or as a single server. The server can also be a server in a distributed system, or a server integrated with blockchain. The server can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0038] It is worth noting that the audio processing methods provided in the embodiments of this specification are generally executed by the server. However, in other embodiments of this specification, the client may also have similar functions to the server, thereby executing the audio processing methods provided in the embodiments of this specification. In other embodiments, the audio processing methods provided in the embodiments of this specification may also be executed jointly by the client and the server.

[0039] In one optional embodiment of this specification, the audio processing method may include the following steps.

[0040] Receive the audio to be processed from the target object during the interaction process; The audio to be processed is detected according to at least one detection dimension to obtain audio detection information corresponding to the audio to be processed, wherein the audio detection information is used to characterize the audio quality and reliability of the audio to be processed; Based on the audio detection information and the reference audio information corresponding to the target object, the audio processing result corresponding to the audio to be processed is determined. The audio processing result is used to determine the interaction strategy corresponding to the interaction process. The audio processing result includes the target audio corresponding to the target object and / or the audio detection signal corresponding to the audio to be processed. The target audio is obtained by extracting the audio to be processed. The audio detection signal is used to characterize the relationship between the audio to be processed and the target object.

[0041] The audio processing methods provided in the embodiments of this specification can be applied to various scenarios. In the full-duplex voice interaction scenario of a smart speaker, when the speaker is broadcasting weather information, the speaker (i.e., the registered user, the target audience) may want to interrupt the broadcast to ask other questions. In this case, the system receives the speaker's audio to be processed and detects it by combining it with the speaker's reference audio information, accurately identifying the speaker's interruption intention while ignoring the voice interference from other people nearby. In the scenario of an in-vehicle voice assistant, the driver may need to interrupt the system to change the destination when it is broadcasting navigation information. The system uses the above method to distinguish between the driver's voice and the passenger's voice, and only responds to the driver's interruption request. In the scenario of a smart conference terminal, when the system is broadcasting meeting minutes, the speaker's interruption voice can be accurately detected, while the conversations of other participants will not trigger false interruptions. In the scenario of remote customer service, when the customer service system is broadcasting automatic response content, it can detect the customer's interruption intention using the above method and respond to the interruption request after verifying the customer's identity. In smart home scenarios, when a smart terminal is broadcasting information, the system can distinguish interruptions from specific family members, enabling a personalized voice interaction experience.

[0042] See Figure 2 , Figure 2 A flowchart of an audio processing method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0043] Step 202: Receive the audio to be processed from the target object during the interaction process.

[0044] In one optional embodiment of this specification, the audio processing method provided in this embodiment can be applied to an interactive terminal, which executes the audio processing method. In another optional embodiment of this specification, the audio processing method can also be applied to a server corresponding to the interactive terminal, whereby the interactive terminal can send the received audio of the target object to the server for the server to execute the audio processing method. This embodiment of the specification does not limit this.

[0045] The interactive terminal can be understood as a terminal device with voice interaction capabilities, such as a smart speaker, smartphone, in-vehicle voice system, smart conference terminal, robot, etc. This specification does not limit the specific form of the interactive terminal in its embodiments. The audio processing method provided in this specification's embodiments can also be applied to the server corresponding to the interactive terminal.

[0046] The target object can be understood as a registered user or speaker in the voice interaction system (hereinafter referred to as the system) corresponding to the interactive terminal, and can be a specific speaker that the interactive terminal expects to recognize and respond to. The interaction process can be understood as the process of voice dialogue between the target object and the interactive terminal, including but not limited to the interactive terminal broadcast stage and the user input stage. The interactive terminal broadcast stage can be understood as the stage in which the interactive terminal performs voice broadcast, and the user input stage can be understood as the stage in which the user speaks to the interactive terminal. The audio to be processed can be understood as the audio signal between the target object and the interactive terminal during the interaction process. In some optional embodiments, the audio to be processed is the original audio signal collected by a near-end microphone. In other optional embodiments, the audio to be processed is the audio signal after preliminary processing of the original audio signal.

[0047] In some alternative embodiments, the audio to be processed can be received in real time in a streaming manner; in other alternative embodiments, the audio to be processed can be received in batches. In the case of streaming reception, the audio to be processed can be continuously input in units of fixed-length audio frames. The length of each audio frame can be determined according to the actual application scenario; for example, the length of each audio frame could be 10 milliseconds.

[0048] The audio to be processed may contain the speech signal of the target object, the speech signal of other objects (such as the speech signal of other speakers), and various environmental noises. During the interactive playback phase, the audio to be processed can be continuously received to detect in real time whether there is any interrupting speech from the target object, thereby interrupting the voice playback on the interactive end.

[0049] In practical applications, the audio processing methods provided in the embodiments of this specification can be applied to full-duplex voice interaction links. See [link / reference]. Figure 3 , Figure 3This diagram illustrates a full-duplex voice interaction link in an audio processing method according to an embodiment of this specification. Specifically, this audio processing method can be applied to a speaker recognition module in the interaction terminal, which is connected to an existing full-duplex voice interaction link in parallel branches. In this link, the client's voice is acquired near the source and sent to Automatic Speech Recognition (ASR). The ASR continuously outputs the streaming transcription result. A Large Language Model (LLM) simultaneously receives the transcription result and the currently playing text from Text-to-Speech (TTS), determining whether the client's interruption constitutes a genuine interruption intent and outputting an interruption signal. The voice control system summarizes the transcription result and the interruption signal to decide whether to stop the current playback. The speaker recognition module takes one audio path from the client's voice and runs it parallel to the ASR. Its output is divided into two parts: enhanced audio is sent to the ASR to align the ASR with the single-person signal; the speaker signal is bypassed and sent to the voice control system as a rejection dimension independent of semantic judgment.

[0050] Specifically, this link includes a customer voice acquisition module 10, a speaker recognition module 20, an ASR speech recognition module 30, an LLM semantic understanding and generation module 40, a TTS broadcasting module 60, and a voice central control interruption decision module 50. The customer voice acquisition module 10 serves as the audio acquisition entry point, used to acquire the uplink audio stream (i.e., the audio to be processed) from the near end, including the speaker's voice, interference from others, and environmental noise. The speaker recognition module processes the audio in parallel, extracting the target speaker to purify the audio and extracting the speaker's identity features and validity status. The ASR speech recognition module transcribes the audio stream into a text stream in real time for streaming output. The LLM semantic understanding and generation module understands the user's intent based on the text content output by the ASR speech recognition module and generates a response text. The TTS broadcasting module converts the response text generated by the LLM semantic understanding and generation module into speech for playback. The voice central control interruption decision module combines semantic information and speaker identity signals to determine whether to interrupt the current TTS broadcast.

[0051] The uplink audio stream can be obtained from the customer voice acquisition module 10, and then input into the ASR speech recognition module 30 and the speaker recognition module 20.

[0052] After receiving the audio to be processed, the speaker recognition module 20 executes the target speaker extraction algorithm, separates the speaker signal, and determines the effective speech and duration. The enhanced audio extracted based on the speaker's identity information is sent to the ASR speech recognition module 30, and the speaker signal is sent to the voice central control interruption decision module 50. The speaker signal is the audio detection signal, which includes speaker attribution, effective speech flags, and continuous verification results.

[0053] The ASR speech recognition module 30 simultaneously receives the audio to be processed and the enhanced audio, converts them into transcribed text in real time, and sends the transcribed text to the LLM semantic understanding and generation module 40 for semantic analysis, and to the voice control interruption decision module 50 as the basis for semantic judgment. The LLM semantic understanding and generation module 40 performs semantic analysis on the transcribed text, determines the user's intent, and generates corresponding response text. This response text is sent to the TTS broadcasting module 60 for speech synthesis, and then sent to the voice control interruption decision module 50 to determine whether the current dialogue logic can be interrupted. Based on the transcribed text, the response text, and the speaker's signal, the voice control interruption decision module 50 determines whether to generate an interruption command to interrupt the current voice broadcasting of the TTS broadcasting module.

[0054] The speaker identification module is placed before the ASR because speaker differentiation needs to be completed in the acoustic domain; the speaker signal is bypassed and sent to the voice control center without going through the LLM to keep the acoustic evidence independent; the module can bypass upstream and downstream, and the enhanced path is automatically straight-through when the identity representation has not been established.

[0055] In an optional embodiment of this specification, after receiving the audio to be processed from the target object during the interaction process, the method further includes: The audio to be processed is denoised to obtain the denoised audio frequency corresponding to the audio to be processed; Based on the reference audio information corresponding to the target object, the audio to be processed is extracted to obtain the target audio corresponding to the target object; The step of performing validity detection on the audio to be processed according to the second detection dimension to obtain the second audio detection information corresponding to the audio to be processed includes: Based on the second detection dimension, the effectiveness of the denoised audio is detected to obtain the second audio detection information corresponding to the audio to be processed.

[0056] Denoising can be understood as the process of suppressing or eliminating noise components in the audio signal to be processed. It can be achieved using methods such as spectral subtraction, Wiener filtering, and deep learning denoising models. The purpose of denoising is to improve the signal-to-noise ratio of the audio signal and reduce the interference of environmental noise and non-target speech on subsequent processing.

[0057] The target audio can be understood as an audio segment related to the target object extracted from the audio to be processed based on reference audio information. The extraction process can use the reference audio information as a condition to selectively retain speech components that match the acoustic vector features of the target object.

[0058] Based on this, the denoising process first removes environmental noise and non-target speech components to improve the overall audio quality. Target audio extraction uses reference audio information (such as acoustic embedding vectors) as a condition, and a personalized enhancement algorithm separates speech components related to the target object from the audio to be processed, achieving speaker-level speech separation. Validity detection ensures that the audio after denoising and extraction still contains sufficient valid speech information, avoiding wasting subsequent computational resources on invalid audio. The three steps can be executed sequentially or partially in parallel; this specification does not strictly limit the execution order. The final audio processing result includes the extracted target audio and audio detection signal, which are used for subsequent speech recognition and interaction strategy determination, respectively.

[0059] In practical applications, denoising can be implemented in various ways. General denoising methods (such as spectral subtraction and Wiener filtering) are suitable for general noise suppression, do not require prior information about the target speaker, and are relatively simple to implement. Deep learning-based denoising methods can handle more complex noise scenarios, but require significant computational resources. Personalized enhancement methods, based on the acoustic embedding of the target object, can more effectively preserve the speech components of the target object while suppressing speech interference from other speakers, making them suitable for multi-speaker scenarios. Target audio extraction can be achieved based on time-frequency domain masking methods, i.e., generating an enhancement mask in the time-frequency domain based on the acoustic embedding vector, and applying the enhancement mask to the audio signal to extract the target audio from the audio to be processed.

[0060] In a specific implementation of this specification, a three-way parallel processing architecture can be used to process the near-end raw audio (i.e., the audio to be processed), the general noise-reduced frequency (i.e., the denoised frequency), and the personalized enhancement audio (i.e., the target audio) separately. The near-end raw audio can be used for environmental noise and noise level estimation, avoiding the estimation result from being contaminated by the noise reduction result. The general noise-reduced frequency has strong speech and noise separability and can be used for speech activity detection and gating decision. The personalized enhancement audio is audio that has undergone personalized enhancement conditioned on the speaker's identity but without general noise reduction. Its non-speaker components have been suppressed with minimal loss of speech naturalness, and it is used for the final output or uplink transmission. The general noise reduction branch uses a deep learning-based noise reduction model to perform general noise suppression on the audio to be processed, removing environmental noise and non-speech interference. The personalized enhancement branch uses the speaker's acoustic embedding vector as a condition to perform conditional enhancement processing on the audio to be processed, preserving and enhancing speech components that match the speaker's acoustic features, while suppressing the speech components of other speakers. The enhanced audio undergoes post-masking processing, including gain adjustment (differentiated adjustment of gain for different frequency bands), clipping (to prevent signal clipping distortion), and power-law sharpening (to enhance mask contrast) to further optimize the enhancement effect. The personalized enhancement module uses a stateless computation graph to achieve streaming inference, processing audio signals independently frame by frame without buffering historical frame data, effectively reducing processing latency. The parallel processing results of the three audio streams are sent to different downstream modules for use.

[0061] For example, in a living room scenario, a television is playing a program (producing continuous TV sound), and other people are talking nearby. The speaker wants to interrupt the system's current broadcast. After receiving the audio to be processed, which contains various sound components, the general noise reduction branch first suppresses the TV sound and general ambient noise, outputting relatively clean audio. The personalized enhancement branch, conditioned on the speaker's acoustic embedding, further enhances the audio, selectively enhancing the speaker's speech components while suppressing the speech of others nearby. The enhanced audio is then sent to the speech recognition module for transcription. Because noise and other speakers' speech have been effectively suppressed, the accuracy of speech recognition is significantly improved. Simultaneously, the effectiveness detection result of the denoised audio is used to determine whether there is enough valid speech for recognition, avoiding unnecessary speech recognition processing when there is no valid speech, thus saving computational resources. The extracted target audio can also be used for further acoustic comparison and speaker tracking, achieving multi-functional reuse of audio processing.

[0062] Understandably, the process of validating the denoised audio based on the second detection dimension is similar to the process of validating the audio to be processed based on the second detection dimension described below, and will not be repeated here.

[0063] By employing the above-described method, a combined processing flow of denoising, target audio extraction, and validity detection can be used to obtain high-quality target speech signals from complex mixed audio signals. This provides excellent input data for subsequent speech recognition and other deep processing, effectively improving the accuracy of speech recognition and the overall performance of audio processing. It is particularly suitable for challenging scenarios such as multiple speakers and high noise levels.

[0064] In one optional embodiment of this specification, the step of extracting the audio to be processed based on the reference audio information corresponding to the target object to obtain the target audio corresponding to the target object includes: Based on the reference audio information corresponding to the target object, the audio to be processed is extracted to obtain the initial audio corresponding to the target object; If the initial audio meets the preset hit conditions, the associated audio of the initial audio is obtained from the pre-buffered audio corresponding to the target object, and the initial audio and the associated audio are concatenated to obtain the target audio corresponding to the target object.

[0065] The pre-buffered audio can be understood as a mechanism that continuously stores audio data from the most recent period using a circular buffer structure. The write operation to this buffer is independent of any decision result and remains always open, thus preserving historical audio completely regardless of whether the decision is valid. The pre-buffer length covers the decision latency required for multi-level decisions and the duration of the sentence beginning to be filled, for example, 700 milliseconds. The initial audio refers to the audio stream currently being processed after extraction by the target speaker. The preset hit condition refers to the comprehensive criterion that all necessary conditions in the decision chain have been met, including but not limited to the simultaneous fulfillment of a loose density mask and a strict density mask, and the passage of identity information evidence. Optionally, the preset hit condition can be that both the audio detection information and the reference audio information satisfy the existence and validity conditions. Associated audio refers to historical audio segments that are temporally continuous with the current initial audio in the pre-buffer; its length is determined by the cumulative length of the current continuous non-speech audio, with the pre-buffer length as the upper limit.

[0066] Further, after extracting the audio to be processed based on the reference audio information corresponding to the target object to obtain the initial audio corresponding to the target object, the method further includes: If it is determined that the initial audio does not meet the preset hit condition, and the cumulative length of the continuous non-speech audio currently corresponding to the initial audio does not meet the pre-buffer length, the initial audio is withheld to retain the pre-buffered audio corresponding to the target object.

[0067] The initial audio does not meet the preset hit conditions, which can be understood as the judgment not being valid. The continuous non-speech length can be understood as the total length of time from the current moment back when no valid speech was detected. It can be used to determine whether the beginning of the sentence is still within the pre-buffered range and whether backfilling is needed.

[0068] When the judgment is not upheld, the output strategy is divided into two levels: when the length of continuous non-speech has not exceeded the pre-buffer length, no samples are produced for the audio frame to be processed, that is, the output is withheld to ensure that the pre-buffered content is not consumed in advance; when the length of continuous non-speech has exceeded the pre-buffer length, the output is switched to outputting equal-length silent audio frames to maintain the continuity of timing.

[0069] Based on this, the pre-buffering concatenation mechanism decouples the strictness of the decision from the integrity of the output, allowing the system to tighten the decision conditions without affecting the interactive experience. When a decision is made, the sentence-first audio stored in the pre-buffer is retrieved and output along with the current frame. Personalization enhancement occurs before pre-buffering; that is, extraction based on identity representation is performed first, and then its output is written into the circular pre-buffer. Therefore, the sentence-first audio obtained after subsequent backfilling is also the audio extracted by the speaker.

[0070] In practice, the pre-buffer length can be dynamically adjusted according to product requirements. For example, in scenarios prioritizing low latency, the pre-buffer length can be appropriately shortened to reduce storage overhead; in scenarios with high requirements for sentence beginning integrity, the pre-buffer length can be increased to cover longer decision latency. A multi-level pre-buffer structure can also be adopted, maintaining independent ring buffers for different branches, so that the detection branch and the output branch each use the appropriate audio version.

[0071] In one specific implementation of this specification, audio from each branch can be continuously written to a circular pre-buffer, with the writing action independent of any decision result. When a decision is made, the historical audio corresponding to the accumulated length is retrieved from the pre-buffer and output along with the current frame. The selection of the output branch is separated from the detection branch. Detection uses a general noise reduction branch with good separability, while the output uses a personalized enhancement branch where non-speaker components have been suppressed and speech impairment is minimal. Sentence-end locking can be applied to the residual interval where speech transitions to silence after the hold ends, suppressing residual noise and trailing artifacts at the end of the sentence.

[0072] For example, in an intelligent customer service dialogue scenario, the speaker begins to speak during the system's broadcast. At time t2, the system detects the speaker has started speaking, but the joint decision conditions are not yet fully met, so the output is held back, and the initial audio of the sentence is completely stored in the pre-buffer. At time t3, both density masks are simultaneously met, and the identity information evidence is valid, completing the decision. The system retrieves up to 700 milliseconds of the initial audio from the pre-buffer at once and outputs it along with the current frame, ensuring the speech recognition module receives complete initial sentence information and avoiding transcription errors caused by missing the first character.

[0073] In summary, by using a pre-buffering splicing mechanism, the system improves accuracy by tightening decision conditions, while ensuring the integrity of the output audio through sentence-beginning backfilling and output retention strategies, so that the improvement in accuracy does not come at the expense of a decrease in the interactive experience.

[0074] Step 204: Detect the audio to be processed according to at least one detection dimension to obtain audio detection information corresponding to the audio to be processed, wherein the audio detection information is used to characterize the audio quality and reliability of the audio to be processed.

[0075] In this context, the detection dimension can be understood as the dimension used for detection and analysis of the audio to be processed. At least one detection dimension may include, but is not limited to, existence detection dimension, validity detection dimension, environmental noise detection dimension, energy detection dimension, and density detection dimension. Audio detection information can be understood as the comprehensive detection information obtained by detecting the audio to be processed, used to characterize the audio quality and reliability of the audio to be processed. Audio quality may include the signal-to-noise ratio, clarity, and integrity of the audio to be processed, while reliability may include the consistency of speech activity and the stability of the speech signal.

[0076] Based on this, the audio to be processed is detected through at least one detection dimension to obtain more comprehensive and accurate audio detection information, providing sufficient data support for subsequent audio processing results. The detection dimensions included in the at least one detection dimension can be used individually or in combination; this specification does not limit this approach in the embodiments.

[0077] In interactive scenarios, the audio to be processed may contain various audio components such as silence, background noise, brief non-speech audio (such as coughing or knocking sounds), and the speaker's voice. Based on this, it is necessary to accurately judge the various audio components contained in the audio to be processed. The specific implementation method is as follows.

[0078] The step of detecting the audio to be processed based on at least one detection dimension to obtain audio detection information corresponding to the audio to be processed includes: Based on the first detection dimension, an existence detection is performed on the audio to be processed to obtain first audio detection information corresponding to the audio to be processed, wherein the first audio detection information is used to characterize whether speech exists in the audio to be processed; and / or Based on the second detection dimension, the audio to be processed is subjected to validity detection to obtain the second audio detection information corresponding to the audio to be processed, wherein the second audio detection information is used to characterize whether there is valid speech in the audio to be processed.

[0079] Presence detection can be understood as the process of determining whether the audio to be processed contains speech components. The goal of presence detection is to distinguish between speech segments and non-speech segments, which can include silent segments and noise segments. Presence detection can be implemented using speech activity detection algorithms, which can determine the presence of speech activity by analyzing parameters such as short-time energy, zero-crossing rate, and spectral characteristics of the audio to be processed. Alternatively, it can be achieved by detecting environmental noise in the audio to be processed, adjusting the operating parameters of the audio detector based on the environmental noise information, and then performing speech detection on the audio to be processed. The output of the adjusted audio detector is then used to determine whether speech exists in the audio to be processed. The first audio detection information can be understood as the first audio detection result, which can be the result of detecting whether speech exists in the audio to be processed.

[0080] Validity detection can be understood as the process of determining whether the audio to be processed contains speech components, and whether the contained speech components are valid speech. Valid speech can be understood as speech that meets preset validity conditions, such as the speech duration exceeding a preset duration threshold or the speech energy remaining stable within a certain range. Second audio detection information can be understood as the second audio detection result, which may include whether there is valid speech in the audio to be processed, and the marking information of valid and invalid speech in the audio to be processed.

[0081] Based on this, existence detection can be performed on the audio to be processed according to the first detection dimension, quickly determining whether speech exists in the audio with low computational complexity, and obtaining the first audio detection information corresponding to the audio to be processed. And / or, validity detection can be performed on the audio to be processed according to the second detection dimension, obtaining the second audio detection information corresponding to the audio to be processed. Combining the first and second detection dimensions improves the accuracy of audio detection results while maintaining detection sensitivity. It is understood that the detection processes of the first and second detection dimensions can be executed independently, or cascaded sequentially, or, depending on actual needs, only the detection process of the first detection dimension or only the detection process of the second detection dimension can be executed; this specification does not limit this.

[0082] In practical applications, the specific implementation methods of presence detection and validity detection can be adjusted according to the application scenario. For example, in noisy environments, the sensitivity threshold of presence detection can be increased to reduce the probability of false triggers caused by noise; in scenarios requiring rapid response, the duration threshold required for validity detection can be shortened to speed up detection. The results of presence detection and validity detection can be represented in the form of Boolean values, probability values, or confidence scores, etc., and this specification does not specify any particular form. Furthermore, the results of the two detection dimensions can be used independently or in combination to obtain more comprehensive audio detection information.

[0083] In a specific implementation of this specification, the validity detection process first employs a speech activity detection algorithm based on short-time energy and zero-crossing rate. The system analyzes the audio stream frame by frame in 10-millisecond units, calculating the short-time energy and zero-crossing rate features of each frame and comparing them with an adaptive energy threshold. When the frame energy exceeds the energy threshold and the zero-crossing rate is within a reasonable range, the frame is marked as a speech active frame, and the first detection result indicates the presence of speech components. Subsequently, a density-based detection method is used to count the density of speech active frames within a sliding window of a preset time length. When the density exceeds a preset threshold, the second detection result indicates the presence of valid speech components. The length of the sliding window can be set according to actual needs, for example, it can be set to 100 milliseconds.

[0084] For example, in a full-duplex voice interaction scenario, when the system is broadcasting information, background air conditioning noise is present but has low energy, so it is not recognized as voice activity, indicating the absence of voice components. When the speaker gives an interrupt command, the energy and zero-crossing rate characteristics of the speech frame are clearly different from the background noise, so it is recognized as voice activity. Subsequently, it is confirmed that the speech has sufficient duration and density, indicating the presence of valid voice components. If it is just a brief cough, although the presence detection may identify it as voice activity, the validity detection determines that it does not constitute valid voice due to its insufficient duration and low density.

[0085] By employing the above method, through the layered cooperation of existence detection in the first detection dimension and validity detection in the second detection dimension, refined detection of speech components in audio is achieved. This effectively distinguishes valid speech from various non-speech interferences, improving the accuracy and reliability of audio detection information and providing high-quality detection data for determining subsequent audio processing results.

[0086] In one optional embodiment of this specification, the step of performing presence detection on the audio to be processed according to a first detection dimension to obtain first audio detection information corresponding to the audio to be processed includes: Based on the first detection dimension, environmental noise is detected on the audio to be processed to obtain environmental state information corresponding to the audio to be processed. Based on the environmental state information, the audio detection parameters of the audio detector are adjusted to obtain the adjusted audio detector; Based on the adjusted audio detector, speech detection is performed on the audio to be processed to obtain the first audio detection information corresponding to the audio to be processed.

[0087] The environmental state information can be understood as noise characteristic information obtained by analyzing the environmental noise components in the audio to be processed. This can include parameters such as noise energy level, noise spectrum distribution, and noise type, which can be used to indicate the noise characteristics of the current acoustic environment corresponding to the audio to be processed. The audio detector can be understood as a module used to perform speech detection. Operating parameters can be understood as parameter settings that affect the detection behavior of the audio detector. Operating parameters may include, but are not limited to, decision sensitivity, filter parameters, smoothing coefficient, detection threshold values ​​(such as energy threshold, the strictness of speech validity conditions), the detector's internal operating mode (such as switching to high sensitivity mode or high robustness mode), speech validity conditions (such as the conditions that speech features such as pitch and spectral structure must meet to be considered valid speech), whether a reduction in the probability of speech presence is allowed (whether a lower confidence requirement for speech presence is allowed in a specific scenario), and the audio branch fed into the detector (the selected audio preprocessing path, such as whether to select a branch with stronger noise reduction processing). Adjusting the audio detection parameters of the audio detector can be understood as modifying the values ​​of the operating parameters according to the environmental state information, so that the adjusted audio detector can adapt to the current acoustic environment.

[0088] Based on this, environmental noise can be detected on the received audio to be processed. The results of the environmental noise detection are processed using a smoothing strategy of fast attack and slow release. That is, when a sudden increase in noise is detected, a rapid response is made to raise the threshold, and when the noise decreases, a slow recovery is made to maintain sensitivity. The sensitivity levels adopt an integer design, dividing the continuous noise level into a finite number of discrete levels. Each level corresponds to a set of preset detection parameters. This design facilitates parameter management and adjustment, while ensuring the robustness of parameter adjustment. When the noise level jumps from one level to another, the parameter switching is achieved through a smooth transition, avoiding interference from abrupt parameter changes in the detection results.

[0089] Furthermore, after obtaining the environmental state information, the operating parameters of the audio detector can be adaptively adjusted according to the environmental state information to obtain the adjusted audio detector. The adjusted audio detector is then used to detect the audio to be processed, thereby obtaining the first audio detection information.

[0090] In practical applications, there are various ways to implement environmental noise detection. For example, the noise level can be estimated by calculating the statistical characteristics of the audio signal during the silent period, or the spectral distribution of the noise can be estimated by performing spectral analysis on the audio signal. The operating parameters of the audio detector can be adjusted using methods such as table lookup, linear interpolation, or model-based calculations; this specification does not specify any particular method.

[0091] In one specific implementation of this specification, after receiving the audio to be processed, environmental noise detection can be performed to obtain environmental state information. The smoothing weight is set to a larger value when noise increases (fast response) and a smaller value when noise decreases (slow recovery), creating a smooth effect of fast attack and slow playback. The noise energy level is divided into multiple levels, such as quiet, normal, and noisy, with each level corresponding to a set of sensitivity parameters. The detector loads the corresponding parameter configuration according to the current environmental noise level and performs a smooth transition of parameters when switching levels.

[0092] Optionally, multi-dimensional state variables (i.e., environmental state information) such as ambient noise level, bubble noise, non-stationarity, echo residual, or speech presence probability can be extracted in real time from the near-end raw audio branch, or the noise level can be directly obtained as prior information through an external interface. To eliminate parameter oscillations caused by sudden environmental changes, an asymmetric smoothing process of "fast attack, slow release" can be applied to the environmental state information. That is, when an increase in noise is detected, a rapid response is immediately followed, while when the noise decreases, a slow release is performed at a predetermined rate, thereby ensuring that the threshold adjustment is both timely and stable. Based on this, an integer sensitivity level is used as a unified control hub. This level is not only directly used as the sensitivity parameter of the underlying detector after being limited, but also indexes the upper-level strategy branches with its original value. Different strategy branches correspond to different dynamic parameter mapping tables. Based on the current environmental status information, product mode and the speaker's historical volume level, the threshold value of the detector, the internal working mode (such as whether to allow the probability of speech to be reduced) and even the input audio branch are jointly adjusted: for example, in quiet scenes, the conditions for speech to be established, such as pitch or spectral structure, are automatically relaxed to prevent missed hearing, while in high noise or strong echo scenes, the judgment threshold is automatically increased to prioritize the suppression of false triggers.

[0093] For example, if the audio being processed is collected in a quiet library environment, and a low background noise level is detected, the sensitivity setting is set to the quiet level, corresponding to a lower energy threshold and higher detection sensitivity, allowing the system to detect subtle speech activity. When the user moves to a noisy coffee shop, a significant increase in noise level is detected, and the sensitivity setting is quickly switched to the noisy level, corresponding to a higher energy threshold, to suppress false triggers caused by background noise. Simultaneously, the spectral characteristics of the noise are also incorporated into the parameter adjustment; for example, if there is more low-frequency noise in a coffee shop, the system may adjust the filtering parameters to enhance the suppression of low-frequency noise. When the user moves back to a relatively quiet environment, the sensitivity setting slowly returns to a lower level, ensuring that the system maintains high detection sensitivity in the new environment.

[0094] The above method realizes a mechanism for dynamically adjusting the operating parameters of the audio detector based on environmental status information, enabling the detection system to adapt to different acoustic environments. It automatically improves anti-interference capability when the noise is high and maintains high detection sensitivity when the noise is low, effectively ensuring the accuracy and stability of speech detection in various environments.

[0095] In one optional embodiment of this specification, the step of performing validity detection on the audio to be processed according to the second detection dimension to obtain the second audio detection information corresponding to the audio to be processed includes: According to the second detection dimension, energy detection is performed on each audio frame to be processed corresponding to the audio to be processed to obtain energy detection results, and the target audio frame to be processed is determined from each audio frame to be processed based on the energy detection results. Based on a preset length window, density detection is performed on the target audio frame to be processed to obtain density detection results, and second audio detection information corresponding to the audio to be processed is determined based on the density detection results.

[0096] Energy detection can be understood as the process of calculating the short-time energy value of each audio frame to be processed and comparing the short-time energy value with a preset energy threshold. The target audio frame to be processed can be understood as the audio frame to be processed whose short-time energy value is greater than the preset energy threshold, and the target audio frame to be processed may contain valid speech components.

[0097] A preset length window can be understood as a fixed-length time window that slides along the time axis; for example, a preset length window could be a 100-millisecond window. Density detection results can be used to reflect the density of potentially effective speech components in the time dimension. These density detection results can be the ratio of the number of target audio frames to be processed to the number of all audio frames to be processed within the preset length window.

[0098] Based on this, in the detection process of the second detection dimension, energy detection, as the first-stage coarse screening step, can calculate the short-time energy value of each audio frame to be processed with relatively low computational overhead, retaining only the target audio frames whose short-time energy value is greater than a preset energy threshold. Density detection, as the second-stage fine selection step, further distinguishes between real speech and transient interference by analyzing the density distribution of the target audio frames within a preset length window. Real speech has continuity and stability in time, and its target audio frames to be processed exhibit a high density within the preset length window. Transient interference, such as keyboard typing sounds or the sound of objects falling, may have a high short-time energy value, but its duration is short, and the density of the target audio frames to be processed within the preset length window is low. Therefore, a two-stage joint detection strategy of energy detection and density detection can achieve accurate recognition of effective speech while maintaining high computational efficiency.

[0099] In practical applications, the preset energy threshold can be set to a fixed value or a dynamically adaptive threshold. A fixed value is suitable for scenarios with relatively stable noise environments and is simple to implement. A dynamically adaptive threshold can be dynamically adjusted based on recent environmental information and is suitable for scenarios with significant changes in noise environments. The length of the preset window affects the temporal resolution and stability of density detection. When the preset window length is greater than the preset threshold, higher stability can be provided, but the response is slower. When the preset window length is less than the preset threshold, the response is faster, but fluctuations may be introduced. Therefore, the length of the preset window can be set according to actual needs, and this specification does not limit it in the embodiments.

[0100] In one specific implementation of this specification, a single audio frame to be processed can be 10 milliseconds, a preset length window can be 100 milliseconds, and a preset energy threshold can be set to -20 dB full scale. The short-time energy value of each audio frame to be processed is calculated in real time, and audio frames whose short-time energy value exceeds the preset energy threshold are marked as target audio frames to be processed. Within the preset length window of 100 milliseconds (containing approximately 10 audio frames), the number of target audio frames to be processed is counted, and their density values ​​are calculated. When the density value of a target audio frame to be processed exceeds a preset density threshold, it is determined that there is valid speech activity within the current preset length window, and corresponding second audio detection information is output. The preset density threshold can be set to different levels according to the strictness of detection, such as a lenient threshold and a strict threshold, for different detection scenarios and decision-making needs. The second audio detection information can be the marking information of which of the 10 audio frames to be processed within the preset length window are valid speech.

[0101] For example, when a speaker speaks during interactive voice broadcasting, their audio frames to be processed have high energy. Audio frames exceeding a preset energy threshold are marked as target audio frames to be processed. Within a preset 100-millisecond window, most of the speaker's audio frames to be processed exceed the preset energy threshold, resulting in a high density of target audio frames to be processed (e.g., over 80%), thus indicating valid voice activity. Conversely, when a brief cough occurs in the background, although some audio frames may have short-term energy values ​​exceeding the preset energy threshold and be marked as target audio frames to be processed, the cough's duration is very short (e.g., only 30 to 50 milliseconds), resulting in a low density of target audio frames to be processed within the 100-millisecond window (e.g., only 30% to 50%), thus indicating that it does not constitute valid voice activity and preventing accidental interruptions.

[0102] The two-stage strategy described above, which involves determining the target frame through energy detection and then performing density detection within a preset length window, effectively balances computational efficiency and detection accuracy. It can quickly eliminate non-speech frames while accurately identifying valid speech segments, thus improving the efficiency and accuracy of audio detection.

[0103] In one optional embodiment of this specification, the step of performing energy detection on each audio frame to be processed corresponding to the audio to be processed according to the second detection dimension, obtaining energy detection results, and determining the target audio frame to be processed from each audio frame to be processed based on the energy detection results includes: Based on the second detection dimension, the audio to be processed is subjected to gain normalization to obtain the processed audio. Determine each audio frame to be processed corresponding to the processed audio to be processed, perform low-level speech detection on each audio frame to be processed, and obtain the low-level speech detection result corresponding to each audio frame to be processed, wherein the low-level speech detection result is used to characterize whether each audio frame to be processed is speech. Calculate the energy characteristics for each audio frame to be processed; Based on the energy features and underlying speech detection results corresponding to each audio frame to be processed, the target audio frame to be processed is determined from each audio frame to be processed.

[0104] Gain normalization can be understood as a process of standardizing and adjusting the amplitude of the audio signal to be processed. It can be used to eliminate signal energy differences caused by differences in acquisition equipment, transmission gain fluctuations, or changes in speaking distance, so that audio signals from different sources have comparable energy benchmarks.

[0105] Low-level speech detection can be understood as a preliminary detection process that uses lightweight speech activity detection algorithms to determine whether speech activity exists.

[0106] Energy characteristics can be a quantitative description of the energy properties of an audio signal, and can include characteristic values ​​such as short-time energy value and energy change rate.

[0107] Based on this, gain normalization eliminates the influence of differences in acquisition equipment and transmission gain fluctuations, enabling subsequent speech detection and energy feature calculation to be performed on a unified signal benchmark. Low-level speech detection, as an auxiliary detection method, provides initial detection results of speech activity, offering fundamental information for subsequently determining the target audio frame to be processed. Energy feature calculation is performed on the audio to be processed after gain normalization, reflecting the energy distribution of the audio signal under a standardized benchmark. The determination of the target audio processing frame integrates information from both low-level speech detection results and energy features. When low-level speech detection indicates the presence of speech activity and the energy features meet preset feature conditions, the corresponding audio frame to be processed is determined as the target audio frame to be processed.

[0108] In practical applications, gain normalization can be performed at the frame level or the segment level. Frame-level normalization estimates and adjusts the gain frame by frame, accurately tracking rapid changes in gain, but it is computationally intensive. Segment-level normalization estimates and uniformly adjusts the gain over audio segments of a preset length, offering higher computational efficiency but slower tracking of gain changes. Low-level speech detection can employ adaptive thresholding algorithms, dynamically adjusting the detection threshold based on the statistical characteristics of the audio signal to adapt to different signal-to-noise ratio environments. The calculation of energy features can include the comprehensive calculation of multiple indicators such as short-time energy, energy envelope, and energy gradient. These indicators reflect the energy properties of the audio signal from different perspectives, providing rich feature information for subsequent determination of the target audio frame to be processed.

[0109] In one specific implementation of this specification, frame-level gain normalization can be performed on the input audio to be processed. During normalization, the root mean square energy of the audio signal of each audio frame to be processed is estimated and compared with a preset reference energy value to calculate a gain adjustment coefficient. The gain adjustment coefficient is applied to the audio samples of the current audio frame to be processed, adjusting the signal amplitude to a standardized range. Low-level speech detection is performed on the normalized audio to be processed, using an adaptive energy threshold speech activity detection algorithm. The threshold for low-level speech detection is dynamically updated based on the energy statistics of several recent audio frames to adapt to changes in background noise. Simultaneously, the energy characteristics of the normalized audio to be processed are calculated, including short-time energy and energy change rate. When the low-level speech detection result indicates the presence of speech activity, and the energy characteristics meet the preset energy conditions, the corresponding audio frame to be processed is identified as the target audio frame to be processed and proceeds to the subsequent density detection stage.

[0110] For example, when a user uses voice interaction applications on a smartphone and a smart speaker respectively, the original energy levels of the same speech may differ significantly between the two devices due to differences in microphone sensitivity and acquisition circuitry. Gain normalization adjusts the audio signals acquired by both devices to a unified energy benchmark, allowing subsequent detection algorithms to operate with the same parameter settings without requiring individual parameter tuning for each device. Low-level speech detection provides the system with a rapid preliminary assessment of speech activity. When low-level detection indicates that a particular audio frame contains speech activity and energy features confirm that the frame has a sufficient energy level, it is identified as the target audio frame. Even when speaking from a distance from the microphone causes signal attenuation, gain normalization compensates for this attenuation, ensuring that the detection results are unaffected by distance.

[0111] By employing the above methods, through multi-step collaboration of gain normalization, low-level speech detection, and energy feature calculation, the impact of device differences and gain fluctuations on the detection results is effectively eliminated. This improves the universality and robustness of the detection algorithm across different devices and scenarios, ensuring the accuracy and consistency of the target audio frame recognition.

[0112] In one optional embodiment of this specification, the step of performing density detection on the target audio frame to be processed based on a preset length window, obtaining a density detection result, and determining the second audio detection information corresponding to the audio to be processed based on the density detection result includes: Based on a preset length window, the target audio frames to be processed are arranged in the preset length window; Based on the first density mask and the second density mask, density detection is performed on the target audio frame to be processed in the preset length window to obtain the density detection result, and the second audio detection information corresponding to the audio to be processed is determined according to the density detection result. The first density mask is used to detect whether there is valid speech within the preset length window, and the second density mask is used to detect whether the target audio frame to be processed within the preset length window meets the continuous speech condition.

[0113] A density mask can be understood as a binary label generated based on the audio frame density within a preset length window. The first density mask is generated using a first density threshold (relative density threshold) to indicate whether there is speech activity that meets the basic conditions. Its purpose is to capture as many possible speech events as possible and maintain high detection sensitivity. The second density mask is generated using a second density threshold (strict density threshold), where the second density threshold is greater than the first density threshold. The second density mask is used to indicate whether there is speech activity that meets the conditions for continuous speech. Its purpose is to confirm the continuity and stability of speech activity and provide higher detection reliability.

[0114] Continuous speech conditions can be understood as the continuity standards that speech activities need to meet, such as the duration of effective speech being relatively long or the density detection results meeting a preset density threshold.

[0115] Based on this, a first density mask and a second density mask constitute a two-layer density detection mechanism. The first density mask achieves a fast response with a low threshold, providing preliminary detection results at the very beginning of speech activity, suitable for scenarios requiring high response speed. The second density mask achieves strict confirmation with a high threshold, only providing a positive result when the speech activity lasts for a certain period and the density remains at a high level, suitable for scenarios requiring high accuracy. The combined use of the two density masks forms a logical AND relationship, meaning that the existence of valid speech is only finally confirmed when both density masks simultaneously indicate a positive result. This approach effectively reduces the probability of false detection while maintaining a fast response. In practical implementation, the two density masks can be used independently or flexibly combined according to the needs of the application scenario.

[0116] In practical applications, the specific values ​​of the first and second density thresholds can be set according to the application scenario. For example, in noisy environments, both thresholds can be appropriately increased to reduce false triggers; in scenarios requiring high sensitivity, the first threshold can be appropriately decreased to capture more speech events. The difference between the two thresholds can also be adjusted as needed. A larger difference means a more obvious distinction between the two levels, while a smaller difference makes the detection results of the two levels closer. The update frequency of the density mask is related to the sliding window's step size; for example, the density mask value can be updated every 10 milliseconds.

[0117] In one specific implementation of this specification, the system employs a 10-millisecond frame length and a 100-millisecond sliding window for dual-density mask detection. The first density threshold corresponding to the first density mask is set relatively leniently; for example, it is set when the target frame accounts for more than 40% within the 100-millisecond window, in order to quickly capture the start of speech activity. The second density threshold corresponding to the second density mask is set more strictly; for example, it is set when the target frame accounts for more than 70% within the 100-millisecond window, in order to confirm the continuity of speech activity. The system maintains two independent density states for each time slot (10 milliseconds per slot), forming a circular buffer of 100 slots, covering the audio history of the most recent approximately 10 seconds. When the first density mask and the second density mask are both set within a certain time interval, the system confirms the existence of continuous and valid speech activity within that interval.

[0118] For example, when a speaker utters a short phrase (such as an interruption command) during system broadcast, the density of their speech frame is high within a 100-millisecond window. The first density mask is set shortly after the speech begins, indicating that possible speech activity has been detected. As the speech continues, the target frame density remains at a high level, and the second density mask is also set, indicating that the speech meets the continuity condition. If someone nearby speaks a sentence briefly (lasting approximately 200 milliseconds), although the first density mask may be set shortly after the speech begins, the second density mask does not meet the setting condition for most of the time window due to the short duration of the speech. Based on this, the system determines that the speech does not meet the continuity condition, thus effectively distinguishing between the speaker's continuous speech and the brief speech of others.

[0119] Combining the aforementioned low-level speech detection and energy feature calculation, the low-level speech detection result can serve as the first criterion. To suppress isolated transients and noise reduction residue, energy features can be calculated for each audio frame to be processed as the second criterion. The energy feature can be at least one of the following: frame peak amplitude (a temporal feature indicator, the maximum amplitude of all sampling points in the audio frame to be processed), root mean square energy, quantile energy, effective sample percentage, or frequency band energy. When the first criterion indicates speech, and the energy feature calculated by the second criterion is not lower than a preset energy threshold, the audio frame to be processed is recorded as a valid hit, i.e., the audio frame to be processed is determined as the target audio frame to be processed. Understandably, the first and second criteria can be logically ANDed, with the second criterion acting as a tightening mechanism based on the first criterion, preventing the introduction of new sources of false triggers. Subsequently, the number of hits can be accumulated within a preset length window, and the continuous hit length (i.e., the continuous length of the target audio frame to be processed) and the peak value within the window can be recorded simultaneously. A circular mask can be written in units of the preset length window. Specifically, two cryptographic masks with different densities—a first (loose density mask) and a second (strict density mask)—can be maintained in parallel on the same ring structure. The loose density mask is used to determine whether valid speech exists within a preset length window. It allows for a small number of non-missing audio frames within the preset length window, thus tolerating natural pauses and slight fluctuations in speech. The strict density mask requires that all audio frames within the preset length window satisfy the speech condition, meaning that all audio frames within the preset length window are target audio frames. It is used to determine whether stable and continuous speech is constituted within the preset length window, thus providing stronger suppression of transient interference such as keyboard clicks, clinking utensils, and single plosive sounds. Furthermore, the loose density mask superimposes an energy condition (i.e., the aforementioned energy feature calculation) on top of the speech decision, while the strict density mask does not superimpose this energy condition. Its speech condition can be given by a level decision with hysteresis and silence countdown, thus employing the most suitable criteria for both existence and continuity. The loose density mask and the strict density mask can be statistically analyzed separately by a unified underlying detector, or they can be driven separately by two detectors with different sensitivities or operating modes. During the final query, the two density masks can be logically ANDed together, making the existence of valid speech and the achievement of valid duration two independent and non-substitutable necessary conditions. For example, the audio frame to be processed can be 10 milliseconds, and the preset length window is 100 milliseconds. When the frame peak amplitude is not lower than approximately -20 dBFS (i.e., the preset energy threshold), and at least 6 audio frames to be processed within the preset length window simultaneously meet the speech and energy conditions, the loose density mask is set to 1; when all 10 audio frames to be processed within the preset length window meet the speech condition, the strict density mask is set to 1. The aforementioned preset energy threshold can be calibrated based on the sampling bit width, noise floor, device gain, pickup distance, or historical valid speech levels.

[0120] By employing the above method and a dual-layer detection mechanism using a first density mask and a second density mask, a refined distinction can be made between speech activities from both time and continuity dimensions. This effectively differentiates continuous speech activities from brief speech interference, improving the accuracy and anti-interference capability of speech detection and providing a reliable detection basis for subsequent interruption decisions.

[0121] Step 206: Based on the audio detection information and the reference audio information corresponding to the target object, determine the audio processing result corresponding to the audio to be processed, wherein the audio processing result is used to determine the interaction strategy corresponding to the interaction process, and the audio processing result includes the target audio corresponding to the target object and / or the audio detection signal corresponding to the audio to be processed, wherein the target audio is obtained by extracting the audio to be processed.

[0122] The audio detection signal is used to characterize the relationship between the audio to be processed and the target object. Reference audio information can be understood as the audio information of the target object, which may be audio information pre-recorded by the target object or obtained during target object registration. This reference audio information can be the target object's reference audio, the target object's acoustic embedding vector, or other forms of identity feature representation. The reference audio information is used to provide an identity reference for the target object during audio processing, so that the audio processing result not only reflects the audio features of the audio to be processed itself, but also reflects the identity association between the audio to be processed and the target object.

[0123] Audio processing results can be understood as the results obtained through comprehensive analysis of audio detection information and reference audio information, used to determine the interaction strategy corresponding to the interaction process. The audio processing results may include the target audio corresponding to the target object and the audio detection signal corresponding to the audio to be processed.

[0124] Target audio can be understood as the audio segment of the target object extracted from the audio to be processed. For example, it can be an audio segment identified as the speech of the target object that can be extracted from the audio to be processed. The audio detection signal is used to characterize the relationship between the audio to be processed and the target object. For example, it can be used to characterize whether the audio to be processed contains the speech of the target object, the degree of matching between the audio to be processed and the target object, etc.

[0125] Interaction strategy can be understood as the next interaction strategy between the target object and the interactive end. It can include strategies such as whether to trigger an interruption response, whether to continue the current voice broadcast, whether to adjust the voice broadcast content or voice broadcast volume, and whether to enter a new dialogue round.

[0126] In an optional embodiment of this specification, the audio processing method further includes: Receive reference audio information corresponding to the target object; or In response to the registration request of the target object, the reference audio corresponding to the target object is collected; Information is extracted from the reference audio to obtain the reference audio information.

[0127] The reference audio information can be used to characterize the acoustic identity features of the target object. This reference audio information can be, for example, acoustic vectors, speaker embeddings, or identity feature templates. Understandably, the reference audio information can be a pre-obtained identity representation of the target object, which can be acquired before receiving the audio to be processed from the target object during the interaction process.

[0128] Optionally, it can receive reference audio information of the target object sent by an external system, such as the target object's account system, conferencing system, customer service system, terminal device, etc.

[0129] Alternatively, the target object can be registered locally within the interactive terminal, and reference audio of the target object can be collected and reference audio information can be extracted from the reference audio through the registration process.

[0130] During the registration process, the audio data uploaded by the target can be filtered, and audio frames that simultaneously meet the speech judgment and average level thresholds are written to the registration buffer. Registration progress is accumulated according to the number of qualified frames, making local registration robust to intermittent speech and silent segments. After registration begins, an initial audio segment can be skipped to exclude the interval where the device's own playback and echo have not yet converged. Reference audio information can be temporarily effective within a single session between the target and the interactive end, or it can be saved across conversations. It can be kept fixed or adaptively updated based on subsequent reliable speech using moving average, weighted fusion, or confidence control methods.

[0131] Based on this, the dual-channel acquisition mechanism enables the system to flexibly adapt to different deployment scenarios and usage conditions. The external acquisition method can be used for system integration scenarios with existing identity databases, while the local registration method can be used for first-time use or scenarios requiring on-site data collection. The qualified frame accumulation mechanism ensures the registration process's tolerance for user behavior patterns.

[0132] In practical implementation, it can also support adaptive updates of identity representations, continuously updating reference audio information based on subsequent trusted speech after initial registration. Furthermore, it can support the simultaneous storage of multiple identity representations, suitable for scenarios where multiple people share a device or the speaker changes.

[0133] In one specific implementation of this specification, the speaker's identity representation can be either directly issued by an external system or generated during the local registration phase. After registration begins, approximately 2 seconds of initial audio are skipped. Subsequently, only frames that simultaneously meet the speech judgment and average level threshold are written to the registration buffer. Registration is completed after approximately 12 seconds of valid speech. The speaker vector supports export and external injection; during injection, the registration failure flag is cleared.

[0134] For example, in a scenario where an intelligent customer service robot is deployed, a registration request is triggered when a customer service representative logs in for the first time. The system skips the initial 2 seconds and then begins accumulating valid voice frames. The representative speaks a few sentences intermittently, and after the system accumulates approximately 12 seconds of valid voice based on the number of valid frames, registration is completed, and a 192-dimensional acoustic vector is extracted and generated. If a pre-registered acoustic vector already exists, it can also be injected directly through an external interface.

[0135] In summary, through a flexible reference audio information acquisition mechanism, the system can adapt to various deployment scenarios and usage conditions, providing a convenient initialization experience while ensuring the reliability of the identity benchmark.

[0136] In one optional embodiment of this specification, the step of extracting information from the reference audio to obtain the reference audio information includes: The reference audio is divided into multiple audio units, and the audio similarity information between the multiple audio units is calculated. If the audio similarity information is determined to meet the consistency condition, information is extracted from the reference audio to obtain the reference audio information; If the audio similarity information does not meet the consistency condition, a registration failure message is generated.

[0137] In this context, an audio unit can be understood as a segment obtained by dividing the accumulated valid speech (i.e., reference audio) during the registration period into segments of fixed length. Each audio unit independently extracts a speaker embedding vector for similarity comparison. Audio similarity information reflects the degree of consistency of speaker features between different audio units and can be calculated using cosine similarity, Euclidean distance, or other metrics. The consistency condition is a threshold criterion for determining whether the registration environment contains only a single speaker.

[0138] In calculating audio similarity information, in some optional embodiments, audio similarity information between any two audio units can be calculated, while in other optional embodiments, audio similarity information between one audio unit and other audio units can be calculated one by one, based on one of the audio units.

[0139] Therefore, if the audio similarity information meets the consistency condition, it can be understood that all audio similarity information is greater than the preset similarity threshold. If the audio similarity information does not meet the consistency condition, it can be understood that there is any audio similarity information that is less than the preset similarity threshold.

[0140] For example, approximately 12 seconds of valid speech (i.e., reference audio) can be divided into up to 6 segments, each 2 seconds long. A 192-dimensional embedding is extracted from each segment. Using one segment as a benchmark, the cosine similarity between each of the remaining segments and that benchmark is calculated. If any similarity score is below a consistency threshold (e.g., a cosine similarity of approximately 0.35), it is determined that the registration environment may contain multiple speakers or that the data is unstable. Registration failure information is a flag signal output by the system when the consistency condition is not met. Based on this, the system refuses to generate or update the speaker vector and enters a degraded operation mode. Alternatively, pairwise similarity calculations can be performed between each segment to obtain a more comprehensive consistency assessment.

[0141] Based on this, the environmental purity self-checking mechanism is used to detect whether there is only one speaker in the registration environment, thus blocking the subsequent entire link from the source by vector-driven contamination by multiple people's characteristics.

[0142] In practical implementation, a progressive self-check can be adopted. During the registration process, incremental comparison is performed every time a new audio unit is accumulated. If early audio units show inconsistencies, registration is terminated early to save time and computing power. In addition, a confidence level mechanism can be introduced to give a level evaluation of registration quality based on the distribution of similarity, enabling the system to take differentiated follow-up strategies according to different levels.

[0143] In one specific implementation of this specification, approximately 12 seconds of valid voice data are registered, divided into up to 6 segments every 2 seconds. Each segment is then used to extract a 192-dimensional embedding, with a consistency threshold of approximately 0.35 cosine similarity. These values ​​can be calibrated according to the model and scenario. If any similarity score falls below the threshold, the registration environment is deemed unqualified, the generation of the speaker vector is refused, a registration failure flag is set, and the system enters a downgraded operation.

[0144] For example, in an open office environment, a customer service representative is registering while colleagues are whispering nearby. After segmenting the 12-second valid audio into six segments, the system finds that the fourth segment has a similarity of only 0.22 to the baseline, far below the threshold of 0.35. Based on this, the system determines the registration environment is unqualified, refuses to generate a speaker vector, and enters a downgraded operation. After the colleague leaves, the representative re-registers; this time, the similarity between all segments is above the threshold, and registration is successful.

[0145] In summary, through the self-checking mechanism for the purity of the registration environment, the system can automatically identify and reject identity benchmarks contaminated by multiple user characteristics, thus ensuring the reliability of subsequent end-to-end decisions from the source.

[0146] In one optional embodiment of this specification, determining the audio processing result corresponding to the audio to be processed based on the audio detection information and the reference audio information corresponding to the target object includes: Based on the audio detection information and the reference audio information corresponding to the target object, the current audio state corresponding to the audio to be processed is determined; Based on the current audio state, determine the audio processing result corresponding to the audio to be processed.

[0147] The current audio state is a description of the stage in which the current audio segment is located, and it aggregates the low-level decision results frame by frame into a stable state representation.

[0148] In an optional embodiment, the current audio state can include states such as silent, candidate, main speaker, exit hold, and rejection. The silent state indicates that no valid voice activity has been detected; the candidate state indicates that a voice signal meeting some conditions has been detected but has not yet passed all verifications; the main speaker state indicates that all conditions have been met, confirming the current voice as the speaker; the exit hold state represents a transition period after the main speaker's voice ends, during which if voice resumes, the state returns to the main speaker state; the rejection state indicates that the current voice is determined not to belong to the speaker. Different thresholds or hold durations are used for transitions between states to create a hysteresis effect, avoiding frequent state jumps due to signal fluctuations near critical values. The audio processing result is an output signal generated based on the current audio state, and the service layer generates an interruption event only once when first entering the main speaker state.

[0149] In addition, an interface for backtracking queries by time interval is provided, decoupling frame-by-frame audio processing from the upper-layer sentence boundary determination. The circular mask continuously scrolls and records according to window slots. After obtaining the start and end times of a sentence, the upper layer can subsequently query whether valid speaker voice meeting all conditions has appeared within that interval, without needing to synchronously complete interruption decisions within the audio thread. The two query paths handle interval boundaries differently: "Whether valid voice exists" scans segment by segment on the circular structure according to the requested start and end times, requiring handling of circular wraparound and providing full coverage for excessively long intervals exceeding the circular span; "Whether the valid duration is reached" is not constrained by the requested boundary and is statistically analyzed across the entire circular span to prevent persistent evidence falling before the sentence beginning from being truncated and discarded by the interval. Duration can be accumulated sentence by sentence, and the accumulator is reset sentence by sentence after detecting the sentence end to avoid cross-sentence accumulation. The system can also maintain statistics such as the number of sentence segments, audio frames, rejected sentences, and interruption counts for upper-layer performance evaluation and problem localization. For example, a ring mask consists of 100 slots of 100 milliseconds each, covering a total of approximately 10 seconds. This interface enables interruption decisions to be reviewed and traced by upper-layer services on a sentence-by-sentence basis, thereby avoiding occasional fluctuations in frame-by-frame decisions from directly altering the service state.

[0150] Based on this, the state machine architecture transforms the frame-by-frame decision-making of multi-dimensional evidence into stable state control, enabling interruption decisions to be reviewed and traced by upper-layer services on a sentence-by-sentence basis, avoiding the occasional fluctuations in frame-by-frame decisions from directly changing the service state. Entry and exit employ different threshold hysteresis designs, effectively preventing jitter in critical states.

[0151] In practical implementation, more intermediate states can be introduced to support finer-grained control logic. For example, pre-candidate states can be added to prepare subsequent processing resources in advance when voice activity first appears. Alternatively, a probabilistic state model can be used instead of a deterministic state machine, assigning a confidence score to each state, enabling upper-layer services to make more flexible decisions based on confidence levels.

[0152] In one specific implementation of this specification, when determining the current audio state corresponding to the audio to be processed based on the audio detection information and the reference audio information corresponding to the target object, the fusion input includes the duration corresponding to the loose density mask and the strict density mask, the speaker's identity information evidence, and the environmental noise level. Optionally, the remote playback status and echo residual information can be included. The output is the current audio state corresponding to the audio to be processed. The fusion method can employ logical and formal hierarchical gating, and in other embodiments, it can be replaced with weighted scoring, a probabilistic model, or a lightweight classifier trained based on labeled data. The system also maintains statistics such as the number of sentence segments, the number of speech frames, the number of rejected sentences, and the number of interruptions for upper-layer performance evaluation and problem localization.

[0153] For example, in a noisy office environment, the system is currently in broadcast mode. When others start talking, the underlying speech detection may be triggered, and the system enters a candidate state from silence. However, because the strict density mask and identity information evidence are not simultaneously valid, it will not enter the main speaker state. Subsequently, the speaker begins to speak, and as various conditions are gradually met, the system enters the main speaker state and generates an interruption event. After the speaker finishes speaking, there is a short pause, and the system enters an exit hold state. If no valid speech is received within the hold duration, it returns to silence.

[0154] In summary, through the hierarchical management mechanism of audio status, the system organizes the frame-by-frame judgment of multidimensional evidence into stable high-level control signals, thereby achieving accuracy in interruption judgment and stability in service status switching.

[0155] In one optional embodiment of this specification, the audio to be processed includes a plurality of audio frames to be processed; The process of receiving the audio to be processed from the target object during the interaction includes: The target object receives the audio to be processed during the interaction process in a streaming manner; The step of detecting the audio to be processed based on at least one detection dimension to obtain audio detection information corresponding to the audio to be processed includes: Based on at least one detection dimension, each audio frame to be processed is detected to obtain audio detection information corresponding to each audio frame to be processed. The step of determining the audio processing result corresponding to the audio to be processed based on the audio detection information and the reference audio information corresponding to the target object includes: The audio processing result corresponding to the audio to be processed is determined based on the audio to be processed, the audio detection information corresponding to each audio frame to be processed, and the reference audio information corresponding to the target object.

[0156] Streaming reception refers to a working mode where audio to be processed arrives continuously in units of audio frames and is processed by the system, unlike the offline mode where the entire audio segment is recorded and then processed in batches. In this embodiment, the duration of the audio frame to be processed can be 10 milliseconds. For detectors that take longer frames as input, adjacent short frames can be spliced ​​together and sent into inference, for example, spliced ​​into a 20-millisecond frame. Frame-by-frame detection means that the detection process is executed independently when each audio frame arrives, including operations such as voice activity detection, energy calculation, and mask update.

[0157] The determination of the audio processing result depends not only on the detection result of the current frame, but also on the accumulated information of historical frames and reference audio information, thereby achieving real-time performance at the frame level while maintaining the statistical robustness of the decision. The circular mask is continuously recorded by scrolling through window slots; in this embodiment, it consists of 100 slots of 100 milliseconds each, covering approximately 10 seconds in total.

[0158] Based on this, the streaming processing architecture enables the system to perform detection and decision-making frame by frame as audio arrives, meeting the stringent low-latency requirements of real-time voice interaction. The design combining frame-by-frame detection with statistical windows ensures real-time performance while improving the reliability of decisions through the accumulation of time data.

[0159] In practice, the frame length and processing cycle can be adjusted according to the latency requirements of different scenarios. For example, a shorter frame length can be used in scenarios with extremely strict latency requirements. An asynchronous processing pipeline can also be used to decouple front-end frame-by-frame detection from back-end identity recognition and other time-consuming operations, ensuring that the real-time performance of the front-end is not affected by back-end processing latency.

[0160] In one specific implementation of this specification, the system acquires near-end audio from the microphone, unifies it to a predetermined sampling rate, and divides it into consecutive short frames. Environmental estimation and dynamic parameter updates can also be performed according to the concatenated timeframe, thereby reducing inference frequency without increasing buffer latency. The system also maintains a circular pre-buffer, the length of which covers the decision latency required for multi-level decisions and the sentence beginning to be backfilled.

[0161] For example, in a smart speaker scenario, the speaker continuously receives audio from the microphone in 10-millisecond frames, independently performing voice activity detection and energy calculation for each frame. A valid hit occurs when a frame simultaneously meets both the voice detection and energy thresholds. The number of hits is accumulated within a 100-millisecond statistical window, and the ring mask is updated. Through frame-by-frame processing and window accumulation, the system can track audio state changes in real time.

[0162] In summary, through the processing architecture of streaming reception and frame-by-frame detection, the system not only meets the low latency requirements of real-time voice interaction, but also ensures the accuracy and stability of the judgment through statistical accumulation over time.

[0163] In one optional embodiment of this specification, after determining the audio processing result corresponding to the audio to be processed based on the audio detection information and the reference audio information corresponding to the target object, the method further includes: Based on the audio processing results, the interaction strategy corresponding to the target object is determined and executed.

[0164] The interaction strategy can be understood as the interaction behavior strategy determined based on the audio processing results, which is used to determine the response method for the current audio to be processed.

[0165] Interaction strategies can include interruption strategies, maintenance strategies, degradation strategies, and recovery strategies. An interruption strategy can be understood as the system suspending the current voice playback and switching to receiving and processing speaker input when audio processing results indicate that the current voice does not belong to the speaker or does not constitute a valid interruption. A degradation strategy can be understood as the system degrading to a lightweight decision mode based on voice activity detection and duration when speaker identity information is unavailable or the detection system malfunctions. A recovery strategy can be understood as the system restoring full identity gating capabilities when the system, previously degraded due to unavailable identity information, regains reliable identity information.

[0166] Further, determining and executing the interaction strategy corresponding to the target object based on the audio processing result includes: If the audio detection signal included in the audio processing result indicates that the audio to be processed does not belong to the target object, the interaction strategy corresponding to the target object is determined to be not to trigger the interruption of the interaction process, and the interaction strategy is executed.

[0167] Specifically, if the audio detection signal indicates that the audio to be processed does not belong to the target object, it means that the audio to be processed is not the audio of the target object speaking. In this case, it can be determined that the interaction strategy is not to trigger the interruption of the interaction process.

[0168] Understandably, the decision-making unit can determine the interaction strategy based on the audio detection signal and the semantic understanding results corresponding to the audio to be processed and the target audio.

[0169] The execution of the interaction strategy involves the coordinated operation of multiple aspects, such as system broadcast control, audio output path, and state machine transition.

[0170] Based on this, the design of transforming audio processing results into specific interaction strategies enables the system to make differentiated responses based on different audio states. The abstraction at the strategy level keeps the interface between the audio detection module and the service control module clear, which facilitates the modular design and maintenance of the system.

[0171] In practice, richer interaction strategies can be customized based on product form and service needs, such as adding a directional audio pickup strategy in multi-person meeting scenarios. Strategy priority management can also be introduced, allowing the selection of the strategy to be executed according to preset priority rules when multiple strategies are applicable simultaneously.

[0172] In one specific implementation of this specification, the service layer generates an interruption event only once when the speaker first enters the main speaking state to avoid triggering it repeatedly frame by frame. The system also provides an interface for backtracking queries by time interval. When the speaker has not yet registered or the registration environment self-check fails, the system degrades to a lightweight solution with personalized enhancements and automatic bypass, while the frame length, buffer timing, and external interface of the link remain unchanged.

[0173] For example, in an intelligent customer service scenario, the system is broadcasting a product introduction. After the speaker begins speaking, the system's voice is confirmed as valid after multiple levels of judgment. Upon receiving an interruption signal, the voice control center stops broadcasting and sends the enhanced audio to the speech recognition module for transcription. If someone interrupts at this time, the system determines that it is not the speaker's voice and executes a maintenance strategy to continue broadcasting.

[0174] In summary, by transforming audio processing results into differentiated interaction strategies and executing them, the system can make accurate and targeted responses based on the audio state, ensuring the naturalness and accuracy of voice interaction.

[0175] See Figure 4 , Figure 4 This diagram illustrates a processing flowchart of an audio processing method according to an embodiment of this specification within a speaker recognition module, as shown below. Figure 4As shown, the speaker recognition module can acquire near-end microphone audio as the audio to be processed (i.e., near-end audio input), and optionally acquire far-end speaker reference, system playback status, echo residual, device gain, sampling rate, scene mode (i.e., auxiliary input reference, playback status device gain, and scene mode), as well as externally sent speaker identity representation. The audio to be processed is unified into consecutive short frames at a preset sampling rate, such as 10-millisecond audio frames. For detectors that take audio frames longer than 10 milliseconds as input, adjacent short frames can be concatenated and sent into inference. For example, 10-millisecond audio frames can be concatenated into 20-millisecond audio frames, thereby achieving near-end audio input framing and state synchronization.

[0176] Before the speaker recognition module is connected to the automatic speech recognition module in parallel branches, the audio signal is split into two paths for processing before entering the speech recognition module. One audio signal maintains the original processing flow and directly enters the speech recognition module for speech transcription; the other audio signal enters the speaker recognition module for identity detection and speech enhancement processing. The speaker recognition module outputs two results: one is the enhanced audio, which is sent to the speech recognition module for transcription to reduce missed interruptions caused by suppressed speech; the other is the speaker signal, which is sent to the speech control module for interruption decision to reduce false interruptions caused by non-speaker speech. This parallel branch architecture allows identity information and speech enhancement results to be transmitted independently without affecting the original speech recognition processing flow.

[0177] In continuous writing with pre-buffered operation, the system uses frames of fixed duration as the basic processing unit, such as 10 milliseconds per frame, and is equipped with a circular pre-buffer to cache recent audio data. The pre-buffer depth can be set to approximately 700 milliseconds, meaning it can cache data from the most recent 70 audio frames. When an interruption decision is made, the system can retrieve the audio data from the pre-buffer before the decision, ensuring that the beginning of the interrupted speech is not truncated and improving the integrity of speech recognition. The circular buffer uses a first-in, first-out (FIFO) data management method; when the buffer is full, newly written data overwrites the oldest data, ensuring efficient use of buffer space.

[0178] In terms of audio processing architecture, the system adopts a three-channel parallel audio processing architecture (i.e., preprocessing and three-channel audio hold), processing the near-end raw audio, general noise-reduced audio, and personalized audio enhancement respectively. The near-end raw audio branch holds the acquired raw signal without additional processing, providing a reference for modules that require raw data. The general noise reduction branch uses a general noise reduction algorithm to process the audio, removing non-speech interference components such as environmental noise, suitable for general speech activity detection. The personalized enhancement branch uses the reference audio information of the target object as a condition to perform targeted enhancement processing on the audio, preserving and enhancing speech components that match the acoustic features of the target object, while suppressing the speech and noise of other speakers. The parallel processing results of the three audio channels provide rich data support for subsequent downstream modules such as speech recognition and interruption decision.

[0179] In the environmental estimation and dynamic threshold strategy, the environmental noise estimation module estimates the noise intensity and spectral characteristics of the current environment in real time and dynamically adjusts the detector's operating parameters based on the noise level. Environmental noise estimation is updated during system broadcasts or when continuous silence is detected to avoid erroneously including speech signals in the noise estimation. The detector employs an integer-based sensitivity threshold design, discretizing the continuous parameter space into a finite number of thresholds, each corresponding to a set of preset detection parameters. This design facilitates parameter management and debugging while ensuring the robustness and reproducibility of parameter adjustments. Parameter adjustment uses a smooth, rapid-attack, slow-release strategy: when noise suddenly increases, the detection threshold is quickly raised to avoid false triggering; when noise decreases, the threshold is slowly restored to maintain detection sensitivity.

[0180] In gain normalization and energy gating, as well as effective speech determination based on loose density masks and duration determination based on strict density masks, the gain normalization module performs gain normalization processing on the audio signal to eliminate gain differences caused by differences in acquisition equipment, transmission link fluctuations, or changes in speaking distance, enabling subsequent speech detection algorithms to work on a unified signal benchmark. The speech detection module adopts a layered detection strategy of energy gating plus dual-density hit verification. Energy gating, as the first layer of detection, performs threshold judgment on the short-term energy of audio frames. When the frame energy is lower than a preset threshold (e.g., -20 dB full scale), the frame is directly determined to be a non-speech frame, thereby quickly eliminating low-energy silent frames and weak noise frames. Dual-density hit verification, as the second layer of detection, counts the density of target frames with energy exceeding the threshold within a sliding window of a preset time length. The first density mask uses a relatively loose density threshold to quickly capture possible speech activity; the second density mask uses a relatively strict density threshold to confirm the continuity and stability of speech activity. The collaborative work of the two layers of detection achieves a balance between fast response and accurate judgment.

[0181] During the process of acquiring the speaker's identity, the registration management module is responsible for collecting the speaker's reference audio and extracting the acoustic embedding vector. During registration, the system skips the initial audio interval (e.g., skipping the first approximately 2 seconds) to avoid transient interference during system startup affecting registration quality. A certain duration of reference audio is accumulated based on the number of qualified frames (e.g., approximately 12 seconds), then divided into multiple segments of fixed length (e.g., 2-second segments, up to 6 segments). A fixed-dimensional embedding vector (e.g., a 192-dimensional embedding vector) is calculated for each segment, and the cosine similarity between the embedding vectors of each segment is checked to ensure it meets the consistency condition (i.e., purity self-check). The consistency condition can be set to a cosine similarity not lower than a preset threshold (e.g., 0.35). When the consistency condition is met, the final reference audio information is extracted for subsequent audio processing; when the consistency condition is not met, registration fails, and the user is prompted to re-register.

[0182] In personalized speech enhancement, the personalized enhancement module performs personalized enhancement processing on the audio based on the speaker's acoustic embedding. The core idea of ​​the enhancement processing is to utilize the acoustic feature information of the target object to generate an enhancement mask in the frequency domain or time-frequency domain. This mask is then applied to the input audio signal to preserve speech components that match the acoustic features of the target object and suppress speech and noise interference from other speakers. Post-mask processing includes operations such as banded gain adjustment, clipping, and power-law sharpening to further optimize the enhancement effect. Banded gain adjustment divides the value of the enhancement mask into multiple intervals, each corresponding to a different gain adjustment strategy; clipping is used to prevent clipping distortion in the enhanced signal; power-law sharpening enhances the contrast of the mask through nonlinear transformation, making the distinction between the target speech components and interference components more obvious. The enhancement module uses a stateless computation graph to implement streaming inference, processing each frame independently without caching historical frame data, thus reducing processing latency and memory usage.

[0183] After speaker selection is completed in the signal domain, the speaker embedding, the frequency domain features or time domain frame of the current frame, and the model state carrying historical information can be input into the enhancement model. The model then outputs a complex mask, a time domain mask, or the enhancement result (i.e., the target video) for the target speaker. The suppression target is no longer limited to noise, but includes the speech of other real people, which is the main source of accidental interruptions in multi-speaker scenarios.

[0184] The mask output by the enhancement model can undergo adaptive post-processing on the host side: applying different gains according to the output amplitude, limiting the amplitude, applying power-law sharpening to the amplitude, and then multiplying it by the complex number of the input spectrum. This post-processing is located outside the enhancement model, thus achieving a trade-off between suppression strength, noise amplification, and speech impairment according to the scenario without retraining the model. The enhancement model can be derived as a stateless computation graph, whose recursive state is carried and written back frame by frame by frame by the caller in a fixed-length buffer. With the input and output two-level refraction buffers, any upper-level frame length can be mapped to the model's fixed inference step size, thereby realizing streaming causal inference. Finally, the enhancement processing can be enabled when the speaker has completed registration and automatically bypassed when not registered, thus not introducing additional computational overhead or causing unnecessary damage to the audio.

[0185] In logical AND merging interval backtracking query and multi-evidence fusion with anti-interruption state control, the multi-evidence fusion module comprehensively judges evidence from different detection dimensions. Fusion strategies include logical AND gating, weighted scoring, and classifier fusion. Logical AND gating requires all detection dimensions to simultaneously meet conditions before outputting a positive decision, offering high accuracy but potentially reduced sensitivity. Weighted scoring assigns different weights to the results of each detection dimension and calculates a comprehensive score; when the score exceeds a threshold, a positive decision is output, allowing for flexible adjustment of the influence of each dimension. Classifier fusion uses machine learning models to classify and judge multi-dimensional detection results, capturing complex relationships between dimensions. The state machine module maintains multiple detection states, including silent state, candidate state, main speaker voice state, exit hold state, and rejection state, transitioning between states based on the fusion decision. The interval backtracking query module queries detection information within historical time intervals, supporting state machine decision-making. The system maintains a certain number of time slots (e.g., 100 100-millisecond slots) covered with a circular mask over the most recent preset time period (e.g., the last 10 seconds), providing data support for state judgment and interval backtracking.

[0186] In the output control backfilling and sentence-end locking, the pre-buffer backfilling module retrieves the sentence-beginning audio from the pre-buffer when the judgment is valid, ensuring the integrity of the interrupted speech. When the state machine transitions from the candidate state to the main speaker state, the system retrieves the audio data cached before the judgment was valid from the pre-buffer and concatenates it with the currently received audio data to form a complete interrupted speech segment. The withholding output module employs a two-level strategy to control the output of detection results, achieving separate management of the detection branch and the output branch. The first level is the internal caching and confirmation of the detection results, and the second level is the external transmission of the confirmed results. This two-level strategy can prevent unstable detection results from affecting downstream modules. The sentence-end locking mechanism is used to prevent the speech of other speakers from being mistakenly judged as new interruption events in the short period after the main speaker's speech ends.

[0187] When the speaker's identity information is unavailable (e.g., registration information is lost or the acoustic embedding vector quality degrades), the system provides a degradation mechanism, automatically degrading to a lightweight detection scheme by bypassing the unregistered state. This lightweight detection scheme does not rely on acoustic embedding information, making interruption decisions solely based on energy detection and speech activity detection. While accuracy is slightly reduced, it ensures basic system functionality. During degradation, the external interface remains unchanged, ensuring that upper-layer service logic is unaffected. This degradation mechanism enhances the system's robustness and fault tolerance.

[0188] See Figure 5 , Figure 5 A schematic diagram of a timing scenario in an audio processing method according to an embodiment of this specification is shown, such as... Figure 5 As shown, in a specific time-series scenario, the interactive terminal starts broadcasting content at time t0, and the circular pre-buffer is immediately written in, with no samples generated uplink at this time. At time t1, a person speaking briefly nearby may satisfy the underlying speech decision in the detection branch, but its level and duration are difficult to pass both energy gating and strict density verification simultaneously, so the speaker signal is not set; at the same time, this component has been suppressed in the output branch by extraction conditioned on the speaker's identity. The voice control center continues broadcasting due to the lack of this acoustic evidence, and the broadcast is not interrupted by irrelevant sound sources. At time t2, the speaker starts speaking, intending to interrupt, and the hits in the detection branch begin to accumulate, but the joint conditions are not yet fully met, and the output is still in a withheld state—for this reason, the initial audio of the sentence between t2 and t3 is completely retained in the pre-buffer. At time t3, the interactive terminal completes the interruption decision. Both the loose density mask and the strict density mask are valid, and the identity information evidence is valid. The decision is complete, and the output changes from withholding to backfilling, retrieving the sentence beginning from the pre-buffered buffer and starting continuous uplink. At time t4, the interruption takes effect and playback stops. The delay from the speaker speaking (t2) to the interruption taking effect (t4) is controlled within a small range, ensuring timely interruption response and user experience. During this timeline, the voice of a bystander is detected by the system at time t1, but is marked as rejected due to identity mismatch and does not trigger an interruption. After the speaker's voice is detected at time t2, the system confirms the speaker's identity at time t3 after a candidate state confirmation process, and enters the speaker's voice state, finally completing the interruption response at time t4. The decision between t1 and t3 is intentionally tightened to improve accuracy, and the resulting response delay and sentence beginning loss are compensated by pre-buffering and backfilling on the output side. Therefore, the improvement in accuracy does not come at the cost of a decrease in the interactive experience.

[0189] Based on this, the audio processing method provided in the embodiments of this specification, by introducing the identity information of the target object at the speech detection stage, upgrades the target of interruption judgment from the existence of speech to the existence of valid speech of the speaker, effectively solving the problem in existing solutions where identity information is lost during the transcription stage, resulting in the inability to distinguish the speaker early. The processing structure employs four stages: identity representation acquisition, identity-driven suppression, hierarchical robust confirmation, and temporal compensation output. These stages work together to achieve accurate and timely interruption detection. The identity representation acquisition stage obtains the speaker's acoustic embedding vector through the registration management module, providing conditional information for subsequent identity-driven suppression. The identity-driven suppression stage enhances the audio using a personalized enhancement module with acoustic embedding as a condition, suppressing speech interference from non-speakers. The hierarchical robust confirmation stage uses a multi-level detection mechanism (energy gating, density verification, state machine decision, etc.) to perform multi-dimensional confirmation of the enhanced audio, ensuring the accuracy and stability of the judgment result. The temporal compensation output stage ensures the integrity of the interrupted speech and the timeliness of the output through pre-buffering backfilling and a two-level output strategy.

[0190] For example, in a smart speaker application scenario, the speaker (i.e., the registered user) completes acoustic registration beforehand, and the system stores the speaker's acoustic embedding vector as reference audio information. When the speaker is broadcasting a weather forecast, the speaker gives an interrupt command, such as "What about tomorrow?". After receiving the audio to be processed containing the speaker's voice, the system first estimates the environmental noise and adjusts the detector parameters. Then, it quickly filters out audio frames with high energy using energy gating, and confirms the validity of the filtered audio frames using a dual-density hit check. Simultaneously, the personalized enhancement module enhances the audio to be processed based on the speaker's acoustic embedding, extracting speech components that match the speaker's acoustic features. The multi-evidence fusion module integrates the results of energy detection, density detection, and identity detection. After confirming that the current voice comes from the speaker, the state machine transitions from a silent state through candidate states to the speaker's voice state, triggering an interruption response. The pre-buffering backfilling module retrieves the first-sentence audio from the buffer, concatenates it with the current audio, and sends it to the speech recognition module for transcription. If someone else speaks during the broadcast, their voice may pass energy gating and partial density detection, but because their acoustic features don't match the speaker's, the personalized enhancement module will suppress their speech components. The identity detection result will then fail, the state machine will enter a rejection state, and no interruption response will be triggered. In this way, the system can robustly detect the speaker's interruption intent in multi-speaker environments, significantly improving the user experience of full-duplex voice interaction.

[0191] In summary, the audio processing method provided in this specification, by upgrading the interruption judgment target from the presence or absence of speech to the presence or absence of valid speech from the speaker, introduces the target object's identity information during the speech detection stage, effectively solving the technical problems of lost identity information and inability to distinguish between speaker and non-speaker speech in existing solutions. Through a multi-dimensional detection mechanism, identity-driven personalized enhancement, a hierarchical and robust confirmation strategy, and a timing-compensated output mechanism, the accuracy of interruption detection is significantly improved, the false interruption rate is reduced, and the user experience of full-duplex voice interaction is enhanced. In actual testing, the interruption accuracy can be improved from approximately 74% to approximately 85%, and the false interruption rate can be controlled below 10%, demonstrating promising application prospects.

[0192] Taking the application of the audio processing method provided in this specification in human-computer interaction as an example, the audio processing method will be further explained.

[0193] Specifically, during system broadcasts, the interactive terminal streams audio frames captured by the microphone. After the audio is standardized to a predetermined sampling rate, it is divided into consecutive short frames of 10 milliseconds. Simultaneously, the system maintains a 700-millisecond circular pre-buffer, continuously writing audio from each branch into it. In a specific scenario, the intelligent customer service robot is broadcasting product information to a user, while the microphone continuously captures a mixture of audio including the user's voice, conversations of nearby colleagues, and ambient noise.

[0194] Preprocessing is performed on the input audio, maintaining three data streams in parallel: the near-end raw audio for environment estimation, the general noise-reduced audio for speech detection, and the personalized enhanced audio for the final output. The three data streams are strictly aligned using a common buffer index. The echo cancellation module first removes the echo generated by the robot's own broadcasts, and then processes the three branches separately.

[0195] Environmental state variables, such as the level of noise, are estimated from the near-end original branch. A smoothing process, combining fast attack and slow release, is applied, and the sensitivity and policy pattern are encoded based on integer sensitivity levels. Environmental estimation identifies the current environment as a moderately noisy open office environment, and the sensitivity level is selected to appropriately tighten the speech validity conditions in the corresponding policy branch.

[0196] Low-level speech activity detection is performed on each audio frame to obtain the first audio detection information. At the same time, energy features are calculated, and gain normalization is applied before they are sent for detection. The low-level VAD detector identifies that the user's and colleague's speech both meet the speech decision conditions, but energy gating filters out low-energy ambient noise and noise reduction residue.

[0197] A frame is counted as a valid hit only if both the speech threshold and energy threshold are met simultaneously. The number of hits is accumulated within a 100-millisecond statistical window, and the dual-density mask is updated accordingly. The relaxed density mask is set when at least 6 frames meet the conditions within the window, while the strict density mask is set when all 10 frames meet the conditions. Intermittent conversations between colleagues do not satisfy the strict density mask, while both masks are set when the user speaks continuously.

[0198] Target speaker extraction is performed based on the reference audio information of the target audience to obtain personalized enhanced audio. Using the user's pre-registered 192-dimensional acoustic vector as a condition, the enhancement model extracts the user's speech components and suppresses colleague speech and environmental noise.

[0199] The system integrates multiple dimensions of evidence, including loose density masks, strict density masks, identity information evidence, and environmental noise levels, to determine the current audio state. When both masks are valid and the identity information evidence confirms that the current voice is the user's, the system state changes from candidate to main speaker, generating an interruption event.

[0200] When a judgment is made, up to 700 milliseconds of sentence-initial audio is retrieved from the circular pre-buffer and output along with the current frame to compensate for the sentence-initial audio during the judgment delay. The approximately 300 milliseconds of sentence-initial audio between the user speaking and the completion of the judgment are fully backfilled.

[0201] The interaction strategy is determined and executed based on the audio processing results. Upon receiving an interruption signal, the voice control center stops the current broadcast, sends the enhanced audio to the speech recognition module for transcription, and then sends the transcription result to the large language model for intent understanding and response generation.

[0202] Through the above steps, in complex scenarios where multiple people speak simultaneously or alternately, the intelligent voice customer service system narrows the basis for interruption judgment from the presence of voice to the presence of the speaker's valid voice, thus achieving a simultaneous improvement in interruption accuracy and interactive experience.

[0203] Corresponding to the above method embodiments, this specification also provides embodiments of audio processing apparatus. Figure 6 A schematic diagram of an audio processing apparatus according to one embodiment of this specification is shown. Figure 6 As shown, the device includes: The receiving module 602 is configured to receive the audio to be processed by the target object during the interaction process; The detection module 604 is configured to detect the audio to be processed according to at least one detection dimension to obtain audio detection information corresponding to the audio to be processed, wherein the audio detection information is used to characterize the audio quality and reliability of the audio to be processed; The determining module 606 is configured to determine the audio processing result corresponding to the audio to be processed based on the audio detection information and the reference audio information corresponding to the target object. The audio processing result is used to determine the interaction strategy corresponding to the interaction process. The audio processing result includes the target audio corresponding to the target object and / or the audio detection signal corresponding to the audio to be processed. The target audio is obtained by extracting the audio to be processed.

[0204] In an optional embodiment, the detection module 604 is further configured to: Based on the first detection dimension, an existence detection is performed on the audio to be processed to obtain first audio detection information corresponding to the audio to be processed, wherein the first audio detection information is used to characterize whether speech exists in the audio to be processed; and / or Based on the second detection dimension, the audio to be processed is subjected to validity detection to obtain the second audio detection information corresponding to the audio to be processed, wherein the second audio detection information is used to characterize whether there is valid speech in the audio to be processed.

[0205] In an optional embodiment, the detection module 604 is further configured to: Based on the first detection dimension, environmental noise is detected on the audio to be processed to obtain environmental state information corresponding to the audio to be processed. Based on the environmental state information, the audio detection parameters of the audio detector are adjusted to obtain the adjusted audio detector; Based on the adjusted audio detector, speech detection is performed on the audio to be processed to obtain the first audio detection information corresponding to the audio to be processed.

[0206] In an optional embodiment, the detection module 604 is further configured to: According to the second detection dimension, energy detection is performed on each audio frame to be processed corresponding to the audio to be processed to obtain energy detection results, and the target audio frame to be processed is determined from each audio frame to be processed based on the energy detection results. Based on a preset length window, density detection is performed on the target audio frame to be processed to obtain density detection results, and second audio detection information corresponding to the audio to be processed is determined based on the density detection results.

[0207] In an optional embodiment, the detection module 604 is further configured to: Based on the second detection dimension, the audio to be processed is subjected to gain normalization to obtain the processed audio. Determine each audio frame to be processed corresponding to the processed audio to be processed, perform low-level speech detection on each audio frame to be processed, and obtain the low-level speech detection result corresponding to each audio frame to be processed, wherein the low-level speech detection result is used to characterize whether each audio frame to be processed is speech. Calculate the energy characteristics for each audio frame to be processed; Based on the energy features and underlying speech detection results corresponding to each audio frame to be processed, the target audio frame to be processed is determined from each audio frame to be processed.

[0208] In an optional embodiment, the detection module 604 is further configured to: Based on a preset length window, the target audio frames to be processed are arranged in the preset length window; Based on the first density mask and the second density mask, density detection is performed on the target audio frame to be processed in the preset length window to obtain the density detection result, and the second audio detection information corresponding to the audio to be processed is determined according to the density detection result. The first density mask is used to detect whether there is valid speech within the preset length window, and the second density mask is used to detect whether the target audio frame to be processed within the preset length window meets the continuous speech condition.

[0209] In an optional embodiment, the detection module 604 is further configured to: The audio to be processed is denoised to obtain the denoised audio frequency corresponding to the audio to be processed; Based on the reference audio information corresponding to the target object, the audio to be processed is extracted to obtain the target audio corresponding to the target object; Based on the second detection dimension, the effectiveness of the denoised audio is detected to obtain the second audio detection information corresponding to the audio to be processed.

[0210] In an optional embodiment, the detection module 604 is further configured to: Based on the reference audio information corresponding to the target object, the audio to be processed is extracted to obtain the initial audio corresponding to the target object; If the initial audio meets the preset hit conditions, the associated audio of the initial audio is obtained from the pre-buffered audio corresponding to the target object, and the initial audio and the associated audio are concatenated to obtain the target audio corresponding to the target object.

[0211] In an optional embodiment, the determining module 606 is further configured to: Based on the audio detection information and the reference audio information corresponding to the target object, the current audio state corresponding to the audio to be processed is determined; Based on the current audio state, determine the audio processing result corresponding to the audio to be processed.

[0212] In one optional embodiment, the audio to be processed includes a plurality of audio frames to be processed; The receiving module 602 is further configured to: The target object receives the audio to be processed during the interaction process in a streaming manner; The detection module 604 is further configured as follows: Based on at least one detection dimension, each audio frame to be processed is detected to obtain audio detection information corresponding to each audio frame to be processed. The determining module 606 is further configured to: The audio processing result corresponding to the audio to be processed is determined based on the audio to be processed, the audio detection information corresponding to each audio frame to be processed, and the reference audio information corresponding to the target object.

[0213] In an optional embodiment, the receiving module 602 is further configured to: Receive reference audio information corresponding to the target object; or In response to the registration request of the target object, the reference audio corresponding to the target object is collected; Information is extracted from the reference audio to obtain the reference audio information.

[0214] In an optional embodiment, the receiving module 602 is further configured to: The reference audio is divided into multiple audio units, and the audio similarity information between the multiple audio units is calculated. If the audio similarity information is determined to meet the consistency condition, information is extracted from the reference audio to obtain the reference audio information; If the audio similarity information does not meet the consistency condition, a registration failure message is generated.

[0215] In an optional embodiment, the determining module 606 is further configured to: Based on the audio processing results, the interaction strategy corresponding to the target object is determined and executed.

[0216] The above is an illustrative scheme of an audio processing apparatus according to this embodiment. It should be noted that the technical solution of this audio processing apparatus and the technical solution of the above-described audio processing method belong to the same concept. For details not described in detail in the technical solution of the audio processing apparatus, please refer to the description of the technical solution of the above-described audio processing method.

[0217] Corresponding to the above method embodiments, this specification also provides an audio processing system, including: The acquisition unit is configured to receive the audio to be processed by the target object during the interaction process; An object recognition unit is configured to detect the audio to be processed according to at least one detection dimension to obtain audio detection information corresponding to the audio to be processed, wherein the audio detection information is used to characterize the audio quality and reliability of the audio to be processed; and to determine the audio processing result corresponding to the audio to be processed based on the audio detection information and reference audio information corresponding to the target object, wherein the audio processing result is used to determine the interaction strategy corresponding to the interaction process, and the audio processing result includes target audio corresponding to the target object and / or audio detection signal corresponding to the audio to be processed, wherein the target audio is obtained by extracting the audio to be processed; A speech recognition unit is configured to convert the audio to be processed and the target audio into converted text; The semantic understanding unit is configured to generate response text based on the converted text; The voice broadcasting unit is configured to synthesize and broadcast the reply text. The decision unit is configured to determine an interaction strategy based on the transcribed text, the response text, and the audio detection signal.

[0218] Among them, the acquisition unit can be the aforementioned customer voice acquisition module 10, the object recognition unit can be the aforementioned speaker recognition module 20, the speech recognition unit can be the aforementioned ASR speech recognition module 30, the semantic understanding unit can be the aforementioned LLM semantic understanding and generation module 40, the speech broadcasting unit can be the aforementioned TTS broadcasting module 60, and the decision-making unit can be the aforementioned voice central control interruption decision-making module 50.

[0219] The above is an illustrative scheme of an audio processing system according to this embodiment. It should be noted that the technical solution of this audio processing system and the technical solution of the audio processing method described above belong to the same concept. For details not described in detail in the technical solution of the audio processing system, please refer to the description of the technical solution of the audio processing method described above.

[0220] Figure 7 A structural block diagram of an electronic device 700 provided according to one embodiment of this specification is shown.

[0221] A memory 710 and a processor 720 are connected via a bus 730; The memory 710 is used to store computer programs / instructions, and the processor 720 is used to execute the computer programs / instructions, which, when executed by the processor 720, implement the steps of the method.

[0222] Specifically, the components of the electronic device 700 include, but are not limited to, a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and the database 750 is used to store data.

[0223] Electronic device 700 also includes access device 740, which enables electronic device 700 to communicate via one or more networks 760. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. Access device 740 may include one or more of any type of wired or wireless network interface (e.g., network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, Wi-MAX (Worldwide Interoperability for Microwave Access) interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth interface, Near Field Communication (NFC) interface, and so on.

[0224] In one embodiment of this specification, the above-described components of the electronic device 700 and Figure 7 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 7 The block diagram of the electronic device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0225] Electronic device 700 can be any type of stationary or mobile electronic device, including mobile computers or mobile electronic devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable electronic devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary electronic devices such as desktop computers or personal computers (PCs). Electronic device 700 can also be a mobile or stationary server.

[0226] The above is an illustrative scheme of an electronic device according to this embodiment. It should be noted that the technical solution of this electronic device and the technical solution of the above method belong to the same concept, and all details not described in detail in the technical solution of the electronic device can be referred to the description of the technical solution of the above method.

[0227] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0228] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the method described above belong to the same concept, and all details not described in detail in the technical solution of the storage medium can be referred to the description of the technical solution of the method described above.

[0229] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0230] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the above method belong to the same concept, and all details not described in detail in the technical solution of the computer program product can be referred to the description of the technical solution of the above method.

[0231] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0232] The computer program / instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0233] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0234] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0235] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. An audio processing method, comprising: Receive the audio to be processed from the target object during the interaction process; The audio to be processed is detected according to at least one detection dimension to obtain audio detection information corresponding to the audio to be processed, wherein the audio detection information is used to characterize the audio quality and reliability of the audio to be processed; Based on the audio detection information and the reference audio information corresponding to the target object, the audio processing result corresponding to the audio to be processed is determined. The audio processing result is used to determine the interaction strategy corresponding to the interaction process. The audio processing result includes the target audio corresponding to the target object and / or the audio detection signal corresponding to the audio to be processed. The target audio is obtained by extracting the audio to be processed.

2. The method according to claim 1, wherein detecting the audio to be processed based on at least one detection dimension to obtain audio detection information corresponding to the audio to be processed includes: Based on the first detection dimension, an existence detection is performed on the audio to be processed to obtain first audio detection information corresponding to the audio to be processed, wherein the first audio detection information is used to characterize whether speech exists in the audio to be processed; and / or Based on the second detection dimension, the audio to be processed is subjected to validity detection to obtain the second audio detection information corresponding to the audio to be processed, wherein the second audio detection information is used to characterize whether there is valid speech in the audio to be processed.

3. The method according to claim 2, wherein performing existence detection on the audio to be processed according to a first detection dimension to obtain first audio detection information corresponding to the audio to be processed includes: Based on the first detection dimension, environmental noise is detected on the audio to be processed to obtain environmental state information corresponding to the audio to be processed. Based on the environmental state information, the audio detection parameters of the audio detector are adjusted to obtain the adjusted audio detector; Based on the adjusted audio detector, speech detection is performed on the audio to be processed to obtain the first audio detection information corresponding to the audio to be processed.

4. The method according to claim 2, wherein performing validity detection on the audio to be processed according to the second detection dimension to obtain the second audio detection information corresponding to the audio to be processed includes: According to the second detection dimension, energy detection is performed on each audio frame to be processed corresponding to the audio to be processed to obtain energy detection results, and the target audio frame to be processed is determined from each audio frame to be processed based on the energy detection results. Based on a preset length window, density detection is performed on the target audio frame to be processed to obtain density detection results, and second audio detection information corresponding to the audio to be processed is determined based on the density detection results.

5. The method according to claim 4, wherein performing energy detection on each audio frame to be processed corresponding to the audio to be processed according to the second detection dimension, obtaining energy detection results, and determining the target audio frame to be processed from each audio frame to be processed according to the energy detection results, comprises: Based on the second detection dimension, the audio to be processed is subjected to gain normalization to obtain the processed audio. Determine each audio frame to be processed corresponding to the processed audio to be processed, perform low-level speech detection on each audio frame to be processed, and obtain the low-level speech detection result corresponding to each audio frame to be processed, wherein the low-level speech detection result is used to characterize whether each audio frame to be processed is speech. Calculate the energy characteristics for each audio frame to be processed; Based on the energy features and underlying speech detection results corresponding to each audio frame to be processed, the target audio frame to be processed is determined from each audio frame to be processed.

6. The method according to claim 4, wherein performing density detection on the target audio frame to be processed based on a preset length window to obtain a density detection result, and determining the second audio detection information corresponding to the audio to be processed based on the density detection result, comprises: Based on a preset length window, the target audio frames to be processed are arranged in the preset length window; Based on the first density mask and the second density mask, density detection is performed on the target audio frame to be processed in the preset length window to obtain the density detection result, and the second audio detection information corresponding to the audio to be processed is determined according to the density detection result. The first density mask is used to detect whether there is valid speech within the preset length window, and the second density mask is used to detect whether the target audio frame to be processed within the preset length window meets the continuous speech condition.

7. The method according to claim 2, further comprising, after receiving the audio to be processed from the target object during the interaction process: The audio to be processed is denoised to obtain the denoised audio frequency corresponding to the audio to be processed; Based on the reference audio information corresponding to the target object, the audio to be processed is extracted to obtain the target audio corresponding to the target object; The step of performing validity detection on the audio to be processed according to the second detection dimension to obtain the second audio detection information corresponding to the audio to be processed includes: Based on the second detection dimension, the effectiveness of the denoised audio is detected to obtain the second audio detection information corresponding to the audio to be processed.

8. The method according to claim 7, wherein extracting the audio to be processed based on the reference audio information corresponding to the target object to obtain the target audio corresponding to the target object includes: Based on the reference audio information corresponding to the target object, the audio to be processed is extracted to obtain the initial audio corresponding to the target object; If the initial audio meets the preset hit conditions, the associated audio of the initial audio is obtained from the pre-buffered audio corresponding to the target object, and the initial audio and the associated audio are concatenated to obtain the target audio corresponding to the target object.

9. The method according to claim 8, further comprising, after extracting the audio to be processed based on the reference audio information corresponding to the target object to obtain the initial audio corresponding to the target object: If it is determined that the initial audio does not meet the preset hit condition, and the cumulative length of the continuous non-speech audio currently corresponding to the initial audio does not meet the pre-buffer length, the initial audio is withheld to retain the pre-buffered audio corresponding to the target object.

10. The method according to any one of claims 1-9, wherein determining the audio processing result corresponding to the audio to be processed based on the audio detection information and the reference audio information corresponding to the target object includes: Based on the audio detection information and the reference audio information corresponding to the target object, the current audio state corresponding to the audio to be processed is determined; Based on the current audio state, determine the audio processing result corresponding to the audio to be processed.

11. The method according to any one of claims 1-9, wherein the audio to be processed comprises a plurality of audio frames to be processed; The process of receiving the audio to be processed from the target object during the interaction includes: The target object receives the audio to be processed during the interaction process in a streaming manner; The step of detecting the audio to be processed based on at least one detection dimension to obtain audio detection information corresponding to the audio to be processed includes: Based on at least one detection dimension, each audio frame to be processed is detected to obtain audio detection information corresponding to each audio frame to be processed. The step of determining the audio processing result corresponding to the audio to be processed based on the audio detection information and the reference audio information corresponding to the target object includes: The audio processing result corresponding to the audio to be processed is determined based on the audio to be processed, the audio detection information corresponding to each audio frame to be processed, and the reference audio information corresponding to the target object.

12. The method according to any one of claims 1-9, further comprising: Receive reference audio information corresponding to the target object; or In response to the registration request of the target object, the reference audio corresponding to the target object is collected, and information is extracted from the reference audio to obtain the reference audio information.

13. The method according to claim 12, wherein extracting information from the reference audio to obtain the reference audio information comprises: The reference audio is divided into multiple audio units, and the audio similarity information between the multiple audio units is calculated. If the audio similarity information is determined to meet the consistency condition, information is extracted from the reference audio to obtain the reference audio information; If the audio similarity information does not meet the consistency condition, a registration failure message is generated.

14. The method according to any one of claims 1-9, wherein after determining the audio processing result corresponding to the audio to be processed based on the audio detection information and the reference audio information corresponding to the target object, the method further comprises: Based on the audio processing results, determine and execute the interaction strategy corresponding to the target object; The step of determining and executing the interaction strategy corresponding to the target object based on the audio processing result includes: If the audio detection signal included in the audio processing result indicates that the audio to be processed does not belong to the target object, the interaction strategy corresponding to the target object is determined to be not to trigger the interruption of the interaction process, and the interaction strategy is executed.

15. An audio processing apparatus, comprising: The receiving module is configured to receive the audio to be processed by the target object during the interaction process; The detection module is configured to detect the audio to be processed according to at least one detection dimension to obtain audio detection information corresponding to the audio to be processed, wherein the audio detection information is used to characterize the audio quality and reliability of the audio to be processed; The determining module is configured to determine the audio processing result corresponding to the audio to be processed based on the audio detection information and the reference audio information corresponding to the target object. The audio processing result is used to determine the interaction strategy corresponding to the interaction process. The audio processing result includes the target audio corresponding to the target object and / or the audio detection signal corresponding to the audio to be processed. The target audio is obtained by extracting the audio to be processed.

16. An audio processing system, comprising: The acquisition unit is configured to receive the audio to be processed by the target object during the interaction process; An object recognition unit is configured to detect the audio to be processed according to at least one detection dimension to obtain audio detection information corresponding to the audio to be processed, wherein the audio detection information is used to characterize the audio quality and reliability of the audio to be processed; and to determine the audio processing result corresponding to the audio to be processed based on the audio detection information and reference audio information corresponding to the target object, wherein the audio processing result is used to determine the interaction strategy corresponding to the interaction process, and the audio processing result includes target audio corresponding to the target object and / or audio detection signal corresponding to the audio to be processed, wherein the target audio is obtained by extracting the audio to be processed; A speech recognition unit is configured to convert the audio to be processed and the target audio into converted text; The semantic understanding unit is configured to generate response text based on the converted text; The voice broadcasting unit is configured to synthesize and broadcast the reply text. The decision unit is configured to determine an interaction strategy based on the transcribed text, the response text, and the audio detection signal.

17. An electronic device comprising: A memory and a processor, the memory and the processor being connected via a bus; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 12.

18. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 12.

19. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 12.

Images